Information processing device, information processing method and program
Patent Information
- Application Number
- JP2021206251
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-12-20
- Publication Date
- 2024-11-27
- Estimated Expiration
- 2041-12-20
AI Technical Summary
In mixed reality (MR) and augmented reality (AR), the inconsistency between the virtual and real viewpoints during a walkthrough function leads to difficulties in understanding the gaze point of a user's virtual 3D model, making communication challenging among multiple users.
An information processing apparatus that generates a video effect based on the real viewpoint of other users to align the virtual and real viewpoints, using devices with imaging units and display units to present a mixed reality image that indicates the virtual viewpoint's position relative to the real viewpoint.
This solution eliminates inconsistencies between the virtual and real viewpoints, allowing users to easily grasp the location of other users during a walkthrough, enhancing communication and coherence in MR and AR environments.
Abstract
Description
[Technical Field]
[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]
[0002] In recent years, virtual reality (VR), mixed reality (MR), and augmented reality (AR) have been widely used. VR, MR, and AR have a function that allows a user to move only the viewpoint (hereinafter referred to as the virtual viewpoint) that renders a virtual 3D model of, for example, a building or a large structure represented by a virtual 3D model. Such a function that allows a user to move the virtual viewpoint by user operation is called a walk-through function. The walk-through function allows a user to observe various places without moving themselves. To smoothly use the walk-through function, it is preferable to be able to grasp the positional relationship between the user and the virtual 3D model to be observed.
[0003] Patent Document 1 discloses a virtual reality presentation device that has a real-size observation mode in which a user is immersed in a real-size virtual three-dimensional space and observes it, and a reduced-size observation mode in which a virtual three-dimensional space CG is reduced and observed. The virtual reality presentation device of Patent Document 1 presents a CG map with user position and orientation information added in the reduced-size observation mode, allowing the user to grasp their position in the virtual three-dimensional space. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-79174 Summary of the Invention [Problem to be solved by the invention]
[0005] In mixed reality (MR) and augmented reality (AR), which display a virtual 3D model superimposed on real space, using the walk-through function causes the virtual viewpoint to move, resulting in mismatches between the real space and the virtual 3D model. Consider a scenario in which multiple people are observing a virtual 3D model. When a specific user uses the walk-through function, a mismatch occurs between the actual head viewpoint of the user during the walk-through (hereinafter referred to as the real viewpoint) as seen by other users and the virtual viewpoint that renders the 3D model displayed to the user during the walk-through. This creates a problem in that other users cannot understand the point of gaze of the user during the walk-through on the virtual 3D model, making communication difficult.
[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to resolve the inconsistency in the relationship between the virtual viewpoint and the real viewpoint of another user. [Means for solving the problem]
[0007] The present invention is an information processing device for sharing a space in which a virtual three-dimensional model is presented with other users, characterized in that it has an acquisition means for acquiring information on the real viewpoint of the other users, and a generation means for generating a visual effect based on information on the real viewpoint of the other users when the virtual viewpoint of the other users differs from the real viewpoint of the other users. [Effects of the Invention]
[0008] According to the present invention, it is possible to eliminate inconsistencies in the relationship between the virtual viewpoint and the real viewpoint of another user. [Brief explanation of the drawings]
[0009] [Figure 1] FIG. 10 is a schematic diagram showing a case where a specific user uses the walk-through function. [Figure 2] FIG. 1 is a diagram illustrating a functional configuration of an information processing system according to a first embodiment. [Figure 3] FIG. 1 is a diagram illustrating a hardware configuration of an information processing device. [Figure 4] FIG. 2 is a diagram schematically illustrating a mixed reality image seen from the user's perspective. [Figure 5] 10 is a flowchart showing processing by the information processing device. [Figure 6] FIG. 10 is a plan view schematically showing the situation of the second embodiment. [Figure 7] FIG. 2 is a diagram schematically illustrating a mixed reality image seen from the user's perspective. [Figure 8] FIG. 10 is a diagram illustrating a functional configuration of an information processing system according to a second embodiment. [Figure 9] 10 is a flowchart showing processing by the information processing device. [Figure 10] FIG. 10 is a plan view schematically showing the situation of the third embodiment. [Figure 11] FIG. 2 is a diagram schematically illustrating a mixed reality image seen from the user's perspective. [Figure 12] 10 is a flowchart showing processing by the information processing device. DETAILED DESCRIPTION OF THE INVENTION
[0010] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. FIG. 1 is a plan view that schematically illustrates the positional relationships when multiple people observe a virtual three-dimensional model and a specific user uses the walk-through function. Here, user A101 and user B102 share a space in which a virtual three-dimensional model is presented. User A101 is the user himself / herself, and user B102 is the other user. User A101's real viewpoint 104 and virtual viewpoint 105 are in the same position. On the other hand, user B102 uses the walk-through function, and therefore his / her real viewpoint 106 and virtual viewpoint 107 are different. User B's avatar 103 is displayed at the position of user B102's virtual viewpoint 107. When user A101 views this situation from real viewpoint 104, user B102's real viewpoint 106 is located to the right of virtual model 108, while user B102's virtual viewpoint 107 is located to the left of virtual model 108. This makes it difficult for user A101 to grasp the location of user B102.
[0011] In each of the following embodiments, when another user's virtual viewpoint differs from their real viewpoint, a visual effect based on information about the other user's real viewpoint is generated to resolve the inconsistency in the relationship between the other user's virtual viewpoint and their real viewpoint.
[0012] [First embodiment] In this embodiment, a visual effect is generated based on information about the real viewpoint of another user during a walkthrough. Specifically, an image indicating that the virtual viewpoint of the other user does not exist is presented at a position corresponding to the real viewpoint of the other user. When the user looks at the position corresponding to the real viewpoint of the other user, the user can recognize the image indicating that the virtual viewpoint of the other user does not exist, and therefore the user can easily understand that the virtual viewpoint of the other user is not the position of the real viewpoint.
[0013] 4 is a diagram showing a schematic diagram of a mixed reality image seen from the real viewpoint of the user A 101. Note that the positional relationship of each object, such as the avatar 103 and the virtual model 108, corresponds to that shown in FIG. User B102 is currently walking through, and avatar 103 of user B102 is displayed at the position and orientation of the virtual viewpoint of user B102. Here, the position and orientation refers to the position and orientation (direction). Also, the color of area 401 surrounding the body of user B102 has changed. Text information 402 indicating that a walkthrough is in progress is displayed near area 401.
[0014] FIG. 2 is a diagram illustrating an example of a functional configuration of the information processing system. The information processing system can realize a mixed reality or an augmented reality in which a virtual three-dimensional model is observed by multiple people by having devices used by each user cooperate via a transmitter and a receiver implemented in each device. The information processing system shown in FIG. 2 is configured so that there are two users, and two devices, device 201 and device 221, cooperate with each other. Note that when three or more people are observing the virtual three-dimensional model, three or more devices cooperate with each other. An embodiment in which there are two users will be described below.
[0015] The device 201 includes a terminal 202 , an operation input unit 205 , and an information processing device 217 . The terminal 202 captures images of the surrounding real environment and presents mixed reality images to the user. The terminal 202 includes an imaging unit 203 and a display unit 204. The image capturing unit 203 captures images of the surrounding real environment. The display unit 204 presents a mixed reality image to the user.
[0016] The terminal 202 of this embodiment uses a head-mounted display (hereinafter referred to as HMD) in which an imaging unit 203 and a display unit 204 are mounted in a head-mounted housing. When a user wears the HMD on their head, the HMD moves in conjunction with the user's movements and the movements of their head. The terminal 202 may be a tablet terminal, a smartphone terminal, or the like. The terminal 202 of this embodiment is a video see-through type that superimposes a virtual three-dimensional model on an actual image captured by the imaging unit 203 and displays it on the display unit 204, but it may also be an optical see-through type.
[0017] The imaging unit 203 has a camera that captures images to be presented to the user, and a camera that acquires images necessary for estimating the position and orientation of the terminal 202 . The imaging unit 203 may use the same camera to both capture an image and acquire an image necessary for estimating the position and orientation. The imaging unit 203 may omit one or both of the two cameras depending on the method for acquiring the position and orientation of the terminal 202 and the method for superimposing a virtual 3D model.
[0018] Furthermore, in the terminal 202 of this embodiment, a camera for capturing images to be presented to the user to realize stereoscopic vision and a display device for presenting mixed reality images are separately mounted on the right and left eyes. However, the configurations of the camera and display device mounted on the terminal 202 can be changed depending on the form of the terminal 202, the method for acquiring the position and orientation of the terminal 202, and the method for superimposing a virtual three-dimensional model.
[0019] The operation input unit 205 accepts an operation for setting the walk-through function so that the user can use it, and an operation for moving the virtual viewpoint input for walk-through. Specifically, the operation input unit 205 accepts one or more of button input from a controller device or the like, gesture input, voice input, etc. The operation input unit 205 may be provided in the terminal 202, or may be configured as a separate device.
[0020] The information processing device 217 is a device for sharing a space in which a virtual three-dimensional model is presented with other users. The information processing device 217 estimates the position and orientation of the terminal 202 and generates a mixed reality image. The information processing device 217 includes an operation information acquisition unit 206 , a captured image acquisition unit 207 , a real viewpoint position and orientation acquisition unit 208 , a virtual viewpoint position and orientation generation unit 209 , a transmission unit 210 , a reception unit 211 , a virtual image generation unit 212 , and a synthesis unit 213 .
[0021] The operation information acquisition unit 206 acquires the operation information received by the operation input unit 205 . The captured image acquisition unit 207 acquires the captured image of the real environment that is transmitted from the terminal 202 to the information processing device 217 . The real viewpoint position and orientation acquisition unit 208 acquires information about the real viewpoint of the user himself / herself. Specifically, the real viewpoint position and orientation acquisition unit 208 acquires information about the real viewpoint of the user himself / herself by calculating the position and orientation of the real viewpoint of the user himself / herself from the captured image.
[0022] The virtual viewpoint position and orientation generation unit 209 generates information about the user's virtual viewpoint. Specifically, the virtual viewpoint position and orientation generation unit 209 calculates the position and orientation of the virtual viewpoint based on information about the real viewpoint and the operation information acquired by the operation information acquisition unit 206, thereby generating information about the user's virtual viewpoint. The transmission unit 210 transmits to the device 221 information on the position and orientation of the real viewpoint of the user and information on the position and orientation of the virtual viewpoint of the user. The receiving unit 211 receives information on the position and orientation of the real viewpoint of the other user and information on the position and orientation of the virtual viewpoint of the other user from the device 221 .
[0023] The virtual image generation unit 212 generates a virtual three-dimensional model based on information on the position and orientation of the user's virtual viewpoint, information on the position and orientation of the user's real viewpoint, information on the position and orientation of the other user's virtual viewpoint, and information on the position and orientation of the other user's real viewpoint. Furthermore, when the other user is performing a walkthrough, i.e., when the other user's virtual viewpoint and real viewpoint are different, the virtual image generation unit 212 generates a visual effect based on information on the other user's real viewpoint. The synthesis unit 213 synthesizes the virtual image and the image of the real environment to generate a mixed reality image.
[0024] The device 221 has the same configuration as the device 201. Therefore, the device 221 has an HMD as the terminal 202 to be worn on the head of another user. The device 221 also has an information processing device with the same configuration as the information processing device 217 of the device 201. In FIG. 2, only the transmitting unit 230 and the receiving unit 231 that configure the information processing device of the device 221 are shown.
[0025] FIG. 3 is a diagram illustrating an example of the hardware configuration of the information processing device 217. The information processing device 217 includes a CPU 301, a ROM 302, a RAM 303, an input I / F 304, and an output I / F 305. The various components of the hardware configuration of the information processing device 217 are connected via a bus 306.
[0026] The CPU 301 controls the entire information processing device 217. The CPU 301 also implements the functional configuration of the information processing device 217 described above by loading a program stored in a ROM 302 into a RAM 303 and executing the program. The ROM 302 stores programs executed by the CPU 301 and various data. The RAM 303 temporarily stores the program when the CPU 301 executes the program stored in the ROM 302, and temporarily stores the results of the execution.
[0027] The input I / F 304 is an interface for inputting various data from components within the device 201 or the device 221. Specifically, the information processing device 217 receives, via the input I / F 304, operation information from the operation input unit 205, an image captured by the imaging unit 203, and information from the transmission unit 230 of the device 221. The output I / F 305 is an interface for outputting various data to components within the device 201 or to the device 221. Specifically, the information processing device 217 outputs an image to the terminal 202 or outputs information to the receiving unit 231 of the device 221 via the output I / F 305.
[0028] The information processing device 217 may be a desktop computer such as a personal computer or a workstation, or may be a portable computing device. The information processing device 217 may also be integrated with the terminal 202.
[0029] Next, an example of processing by the information processing device 217 will be described with reference to the flowchart in Fig. 5. Each flowchart including Fig. 5 is realized by the CPU 301 of the information processing device 217 expanding a program stored in the ROM 302 into the RAM 303 and executing it. Here, it is assumed that the user and other users are set to use the walk-through function via the operation input unit 205.
[0030] In S501, the real viewpoint position and orientation acquisition unit 208 acquires information on the position and orientation of the real viewpoint of the user in world coordinates as information on the real viewpoint of the user. Here, methods for acquiring the information on the position and orientation of the real viewpoint include a method of estimating the position and orientation of the real viewpoint from the arrangement of feature points of markers in an image based on a captured image of markers placed in space. Another method is to use Simultaneous Localization and Mapping (SLAM), which uses natural feature points in a real image to simultaneously estimate the user's own position and create an environmental map. Alternatively, the information on the position and orientation of the real viewpoint may be acquired by using an external measurement device such as a motion capture device. The origin and orientation of the world coordinate system are determined in advance using a marker or a specific real object, and the origin and orientation of the world coordinate system are set to be common among multiple users.
[0031] In S502, the virtual viewpoint position and orientation generation unit 209 generates information on the position and orientation of the user's virtual viewpoint in world coordinates as information on the user's virtual viewpoint. Specifically, the virtual viewpoint position and orientation generation unit 209 calculates the amount of viewpoint movement based on the operation information acquired by the operation information acquisition unit 206. Next, the virtual viewpoint position and orientation generation unit 209 calculates information on the position and orientation of the user's virtual viewpoint in world coordinates by adding the calculated amount of viewpoint movement to the position and orientation of the user's actual viewpoint in world coordinates. Note that there are no particular limitations on the method for associating the viewpoint movement amount with the operation information. Furthermore, if the device constituting the operation input unit 205 or the device constituting the actual viewpoint position and orientation acquisition unit 208 can acquire the position and orientation of the user's body part such as the hand, the viewpoint movement amount can also be added to the position and orientation of the body part, which can be used for generating an avatar, which will be described later.
[0032] In S503, the transmission unit 210 transmits information about the real viewpoint and information about the virtual viewpoint of the user to the device 221. Furthermore, if information about the position and orientation of the body part has been acquired in S502, the transmission unit 210 transmits information about the position and orientation of the body part of the user and information obtained by adding the amount of movement of the viewpoint to the position and orientation of the body part. Furthermore, the receiving unit 211 receives information on the real viewpoint and information on the virtual viewpoint of the other user from the device 221. Furthermore, when the device 221 has acquired information on the position and orientation of the body part of the other user, the receiving unit 211 receives information on the position and orientation of the body part of the other user and information obtained by adding the amount of movement of the viewpoint to the position and orientation of the body part. Therefore, the device 201 and the device 221 can share information such as the user's actual viewpoint with each other.
[0033] In S504, the virtual image generation unit 212 generates an avatar based on information about the position and orientation of the virtual viewpoint of the other user. The virtual image generation unit 212 may generate an avatar with only the head, or may generate an avatar with representations of body parts such as the torso and legs at a position inferred from the head position. Furthermore, when information is received that adds the amount of viewpoint movement to the position and orientation of the body parts of the other user, the virtual image generation unit 212 may generate an avatar with representations of the virtual body parts of the other user that have moved by the walk-through, based on the received information. Note that identification information for identifying the user may be added to the avatar by superimposing it near the avatar or on the avatar. The identification information may include, for example, the user name, the user ID, a texture indicating the user's characteristics, etc.
[0034] In S505, the virtual image generation unit 212 estimates the region where the other user's body exists based on information about the position and orientation of the other user's real viewpoint. Specifically, the virtual image generation unit 212 estimates the region including the head as an ellipsoidal shape based on information about the position and orientation of the other user's real viewpoint in the world coordinate system. Furthermore, for the whole-body region other than the head, the virtual image generation unit 212 estimates that the feet are located at a predetermined distance vertically downward from the viewpoint based on vertical or floor information obtained by defining the vertical direction or estimating the floor. Next, a cylinder with a predetermined radius around a vertical axis connecting the feet and head is determined as the whole-body region including the other user's entire body. Note that the definition of the vertical direction and the floor can be estimated by setting the origin of the world coordinate system to the floor in S501 and aligning the vertical direction with a predetermined orientation, or by estimating the floor using natural features of SLAM. Alternatively, the vertical direction may be estimated by installing an orientation sensor in the terminal 202, or by other methods. Furthermore, when information on the position and posture of other users' body parts is received, the virtual image generation unit 212 may appropriately modify or add areas where other users' bodies are present based on the received information.
[0035] Furthermore, the body region may be estimated by combining three-dimensional polyhedrons in a world coordinate system, or may be estimated on a two-dimensional image presented to the user. In this case, the image presented to the user is projected onto the two-dimensional image based on the viewpoint position on the world coordinate system, the foot positions of the other users estimated by the above-described method, and, if information on the positions and orientations of the other users' body parts has been received, the received information. The virtual image generation unit 212 may generate a polygon including the body region of the other users on the projected two-dimensional coordinate system. However, the shape of the other users' regions is not limited and may be any shape.
[0036] In S506, the virtual image generation unit 212 generates an image at a position corresponding to the real viewpoint of the other user, indicating that the virtual viewpoint of the other user is different from the real viewpoint of the other user. In this embodiment, the virtual image generation unit 212 generates an image indicating that the virtual viewpoint of the other user is not present in the area where the body of the other user is present.
[0037] Specifically, the virtual image generation unit 212 generates an image in which the color of the area estimated in S505 is changed to a different color. For example, the color of the estimated region can be changed by adding color to the estimated region and drawing it. The virtual image generation unit 212 may be realized by drawing the three-dimensional polyhedron generated in S505 as is, or may be realized by overlaying a specific texture image within the estimated region or the region of the three-dimensional polyhedron. Furthermore, for example, the estimated region can be used to generate a mask, and the color of the real image corresponding to the mask region can be changed to change the color of the region. This can be realized by the synthesis unit 213 synthesizing the real image whose color has been changed by the virtual image generation unit 212 with an image of the real environment. Furthermore, to change the color of the estimated region, at least one of the saturation and brightness of the estimated region may be reduced to create the illusion that another user's virtual viewpoint does not exist. The method for changing the color of the estimated region is not particularly limited, and any method may be used.
[0038] Furthermore, the virtual image generation unit 212 generates text information indicating that other users are not present, specifically, that a walkthrough is in progress, by superimposing it on or near the area where the bodies of other users are present. Note that the information indicating that a walkthrough is in progress is not limited to text information, but may also be symbol information, an image such as an icon, or the like.
[0039] In S507, the synthesis unit 213 synthesizes the virtual image generated by the virtual image generation unit 212 with the image of the real environment to generate a mixed reality image. The synthesis unit 213 also synthesizes an image of an avatar at the position of the virtual viewpoint of another user during the walkthrough. The synthesis unit 213 outputs the generated mixed reality image to the terminal 202.
[0040] The terminal 202 presents the output mixed reality image to the user. Specifically, the display unit 204 of the terminal 202 displays the mixed reality image to present it to the user. The mixed reality image displayed on the display unit 204 includes an image indicating that the virtual viewpoint of the other user is different from the real viewpoint of the other user at a position corresponding to the real viewpoint of the other user, so the user can easily understand that the virtual viewpoint of the other user is not at the position of the real viewpoint. Therefore, it is possible to resolve inconsistencies in the relationship between the real viewpoint and the virtual viewpoint of the other user during a walkthrough in mixed reality or augmented reality.
[0041] In addition, if the information on the real viewpoint of another user and the information on the virtual viewpoint of another user are identical or approximately identical, the virtual image generation unit 212 can omit the process of generating a visual effect based on the information on the real viewpoint of another user.
[0042] [Second embodiment] In this embodiment, a so-called diminished reality effect is applied, in which the area where the other user exists is changed to a background image as a visual effect based on information about the other user's real viewpoint. When the first user observes the other user, the other user's body appears to have disappeared, so the first user can easily understand that the other user's virtual viewpoint is not the position of the real viewpoint.
[0043] 6 is a plan view schematically showing a mixed reality or augmented reality situation to which this embodiment is applied. Components similar to those in FIG. 1 are given the same reference numerals, and descriptions thereof will be omitted where appropriate. Here, a background object 601 is placed behind the body of user B102, who is walking through the experience, as viewed from user A101. The background object 601 is a real object placed within or around the experience space of mixed reality.
[0044] 7 is a diagram showing a schematic representation of a mixed reality image seen from the real viewpoint of user A101. In the area where the body of user B102 is present, an image 701 of a background object 601 that can be observed when the body of user B102 is not present is displayed. In addition, an avatar 103 of user B102 is displayed at the position and orientation of the virtual viewpoint of user B102.
[0045] 8 is a diagram showing an example of the functional configuration of an information processing system. Note that the same components as those in the first embodiment are denoted by the same reference numerals, and descriptions thereof will be omitted as appropriate. Furthermore, the hardware configuration of the information processing device 217 is the same as that in the first embodiment, and descriptions thereof will be omitted as appropriate. The information processing device 217 of the device 201 has a background information generating unit 801 added thereto. The background information generation unit 801 reconstructs background information for the case where no other users are present, based on the captured image captured by the imaging unit 203, information on the position and orientation of the real viewpoint acquired by the real viewpoint position and orientation acquisition unit 208, and information on natural feature points when SLAM is used.
[0046] Next, an example of processing by the information processing device 217 will be described with reference to the flowchart of Fig. 9. The flowchart of Fig. 9 is a flowchart that follows the processing of S504 in the flowchart of Fig. 5, for example.
[0047] In S901, the background information generation unit 801 generates background information by performing 3D reconstruction with texture of the background real space. For example, 3D reconstruction methods include a Structure from Motion (SfM) method based on information on the position and orientation of the real viewpoint and a real image, or a method using natural feature points obtained during the SLAM process. If the imaging unit 203 is a stereo camera, background information may be generated based on information from the stereo camera. Specifically, depth information in the image is generated from the stereo camera information, and point cloud information generated from the depth information is combined between frames by applying an ICP (Interactive Closest Point) algorithm or by using the position and orientation of the real viewpoint. The background information generation unit 801 converts the 3D reconstruction information into a mesh and generates background information by associating the vertices of the mesh with real images captured during the 3D reconstruction to create a texture. However, the 3D reconstruction method is not limited, and any method may be used.
[0048] Note that background information may be generated using a simplified method that does not perform three-dimensional reconstruction. For example, there is a method in which the background space is considered to be a polyhedron such as a sphere or a cube. Specifically, an image of the background space captured in advance using a fisheye camera is pasted as a texture on the inside of the polyhedron. Alternatively, there is a method in which, using the position and orientation of the image capture unit 203 and the actual viewpoint, an image that can be acquired when looking around with the image capture unit 203 is mapped onto the inside of the polyhedron according to the direction of the image capture unit 203. Although the simplified method that does not perform three-dimensional reconstruction lacks accuracy, it is useful in cases where there are insufficient computational resources for performing three-dimensional reconstruction.
[0049] In S902, the virtual image generation unit 212 estimates the area where the other user's body exists based on the position and orientation of the other user's actual viewpoint. In this embodiment, a background image is pasted onto the estimated area, so the estimated area is calculated by calculating the vertices of the other user's outline as two-dimensional coordinates on the image presented to the user. Specifically, first, a group of position information is acquired in a three-dimensional world coordinate system that includes the position of the other user's actual viewpoint during the walkthrough, the position of the other user's feet, and, if the positions and orientations of the other user's body parts are acquired, the positions and orientations of the other user's body parts. Note that the method for estimating the foot presence described in S506 of the first embodiment can be used to estimate the foot position. Next, the acquired group of position information is projected onto a two-dimensional image on the image presented to the user, and an area containing the other user's body is generated from the projected group of position information. For example, the amount of margin on the two-dimensional image is determined using distance information to the position of the other user's actual viewpoint, and four vertices, the upper right and upper left of the actual viewpoint position and the lower right and lower left of the foot position, are calculated based on the determined margin. Then, by using the calculated vertex information to generate a rectangular area that encompasses the body of the other user, the area where the body of the other user exists can be estimated. However, the shape of the area of the other user is not limited, and any shape may be used. Furthermore, the calculation process and method are not limited as long as the same results as the above-mentioned area estimation can be obtained.
[0050] In S903, the background information generation unit 801 generates a background image generated from the three-dimensional reconstruction information in the area where the estimated body of the other user is present. Specifically, the background information generation unit 801 generates the background image by drawing the background information generated in S901 in the area estimated in S902 from the viewpoint of the user observing the background information.
[0051] In S904, the virtual image generation unit 212 generates a composite image of the real space by superimposing the generated background image on the real image. Therefore, a composite image is generated in which the area where the estimated body of the other user is present is changed to the background image. In this embodiment, the body of the other user during the walkthrough is represented as if it does not exist. Therefore, it is assumed that the body of the observing user and the body of the other user during the walkthrough may come into contact when they come close to each other. Therefore, the virtual image generation unit 212 changes the background image according to the positional relationship between the real viewpoint of the observing user and the real viewpoint of the other user during the walkthrough. Specifically, the virtual image generation unit 212 makes the background image transparent when the distance between the real viewpoints is equal to or less than a predetermined distance, or changes the transparency of the background image so that it increases continuously or gradually depending on the distance between the real viewpoints. Furthermore, the color of the background image may change when the distance between the real viewpoints is equal to or less than a predetermined distance, or may change continuously or gradually depending on the distance between the real viewpoints. Furthermore, the color of the background image is not limited to the overall color of the background image, and the color of the outline of the background image may also be changed. Furthermore, the synthesis unit 213 may instruct the terminal 202 to sound an alarm when the distance between the real viewpoints is equal to or less than a predetermined distance or according to the distance between the real viewpoints.
[0052] In S905, the synthesis unit 213 generates a mixed reality image by drawing a virtual three-dimensional model on the synthesized image generated by the virtual image generation unit 212. The synthesis unit 213 also synthesizes an image of an avatar at the position of the virtual viewpoint of another user during the walk-through. The synthesis unit 213 outputs the generated mixed reality image to the terminal 202.
[0053] The terminal 202 presents the output mixed reality image to the user. Specifically, the display unit 204 of the terminal 202 displays the mixed reality image to present it to the user. The mixed reality image displayed on the display unit 204 displays a background image at a position corresponding to the real viewpoint of the other user, so the user can easily understand that the virtual viewpoint of the other user is not at the position of the real viewpoint. Therefore, it is possible to resolve inconsistencies in the relationship between the real viewpoint and virtual viewpoint of the other user during a walkthrough in mixed reality or augmented reality.
[0054] [Third embodiment] In this embodiment, a virtual space or a virtual three-dimensional model is deformed or processed as a visual effect based on information about the real viewpoint of another user. When a user observes a virtual three-dimensional model, a part of the three-dimensional model is displayed as a virtual three-dimensional model as seen from the virtual viewpoint of the other user, so the user can easily understand that the virtual viewpoint of the other user is not the position of the real viewpoint.
[0055] 10 is a plan view showing a schematic diagram of a mixed reality or augmented reality situation to which this embodiment is applied. Note that the same components as those in FIG. 1 are given the same reference numerals and descriptions thereof will be omitted as appropriate. FIG. 11 is a diagram schematically illustrating a mixed reality image seen from the real viewpoint of the user A101. Assume that user B102 moves to the virtual viewpoint 1001 by walk-through. At this time, user B102 is viewing the virtual model 108 from the position and orientation of the virtual viewpoint 1001. If the virtual viewpoint 1001 could return to the position and orientation of the real viewpoint 106 of user B102, i.e., the position and orientation of the virtual viewpoint 1002, while maintaining the relative positional relationship between the virtual viewpoint 1001 and the virtual model 108, the positional inconsistency between the real viewpoint and the virtual viewpoint would be resolved. However, such a virtual model would result in a geometric inconsistency.
[0056] In this embodiment, the virtual space itself or the virtual three-dimensional model is deformed or processed. Specifically, a boundary surface 1003 is set between the position of the real viewpoint 104 of user A101 and the position of the real viewpoint 106 of user B102, and the virtual space is divided across this boundary. Then, in each of the virtual spaces of user A101 and user B102, a coordinate system is set that maintains the relative position and orientation relationship between the virtual viewpoints 105, 1001 and the virtual model. As a result, in the virtual space on the user B102 side, the virtual model 1004 is displayed divided, and the positions of the virtual viewpoint 1002 and real viewpoint 106 of user B102 coincide with each other.
[0057] Next, an example of processing by the information processing device 217 will be described with reference to the flowchart of Fig. 12. The flowchart of Fig. 12 is a flowchart that follows the processing of S503 in the flowchart of Fig. 5, for example. Note that the functional configuration of the information processing system and the hardware configuration of the information processing device 217 are the same as those in the first embodiment, and descriptions thereof will be omitted where appropriate.
[0058] In S1201, the virtual image generation unit 212 calculates an inverse transformation matrix corresponding to the change from the position and orientation of the real viewpoint to the position and orientation of the virtual viewpoint that changes due to the walkthrough of another user. Next, the virtual image generation unit 212 applies the calculated inverse transformation matrix to the virtual space to generate a virtual space to which the inverse transformation has been applied. Note that the virtual image generation unit 212 may also generate a virtual three-dimensional model to which the inverse transformation has been applied by applying the calculated inverse transformation matrix to a virtual three-dimensional model in the virtual space.
[0059] In S1202, the virtual image generation unit 212 sets a boundary surface between the real viewpoint positions of the user and the other users using information on the position and orientation of the real viewpoints of the user and the other users. Specifically, a work plane is first defined with the vertical direction as the normal. If the floor surface described in the first embodiment has been estimated, the floor surface is used as the work plane. If there is no information on the floor surface, the average of the upward vectors of each user's field of view, or any of the X, Y, or Z axes of the world coordinate system that forms the smallest angle with the average vector, is used as the normal, and a plane passing through appropriate coordinates is used as the work plane. Next, the foot of the perpendicular line of each user's real viewpoint to the work plane is calculated. If there are two users, the plane passing through the perpendicular bisector of the perpendicular line and perpendicular to the work plane is used as the boundary surface. If there are three or more users, a Voronoi boundary for each user's perpendicular line is generated on the work plane, and the plane passing through this Voronoi boundary and perpendicular to the work plane is used as the boundary surface. To generate the Voronoi boundary, the real viewpoint of the user and the real viewpoint of the other user are projected onto a work surface, which is a predetermined plane, and the Voronoi boundary can be calculated based on the projected points.
[0060] In S1203, the virtual image generation unit 212 clips (cuts out) the virtual space based on the set boundary surface. Here, a case where a virtual three-dimensional model included in the virtual space is clipped will be described with reference to Fig. 10. First, for a boundary surface 1003 generated between user A101 and user B102, the virtual model 108 on the user A101 side clips the user B102 side at the boundary surface 1003. On the other hand, the virtual model 1004 on the user B102 side after application of the inverse transformation clips the user A101 side at the boundary surface 1003.
[0061] In S1204, the virtual image generation unit 212 generates a virtual image drawn from the viewpoint of the user observing the clipped virtual space. The generated virtual image is an image in which, when the user views the other user from the real viewpoint of the user, the position of the other user's virtual viewpoint appears to coincide with the position of the other user's real viewpoint. At this time, a boundary surface may be added to the virtual image, or the intersection line between the boundary surface and the work surface may be added as a boundary line so that the user can recognize that the virtual space or virtual three-dimensional model is clipped based on the boundary surface.
[0062] Furthermore, the virtual image generating unit 212 generates text information 1101 near the boundary surface or boundary line indicating that the virtual viewpoint of the other user is different from the real viewpoint of the other user, specifically indicating that a walk-through is in progress as shown in Fig. 11. Note that the information indicating that a walk-through is in progress is not limited to text information, and may be symbol information, an image such as an icon, or the like. The synthesis unit 213 outputs the virtual image generated by the virtual image generation unit 212 to the terminal 202. Note that, if necessary, the synthesis unit 213 may generate a mixed reality image by synthesizing the virtual image generated by the virtual image generation unit 212 with an image of the real environment, and output the generated mixed reality image to the terminal 202.
[0063] The terminal 202 presents the output virtual image or mixed reality image to the user. Specifically, the display unit 204 of the terminal 202 displays the virtual image or mixed reality image to present it to the user. Because the image displayed on the display unit 204 shows a virtual three-dimensional model divided by a boundary surface, the user can easily understand that the virtual viewpoint of the other user is not the position of the real viewpoint. Therefore, it is possible to resolve inconsistencies in the relationship between the real viewpoint and virtual viewpoint of the other user during a walkthrough in mixed reality or augmented reality.
[0064] In this embodiment, the case where the virtual space is clipped at the boundary surface has been described, but it is also possible to clip a virtual three-dimensional model at the boundary surface and generate a mixed reality image by combining the clipped virtual three-dimensional model with an image of the real environment. Furthermore, the present invention is not limited to clipping the virtual 3D model on the boundary surface, but may also provide a gap between the boundaries by setting an offset to the boundary surface, or may conversely allow the virtual 3D models to overlap.Furthermore, the side surface of a cylinder having a predetermined radius around a vertical axis passing through the real viewpoint of the user or another user may be used as the boundary surface.
[0065] Furthermore, the transparency of the boundary surface may be added continuously or in stages, or the color of the boundary surface may be changed depending on the distance between the user and the boundary surface. Furthermore, depending on the position or posture moved during walk-through, the entire boundary surface may be blocked by the virtual three-dimensional model, making it impossible to visually recognize what the user is doing. In this case, the area of the other user's body is estimated using the method described in the first embodiment. If the virtual three-dimensional model intersects with the area connecting the estimated area and the actual viewpoint of the user observing the other user, the area of the virtual three-dimensional model including the intersecting portion or the periphery of the intersecting portion may be made transparent or semi-transparent.
[0066] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a recording medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.
[0067] Although the present invention has been described above in conjunction with various embodiments, the present invention is not limited to these embodiments, and modifications can be made within the scope of the present invention, and part of the configuration or part of the processing of each embodiment can be combined with other embodiments.
[0068] In each of the above-described embodiments, when a stereo image is presented to the user, the process of the flowchart in each embodiment is performed appropriately for each of the left and right images.
[0069] In the above-described embodiments, the device 201 and the device 221 have been described as having the same configuration, but this is not limiting and the device may have different configurations. For example, the device 201 may not have a walk-through function, and the device 221 may have a walk-through function. [Explanation of symbols]
[0070] 211: Receiving unit 212: Virtual image generating unit 217: Information processing device
Claims
1. An information processing device for sharing a space in which a virtual three-dimensional model is presented with other users, comprising: An acquisition means for acquiring information on the real viewpoint of the other user; a generating means for generating a visual effect based on information of the real viewpoint of the other user when the virtual viewpoint of the other user differs from the real viewpoint of the other user due to the use of a walk-through function; 13. An information processing device comprising:
2. The generating means includes: The information processing apparatus according to claim 1 , wherein, as the visual effect, an image is generated at a position corresponding to the real viewpoint of the other user.
3. The generating means includes: Estimating an area in which the other user exists based on information of the real viewpoint of the other user; 3. The information processing apparatus according to claim 2, wherein an image obtained by processing an actual image of the estimated area is generated as the visual effect.
4. The generating means includes:
4. The information processing apparatus according to claim 3, wherein an image in which a color of the estimated area is changed to a different color is generated as the visual effect.
5. The generating means includes:
4. The information processing apparatus according to claim 3, wherein an image in which the estimated area is changed to a background image is generated as the visual effect.
6. The generating means includes:
6. The information processing apparatus according to claim 1, further comprising: a generating unit that generates an image that changes in accordance with a relationship between a position of a real viewpoint of the user and a position of a real viewpoint of the other user.
7. The generating means includes: An information processing device as described in any one of claims 3 to 5, characterized in that an image is generated in which the transparency of a background image to be superimposed on the estimated area is changed depending on the relationship between the position of the actual viewpoint of the own user and the position of the actual viewpoint of the other user.
8. The generating means includes: 2 . The information processing device according to claim 1 , wherein, as the visual effect, an image is generated in which the space or the virtual three-dimensional model is deformed based on information of the real viewpoint of the other user and information of the virtual viewpoint of the other user.
9. The generating means includes: The information processing device according to claim 8, characterized in that an image is generated by deforming the spatial or virtual three-dimensional model so that, when the other user is viewed from the real viewpoint of the own user, the position of the virtual viewpoint of the other user appears to coincide with the position of the real viewpoint of the other user.
10. The generating means includes: The information processing device according to claim 8 or 9, characterized in that an image is generated in which the side of the other user in the spatial or virtual three-dimensional model is deformed across a boundary set between the real viewpoint of the user and the real viewpoint of the other user.
11. The generating means includes: The information processing device according to claim 10, characterized in that the real viewpoint of the user and the real viewpoint of the other user are each projected onto a predetermined plane, and an image is generated in which the side of the other user is deformed across a Voronoi boundary calculated based on the projected points.
12. The generating means includes:
12. The information processing apparatus according to claim 1, wherein the visual effect is not generated when the virtual viewpoint of the other user and the real viewpoint of the other user coincide with each other.
13. The generating means: The information processing apparatus according to claim 2 , wherein an image of an avatar is generated at the position of the virtual viewpoint of the other user.
14. An information processing method for sharing a space in which a virtual three-dimensional model is presented with other users, comprising: An acquisition step of acquiring information on the real viewpoints of the other users; a generating step of generating a visual effect based on information of the real viewpoint of the other user when the virtual viewpoint of the other user differs from the real viewpoint of the other user due to the use of a walk-through function; 13. An information processing method comprising:
15. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 13.
