Information processing apparatus, information processing method, and program

JP2023113366A5Active Publication Date: 2024-12-04CANON KK
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2022015693
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2022-02-03
Publication Date
2024-12-04
Estimated Expiration
2042-02-03

AI Technical Summary

Technical Problem

In mixed reality (MR) and augmented reality (AR), the positional inconsistency between the real and virtual viewpoints occurs during the walkthrough function, leading to confusion among multiple users observing a virtual 3D model, hindering effective communication.

Method used

An information processing apparatus that acquires and corrects the relative positions and orientations of virtual and real viewpoints of multiple users, ensuring alignment and consistency through a system of devices with imaging, display, and processing units to synchronize and adjust virtual viewpoints based on real-time user interactions.

Benefits of technology

The solution effectively eliminates inconsistencies in the relationship between real and virtual viewpoints, enhancing user communication and understanding in MR and AR environments by maintaining synchronized views among multiple users.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

To eliminate a mismatch in the relationship between an actual viewpoint and a virtual viewpoint due to a movement of the virtual viewpoint.SOLUTION: An information processing apparatus acquires an attitude at an actual viewpoint position and an attitude at a virtual viewpoint position of a first user, and an attitude at an actual viewpoint position and an attitude at a virtual viewpoint position of a second user. The information processing apparatus calculates a relative position attitude between the attitude at the actual viewpoint position of the first user and the attitude at the actual viewpoint position of the second user, and corrects the attitude at the virtual viewpoint position of the first user based on the relative position attitude and the attitude at the virtual viewpoint position of the second user.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] The present invention relates to an information processing technology that provides virtual reality, mixed reality, augmented reality, and the like. [Background technology]

[0002] In virtual reality, mixed reality, and augmented reality, the virtual viewpoint that renders the virtual 3D (three-dimensional) model can be moved in response to user operation, allowing the user to observe various locations such as buildings represented by the virtual 3D model without moving themselves. This function of moving the virtual viewpoint through user operation is called a walk-through function. Note that virtual reality is often abbreviated as VR (Virtual Reality), mixed reality as MR (Mixed Reality), and augmented reality as AR (Augmented Reality).

[0003] In order to smoothly use the walk-through function, it is important to clarify the positional relationship between the virtual 3D model to be observed and the user so that the user can easily understand what they are looking at and from where. Furthermore, for multiple users to observe a 3D model or communicate with each other together, it is important that the users can understand their own positional relationship with other users without confusion. Patent Document 1 discloses a technology that includes a full-scale observation mode in which the user is immersed in a full-scale virtual 3D space and observes it, and a reduced-scale observation mode in which a reduced-scale CG image of the virtual 3D space is observed, and that presents a CG map with the user's position and orientation information added in the reduced-scale observation mode. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-79174 Summary of the Invention [Problem to be solved by the invention]

[0005] In mixed reality (MR) and augmented reality (AR), which display a virtual 3D model superimposed on real space, using the walk-through function can cause a mismatch between the real space and the virtual 3D model because the virtual viewpoint moves. For example, when multiple users are observing a 3D model, if a specific user uses the walk-through function, a mismatch occurs between the specific user's real viewpoint (the viewpoint of the actual head) as seen by other users and the position of the virtual viewpoint when the specific user is observing the 3D model. In other words, a mismatch occurs in the relationship between the specific user's real viewpoint and the virtual viewpoint moved by the walk-through as seen by other users. As a result, other users cannot understand from where a specific user is viewing the 3D model, making communication between users difficult.

[0006] Therefore, an object of the present invention is to make it possible to eliminate the inconsistency in the relationship between the real viewpoint and the virtual viewpoint caused by movement of the virtual viewpoint. [Means for solving the problem]

[0007] The information processing device of the present invention is characterized by having a first acquisition means for acquiring a real viewpoint position and orientation and a virtual viewpoint position and orientation of a first user, a second acquisition means for acquiring a real viewpoint position and orientation and a virtual viewpoint position and orientation of a second user, a calculation means for calculating a relative position and orientation between the real viewpoint position and orientation of the first user and the real viewpoint position and orientation of the second user, and a correction means for correcting the virtual viewpoint position and orientation of the first user based on the relative position and orientation and the virtual viewpoint position and orientation of the second user. [Effects of the Invention]

[0008] According to the present invention, it is possible to eliminate inconsistencies in the relationship between the real viewpoint and the virtual viewpoint caused by movement of the virtual viewpoint. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing apparatus according to a first embodiment. [Figure 2]FIG. 1 is a diagram used to explain the relationship between a virtual viewpoint and a real viewpoint in a walk-through. [Figure 3] FIG. 10 is a diagram showing an example of a mixed reality image seen from the viewpoint of the user himself / herself. [Figure 4] 4 is a flowchart showing the flow of information processing in the first embodiment. [Figure 5] FIG. 10 is a diagram showing a mixed reality image seen from the user's virtual viewpoint after correction. [Figure 6] FIG. 10 illustrates an example of the configuration of an information processing apparatus according to a second embodiment. [Figure 7] 10 is a flowchart showing the flow of information processing in the second embodiment. [Figure 8] FIG. 10 illustrates an example of the configuration of an information processing apparatus according to a third embodiment. [Figure 9] 10 is a flowchart showing the flow of information processing in the third embodiment. [Figure 10] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). Furthermore, the present invention may be configured by appropriately combining parts of each of the embodiments described below. In the following embodiments, the same components will be described with the same reference symbols.

[0011] First Embodiment FIG. 1 is a diagram showing an example of the configuration of a system including an information processing device 319 according to the first embodiment. The system of this embodiment is a system that can provide virtual reality (VR), mixed reality (MR), augmented reality (AR), and the like to a user. In this embodiment, an example is given in which mixed reality (MR) is provided to a user. The system of this embodiment also provides a walk-through function that allows the user to observe various places, such as buildings, represented by a virtual 3D model without moving, by moving a virtual viewpoint that renders a virtual three-dimensional (3D) model in response to a user operation.

[0012] Before explaining the configuration and operation of the system of this embodiment, we will use Figures 2 and 3 to explain the positional relationship between the viewpoint positions of each user and the 3D model in the mixed reality space when multiple users are observing a 3D model and one user uses walk-through.

[0013] Figure 2 is a diagram showing the positions of multiple users and the arrangement of 3D models in a mixed reality space, as well as the position of a virtual viewpoint of a specific user when the specific user performs a walkthrough in the mixed reality space. In this example, there are two users.

[0014] In FIG. 2, virtual models 105 and 106 are 3D models (three-dimensional objects) arranged in a mixed reality space. User 101 and user 102 are users who exist as entities in a real space. It is assumed that user 101 and user 102 are each wearing a head-mounted device (head-mounted display, hereinafter referred to as HMD 108). It is assumed that user 101 and user 102 are observing the mixed reality space in which virtual models 105 and virtual models 106 are arranged via the HMD 108 worn by each of them. It is also assumed that user 101 and user 102 each hold an operation device 107 for inputting various instructions. Then, user 101 and user 102 can give instructions such as moving a virtual viewpoint within the mixed reality space through operation input to the operation device 107. In this embodiment, instruction input via the operation device 107 includes instructions to perform a walkthrough and move a virtual viewpoint, and instructions to input a predetermined trigger operation, which will be described later.

[0015] In this example, it is assumed that the user 102 has performed a walkthrough, and thus the virtual viewpoint of the user 102 has moved to the position and orientation shown by the virtual viewpoint 104 in Fig. 2. That is, it is assumed that the virtual viewpoint 104 of the user 102 has moved to a position on the other side of the virtual models 105 and 106 and an intermediate position between them, and has a position and orientation such that the virtual models 105 and 106 are observed from behind. Also, in the mixed reality space at this time, an avatar 103 representing the user 102 at the position and orientation of the virtual viewpoint 104 is also arranged (displayed). In this example, the walkthrough has caused the virtual viewpoint 104 of the user 102 to be positioned (and oriented) on the far side of the intermediate position between the virtual models 105 and 106 and facing the front, and therefore the avatar 103 of the user 102 is an avatar facing the front.

[0016] 3 is a diagram schematically illustrating a mixed reality space observed from the viewpoint of a user 101. Note that the positional relationship with the user 102 and the positional relationship between the virtual models 105 and 106 in the mixed reality space correspond to the example in FIG. 2. That is, in this example, the actual body and real viewpoint (the viewpoint of the actual head) of the user 102 are present in front and to the left of the user 101, while the virtual viewpoint 104 of the user 102 is present on the other side of the virtual models 105 and 106 and in a position intermediate between them. Therefore, the actual body of the user 102 and the virtual model 105 are visible to the user 101 on the left, a part of the virtual model 106 is visible in the back right, and an avatar 103 corresponding to the walkthrough of the user 102 is visible behind the virtual models 105 and 106.

[0017] In the example shown in FIGS. 2 and 3 , there is a difference in how the mixed reality space seen by user 101 and the mixed reality space seen by user 102 appear. That is, while user 101 sees the mixed reality space shown in FIG. 3 , a mixed reality image from a virtual viewpoint 104 is displayed on user 102's HMD 108. This results in a significant difference in how the virtual models 105 and 106 seen by user 102 appear, compared to the virtual models 105 and 106 seen by user 101. This makes it difficult for user 101 to correctly understand the situation seen by user 102. That is, when multiple users are observing a mixed reality space and a specific user performs a walkthrough, it becomes difficult for the other users to correctly understand the relationship between their own virtual viewpoint and the virtual and real viewpoints of the specific user during the walkthrough. As a result, confusion is likely to occur in communication between users.

[0018] Therefore, the information processing device of this embodiment performs processing to correct the mismatch between the virtual viewpoint created by a specific user's walkthrough and the virtual viewpoints of other users. The information processing device of this embodiment corrects the positions and orientations of the virtual viewpoints of other users so that they match the position and orientation of the virtual viewpoint of the specific user who moved during the walkthrough.

[0019] In the system according to the present embodiment shown in FIG. 1, when multiple users observe a mixed reality space, each of the users uses a device including an information processing device 319, an operation unit 305, and a terminal 302. The devices 301 and 318 shown in FIG. 1 have the same configuration and similar processing functions, and each includes an information processing device 319, an operation unit 305, and a terminal 302. In the present embodiment, of the multiple users who are observing the mixed reality space together (who are provided with a mixed reality sensation together), a first user uses the device 301, and a second user different from the first user of the device 301 uses the device 318. In the system according to the present embodiment, the devices 301 and 318 cooperate with each other via a transmission unit 315 and a reception unit 314 implemented in each device, thereby enabling the multiple users to observe three-dimensional images (3D images) of the mixed reality space together. 1 assumes a case where there are two users and shows an example of a configuration in which two devices, device 301 and device 318, work together, but if there are three or more users, three or more devices can work together to provide mixed reality to those three or more users. In the following explanation, particularly when using the configuration of device 301 as an example, when it is necessary to distinguish between the users of device 301 and device 318, the first user of device 301 will be referred to as the "own user" and the second user of device 318 will be referred to as the "other user."

[0020] The terminal 302 includes an imaging unit 303 that captures images of the real space (real environment) around the user, and a display unit 304 that presents a mixed reality image to the user. In this embodiment, the terminal 302 is an HMD in which the imaging unit 303 and the display unit 304 are mounted in a head-mounted housing. However, the terminal may take any form, such as a tablet terminal or a smartphone terminal. The HMD exemplified in this embodiment is a video see-through type that superimposes a virtual 3D model on a real image captured by the imaging unit 303 and displays it on the display unit 304. However, it may also be a so-called optical see-through type. The imaging unit 303 includes a camera that captures moving images to be presented to the user and a camera that acquires images necessary for estimating the position and orientation of the terminal 302. Note that a single camera may have both the function of capturing moving images to be presented to the user and the function of acquiring images used for estimating the position and orientation. Depending on the method of acquiring the position and orientation and the method of superimposing and displaying 3D images, one or both of the camera capturing moving images to be presented to the user and the camera capturing images used for estimating the position and orientation may be omitted. In this embodiment, the images presented to the user are assumed to be images that can be viewed stereoscopically. Therefore, in order to realize the stereoscopic view, the terminal 302 of this embodiment is assumed to be equipped with a camera capturing images to be presented to the user and a display device presenting mixed reality images, each for the right eye and left eye. However, the configuration for realizing the stereoscopic view may be changed as appropriate depending on the form of the terminal 302, the method of acquiring the position and orientation of the terminal 302, the method of superimposing and displaying virtual 3D images, etc. Image information of the real space (real environment) captured by the imaging unit 303 is sent to the information processing device 319. The mixed reality space image displayed on the display unit 304 is generated by the information processing device 319 and input to the display unit 304.

[0021] The operation unit 305 is an operation device used by the user to input various instructions. Input to the operation unit 305 is realized by one or more of, for example, button input from a controller device, gesture input, voice input, etc. The operation unit 305 may be mounted on the terminal or configured as a separate device. The user can input various instructions via the operation unit 305. In this embodiment, an example of an operation input instructing movement of the user's virtual viewpoint is given. In this embodiment, the operation input for moving the user's virtual viewpoint includes an operation input for moving the user's virtual viewpoint using the walk-through function of the device 301 and an input of a predetermined trigger operation for instructing movement of the user's virtual viewpoint in accordance with the virtual viewpoint of another user. The predetermined trigger operation is assumed to be input when instructing movement of the user's virtual viewpoint on the device 301 to correspond to the virtual viewpoint of another user through a walk-through performed on the other user's device 318. In other words, a trigger operation that moves the user's virtual viewpoint to match the virtual viewpoint of another user is an operation that triggers the execution of a process to correct a mismatch between the virtual viewpoint of the other user resulting from a walkthrough and the user's virtual viewpoint. In this embodiment, since this trigger operation triggers the movement of the user's virtual viewpoint to match the virtual viewpoint of the other user resulting from a walkthrough, it will be referred to as a "movement trigger" hereinafter.

[0022] The information processing device 319 has a function to estimate the position and orientation of the terminal 302 and a function to generate an image in mixed reality space, as well as a function to share the real viewpoint position and orientation and the virtual viewpoint position and orientation with other users' devices 318, as will be described later. The information processing device 319 is configured to include an operation acquisition unit 306, a movement determination unit 307, an image acquisition unit 308, a real viewpoint acquisition unit 309, a virtual viewpoint generation unit 310, a synthesis unit 311, a correction unit 312, a relative relationship calculation unit 313, a receiving unit 314, and a transmitting unit 315. Note that in the example of FIG. 1, of all the components generally included in an information processing device 319 that provides a user with mixed reality, only functional units mainly related to information processing according to this embodiment are extracted and shown, and other components are not shown.

[0023] The image acquisition unit 308 acquires an image of the real environment captured by the imaging unit 303 of the terminal 302 . The real viewpoint acquisition unit 309 receives a captured image of the real environment from the image acquisition unit 308, and calculates, based on the captured image, the position and orientation of the real viewpoint of the user wearing the terminal 302. The real viewpoint acquisition unit 309 sends the calculated real viewpoint position and orientation information of the user to the virtual viewpoint generation unit 310.

[0024] When a user operation is performed using the operation unit 305, the operation acquisition unit 306 acquires operation information corresponding to the user operation. In this embodiment, when a command operation, such as moving the position and orientation of a virtual viewpoint, is input from the user via the operation unit 305, the operation acquisition unit 306 has a function of acquiring the command operation as operation information. Command operations for moving the position and orientation of a virtual viewpoint include a command operation for moving the user's own virtual viewpoint by walk-through, and a movement trigger operation instruction for moving the user's own virtual viewpoint to match the virtual viewpoint by another user's walk-through. The operation information acquired by the operation acquisition unit 306 is sent to the movement determination unit 307. The operation information is also sent to the virtual viewpoint generation unit 310, the correction unit 312, and the relative relationship calculation unit 313 via the movement determination unit 307.

[0025] The movement determination unit 307 determines whether the operation information acquired by the operation acquisition unit 306 is operation information instructing the user to move his / her virtual viewpoint. In this embodiment, the movement determination unit 307 determines whether the operation information is for moving the user's virtual viewpoint by, for example, walking through the device 301, or is operation information for a movement trigger that moves the user's virtual viewpoint to match the virtual viewpoint of another user's walkthrough. That is, the movement determination unit 307 is a trigger detection unit that detects a movement trigger, and when it detects a movement trigger, it outputs operation information with a flag indicating that the movement trigger has been input turned on. Hereinafter, the flag indicating that the movement trigger has been input will be referred to as a "movement flag." The operation information after the movement determination process by the movement determination unit 307 is sent to the virtual viewpoint generation unit 310, the correction unit 312, and the relative relationship calculation unit 313.

[0026] The virtual viewpoint generation unit 310 receives position and orientation information of the real viewpoint from the real viewpoint acquisition unit 309, and also receives operation information via the movement determination unit 307. The virtual viewpoint generation unit 310 generates the position and orientation of the virtual viewpoint of the user based on the real viewpoint position and orientation information and operation information of the viewpoint position input by the user. Note that when the device 301 executes a walkthrough or the like and moves the user's virtual viewpoint, the virtual viewpoint position and orientation generated by the virtual viewpoint generation unit 310 becomes the virtual viewpoint position and orientation resulting from the user's walkthrough or the like. The virtual viewpoint generation unit 310 transmits the generated virtual viewpoint position and orientation information of the user's own user and the real viewpoint position and orientation information of the user's own user received from the real viewpoint acquisition unit 309 to the transmission unit 315. The virtual viewpoint position and orientation information and real viewpoint position and orientation information of the user's own user are also transmitted to the relative relationship calculation unit 313.

[0027] The transmitting unit 315 transmits the virtual viewpoint position and orientation information and the real viewpoint position and orientation information from the virtual viewpoint generating unit 310 to a receiving unit 317 of a device 318 having the same configuration as the device 301 . The receiving unit 314 receives the real viewpoint position and orientation information and the virtual viewpoint position and orientation information of the other user that are acquired and generated in the same manner as described above by a device 318 having the same configuration as the device 301 and transmitted from the transmitting unit 316. The real viewpoint position and orientation information and the virtual viewpoint position and orientation information of the other user received by the receiving unit 314 are sent to the relative relationship calculation unit 313. As a result, the real viewpoint positions and virtual viewpoint positions and orientations of both users are shared between the device 301 and the user's device 318 .

[0028] When the movement flag of the operation information is on, the relative relationship calculation unit 313 calculates the relative relationship of the real viewpoint positions and orientations between the two users based on the real viewpoint position and orientation information of the own user and the real viewpoint position and orientation information of the other user received from the receiving unit 314. In this embodiment, the relative relationship between the real viewpoint position and orientation of the own user and the real viewpoint position and orientation of the other user is referred to as the "real viewpoint relative position and orientation." Then, the relative relationship calculation unit 313 sends the calculated real viewpoint relative position and orientation information, the virtual viewpoint position and orientation information of the other user, and the virtual viewpoint position and orientation information of the own user to the correction unit 312. Note that in this embodiment, an example has been given in which the relative relationship calculation unit 313 calculates the real viewpoint relative position and orientation when the movement flag is on, but the real viewpoint relative position and orientation may always be calculated regardless of whether the movement flag is on or off.

[0029] When the movement flag of the operation information is on, the correction unit 312 executes a process of correcting a mismatch between the virtual viewpoint of the user himself / herself and the real viewpoint relative position and orientation from the relative relationship calculation unit 313 and the virtual viewpoint position and orientation of the other user. In this embodiment, a process of correcting a mismatch between the virtual viewpoint generated by the other user's walk-through and the user himself / herself's virtual viewpoint is assumed. As will be described in detail later, the correction unit 312 corrects the virtual viewpoint position and orientation of the user himself / herself to match the virtual viewpoint generated by the other user's walk-through by adding the real viewpoint relative position and orientation between the user himself / herself and the other user to the virtual viewpoint position and orientation of the other user. Then, the correction unit 312 sends the virtual viewpoint position and orientation information of the user himself / herself after the correction process to the synthesis unit 311. Note that when the movement flag is off, the correction unit 312 does not execute the correction process described above, and the virtual viewpoint position and orientation information of the user himself / herself generated by the virtual viewpoint generation unit 310 is sent from the relative relationship calculation unit 313 to the synthesis unit 311 via the correction unit 312.

[0030] The synthesis unit 311 generates a 3D image based on the received virtual viewpoint position and orientation, and superimposes and synthesizes it on the captured image of the real environment acquired by the image acquisition unit 308, thereby generating a mixed reality image in which the virtual image and the real environment image are synthesized. Here, if the movement flag is on, the mixed reality image generated by the synthesis unit 311 will be an image that appears to match the virtual viewpoint position and orientation due to the walkthrough of the other user. On the other hand, if a walkthrough or the like is performed on the user's device 301 and the movement flag is off, the mixed reality image generated by the synthesis unit 311 will be an image that appears from the virtual viewpoint position and orientation according to the user's walkthrough or the like.

[0031] 4 is a flowchart showing the flow of information processing in the information processing device 319 of the device 301, from sharing the real viewpoint position and orientation and the virtual viewpoint position and orientation with the device 318 of another user as described above, to generating a mixed reality image depending on whether a movement trigger is detected. Here, only the processing performed in the information processing device 319 of the device 301 will be described, but similar information processing is also performed in the information processing device of the device 318.

[0032] In step S410, the real viewpoint acquisition unit 309 acquires the real viewpoint position and orientation of the user in world coordinates in real space. For example, a method for acquiring the real viewpoint position and orientation can be used, in which a marker placed in real space is photographed and the real viewpoint position and orientation are estimated from the arrangement of feature points of the marker in the captured image. Another method, such as Simultaneous Localization and Mapping (SLAM), can be used, in which natural feature points in a real image captured of real space are used to simultaneously estimate the user's position and create an environmental map. Alternatively, the real viewpoint position and orientation can be acquired using an external measurement device such as a motion capture device. The origin and orientation of the world coordinate system in real space are determined in advance using a marker or a specific real object. When using any of these real viewpoint position and orientation acquisition methods, using different methods between the device 301 and the device 318 may result in errors in the acquisition results. Therefore, the same method is used between the device 301 and the device 318.

[0033] Next, in step S420, the movement determination unit 307 determines whether or not a movement trigger has been detected in the operation information from the operation acquisition unit 306, the movement trigger instructing the user to move the virtual viewpoint of the user himself / herself so as to match the virtual viewpoint of another user through a walk-through on the device 318. If a movement trigger is detected, the movement determination unit 307 outputs operation information with a movement flag set to ON.

[0034] Next, in step S430, the virtual viewpoint generation unit 310 calculates the position and orientation of the virtual viewpoint of the user based on the operation information received via the movement determination unit 307 and the real viewpoint position and orientation information of the user based on the real viewpoint acquisition unit 309. At this time, the virtual viewpoint generation unit 310 calculates the amount of movement of the virtual viewpoint of the user based on the operation information received from the movement determination unit 307. Furthermore, the virtual viewpoint generation unit 310 calculates the virtual viewpoint position and orientation in world coordinates by adding the calculated amount of movement of the virtual viewpoint of the user to the real viewpoint position and orientation of the user based on the real viewpoint acquisition unit 309. For example, when a walkthrough or the like is performed on the device 301, the virtual viewpoint generation unit 310 calculates the amount of movement of the virtual viewpoint corresponding to the walkthrough or the like of the user, and adds this to the real viewpoint position and orientation to calculate the position and orientation of the virtual viewpoint moved by the walkthrough or the like. Any of various existing methods may be used to calculate the amount of movement of the virtual viewpoint from operation information, and a description of these methods will be omitted.

[0035] Furthermore, if the operation unit 305 or the actual viewpoint acquisition unit 309 can acquire the positions and orientations of the user's body parts such as hands, the positions and orientations of those body parts are acquired and the amount of viewpoint movement is also added to them. The positions and orientations of the user's body parts such as hands can be used to express body parts in an avatar whose display position moves in response to viewpoint movement.

[0036] Next, in step S440, the transmitting unit 315 transmits the real viewpoint position and orientation and the virtual viewpoint position and orientation of the user to the device 318, and the receiving unit 314 receives the real viewpoint position and orientation and the virtual viewpoint position and orientation of the other user that have been similarly acquired, calculated, and transmitted by the device 318. As a result, the real viewpoint position and orientation and the virtual viewpoint position and orientation of the other user are shared between the device 301 and the device 318. If both the device 301 and the device 318 have also acquired the positions and orientations of the user's body parts, the position and orientation information of the body parts is also transmitted and received between them and shared.

[0037] Next, in step S450, the relative relationship calculation unit 313 and the correction unit 312 check whether the movement flag of the operation information is on. If the movement flag is on, the processing of the information processing device 319 proceeds to step S460 and subsequent steps. On the other hand, if the movement flag is not on, the virtual viewpoint position and orientation information of the user himself / herself generated by the virtual viewpoint generation unit 310 is sent directly from the relative relationship calculation unit 313 to the synthesis unit 311 via the correction unit 312, and the processing of the information processing device 319 proceeds to step S480.

[0038] In step S460, the relative relationship calculation unit 313 calculates the relative position and orientation between the shared real viewpoint position and orientation of the other user and the real viewpoint position and orientation of the user himself / herself. In this embodiment, the relative relationship calculation unit 313 calculates the real viewpoint relative position and orientation based on the real viewpoint position and orientation information of the user himself / herself received via the virtual viewpoint generation unit 310 and the real viewpoint position and orientation information of the other user received by the receiving unit 314. If the positions and orientations of body parts are also shared between device 301 and device 318, relative relationship calculation unit 313 also calculates the relative positions and orientations of body parts between the own user and the other user.

[0039] Next, in step S470, the correction unit 312 performs a process to correct inconsistencies between the virtual viewpoints of the other user and the user himself / herself. In this embodiment, the correction unit 312 performs a correction process to move the position and orientation of the virtual viewpoint of the user himself / herself so that it matches the position and orientation of the virtual viewpoint of the other user obtained by walk-through on the device 318. Here, the correction unit 312 performs a correction process to move the virtual viewpoint of the user himself / herself to a coordinate system based on the virtual viewpoint of the other user, that is, to move the virtual viewpoint of the user himself / herself to the origin of the coordinate system based on the virtual viewpoint of the other user. That is, the correction unit 312 performs a correction process to move the virtual viewpoint of the user himself / herself in the virtual viewpoint coordinate system of the other user by adding the real viewpoint relative position and orientation calculated by the relative relationship calculation unit 313 to the virtual viewpoint position and orientation of the other user. More specifically, the correction unit 312 adds the real viewpoint relative position and orientation to a virtual viewpoint coordinate system whose origin is the virtual viewpoint position and orientation of the other user, and calculates the relative position and orientation of the virtual viewpoint of the user himself / herself in the virtual viewpoint coordinate system of the other user. In this case, the position and orientation of the virtual viewpoint of the user in question moves to a coordinate system with the other user's virtual viewpoint as its origin, while maintaining the relative position and orientation of the real viewpoints between the users. As a result, in terms of movement in the world coordinate system, the position and orientation of the real viewpoint of the other user and the virtual viewpoint overlap, and the position and orientation of the virtual viewpoint based on the world coordinate system moves by the amount of this movement.

[0040] Next, in step S480, the synthesis unit 311 generates a 3D image based on the user's virtual viewpoint position and orientation, and superimposes and synthesizes it on a captured image of the real environment to generate a mixed reality image in which the virtual image and the image of the real environment are synthesized. For example, if the movement flag is on, the mixed reality image is an image that appears corrected to match the virtual viewpoint position and orientation resulting from the other user's walkthrough. For example, if the movement flag is off and a walkthrough or the like is performed on the device 301, the mixed reality image is an image that appears from the virtual viewpoint position and orientation resulting from the user's walkthrough or the like. In this embodiment, the synthesis unit 311 also generates an avatar of the user based on the virtual viewpoint position and orientation and synthesizes it with the image of the real environment. As a result, the mixed reality image becomes an image that includes the avatar. The other user's avatar may, for example, represent only the head using information about the shared position and orientation of the other user's virtual viewpoint, or may also include body parts such as the torso and legs in a position inferred from the head position.

[0041] Fig. 5 is a diagram showing an example of how the mixed reality space appears as observed from the virtual viewpoint position and orientation of the user in question when another user performs a walkthrough and executes the information processing according to this embodiment shown in the flowchart of Fig. 4. Fig. 5 is a diagram corresponding to the example of Fig. 2, in which the user in question is user 101 and the other user is user 102.

[0042] As described above, the correction unit 312 performs a correction process in which the virtual viewpoint position and orientation calculated by adding the relative position and orientation of the real viewpoint between the user 101 (own user) and the user 102 (other user) to the virtual viewpoint position and orientation of the user 102 is set as the corrected virtual viewpoint position and orientation of the user 101. As a result, the virtual viewpoint position of the user 101 moves to the virtual viewpoint coordinate system of the user 102 while maintaining the relative positional relationship of the real viewpoints between the users 101 and 102. In the example of FIG. 2, the virtual viewpoint of the user 102 moves to the position and orientation of the virtual viewpoint 104 by walk-through, and as described above, the virtual viewpoint 104 of the user 102 is located behind the virtual models 105 and 106 and at an intermediate position between them. Therefore, the appearance from the virtual viewpoint of the user 101 after the correction process is updated from the appearance shown in FIG. 3 described above to the appearance shown in FIG. 5. That is, for each object placed in the mixed reality space, virtual model 105, which was seen on the left side by user 101 in Fig. 2, is now seen from behind and approximately in front in Fig. 5. In addition, the real viewpoint position of user 102 and the virtual viewpoint 104 overlap, and avatar 103 of user 102 becomes visible at the real position of user 102. In addition, virtual model 106, which was seen in the back right side in Fig. 3, is now seen on the left side in Fig. 5.

[0043] Furthermore, if there are multiple other users receiving mixed reality together with the user, the information processing device 319 acquires the real viewpoint position and orientation and the virtual viewpoint position and orientation for each of the other users from the information processing device of each of the other users. The information processing device 319 then calculates the relative real viewpoint position and orientation between the real viewpoint position and orientation of each of the other users and the real viewpoint position and orientation of the user, and corrects the virtual viewpoint position and orientation of the user based on the relative real viewpoint position and orientation calculated for, for example, any selected desired other user. When selecting a desired other user, the user may select the desired other user via the operation unit 305, or the other user may be automatically selected based on, for example, a positional relationship between the user and the user in real space. For example, if another user located close to the user performs a walkthrough in real space, it is expected that inconsistencies in the relationship with the virtual viewpoint due to the other user's walkthrough will be more noticeable. Therefore, it is conceivable to select another user within a predetermined distance from the user.

[0044] 4, an example was given in which the virtual viewpoint position of the user moves while maintaining the relative position and orientation of the real viewpoint with respect to the other user, but the virtual viewpoint position of the user may also be moved to the same position and orientation as the virtual viewpoint position of the other user. In this case, the appearance of the virtual space after the movement will be the same for the user and the other user.

[0045] In the above example, the virtual viewpoint position and orientation of the user himself / herself is corrected to match the virtual viewpoint position and orientation of the other user based on the relative real viewpoint position and orientation between the two users and the virtual viewpoint position and orientation of the other user. As another variation, in this embodiment, the user himself / herself may arbitrarily move the virtual viewpoint position and orientation of the other user while maintaining the relative real viewpoint position and orientation between the other user and the user himself / herself in the mixed reality space. In this case, the appearance of each object placed in the mixed reality space is determined by the appearance from the virtual viewpoint position and orientation of the user himself / herself. Furthermore, in this case, the position of an avatar displayed based on the virtual viewpoint position and orientation of the other user is updated as the virtual viewpoint position and orientation of the user himself / herself moves. In other words, when the position of the other user's avatar and the actual position of the other user are matched by moving the virtual viewpoint position and orientation of the user himself / herself through operation of the operation unit 305, the appearance from the user himself / herself will be as shown in the example of FIG. 5.

[0046] As described above, the information processing device of this embodiment calculates the position of the virtual viewpoint of the user in a virtual viewpoint coordinate system whose origin is the virtual viewpoint of another user during the walkthrough, based on the virtual viewpoint position and orientation information obtained by the walkthrough and the relative position and orientation information of the real viewpoint. That is, the information processing device of this embodiment corrects the position and orientation of the user's virtual viewpoint based on the position and orientation of the other user during the walkthrough and the position and orientation of the user's real viewpoint, thereby reducing the positional discrepancy between the virtual viewpoints of both users. This makes it possible to eliminate inconsistencies in the relationship between the virtual viewpoints and real viewpoints of users during a walkthrough in MR or AR, thereby reducing confusion in communication between users.

[0047] <Second embodiment> In the first embodiment described above, after the movement flag is turned on upon detection of a movement trigger operation input and the virtual viewpoint position is corrected, each device moves the virtual viewpoint of each user based on the individual user operation information until the next movement trigger is detected. In the second embodiment, an example will be described in which, when another user continues to move the virtual viewpoint by walk-through, the correction of the position and orientation of the user's virtual viewpoint is continued so as to match the movement of the position and orientation of the virtual viewpoint of the other user during the walk-through.

[0048] Fig. 6 is a diagram showing an example of the configuration of a system including an information processing device 319 according to the second embodiment. In the configuration of Fig. 6, the same functional units as those in Fig. 1 are given the same reference numerals, and their description will be omitted. Below, the configuration and processing different from the first embodiment will be described.

[0049] The information processing device 319 of the device 301 shown in Fig. 6 includes a continuation determination unit 601. In addition to the same function as the movement determination unit 307 described above, the continuation determination unit 601 also includes a function to determine whether or not predetermined movement continuation information indicating an input of a virtual viewpoint continuation movement operation is present, based on the operation information received from the operation acquisition unit 306. That is, the continuation determination unit 601 is a continuation detection unit that detects the predetermined movement continuation information, and when the predetermined movement continuation information is detected, it outputs operation information with a continuation flag set to on, for example. When the continuation flag is on, the relative relationship calculation unit 313 and the correction unit 312 continuously perform the above-described processing.

[0050] Fig. 7 is a flowchart showing the flow of information processing performed in the information processing device 319 of the device 301 of the second embodiment. As with the example of the flowchart in Fig. 4, the processing of the flowchart in Fig. 7 is also performed in the information processing devices of the device 301 and device 318. In the flowchart in Fig. 7, the same processing steps as in Fig. 4 are assigned the same reference numerals as in Fig. 4, and their description will be omitted.

[0051] In the flowchart of FIG. 7, after processing step S410, the processing of the information processing device 319 proceeds to step S710. In step S710, the continuation determination unit 601 performs processing to detect, from the operation information received from the operation acquisition unit 306, movement continuation information indicating that movement to another user's virtual viewpoint is to be continued. Furthermore, in step S720, the continuation determination unit 601 determines whether or not to continue moving the virtual viewpoint based on the detection result of the operation information. If the continuation determination unit 601 determines that the virtual viewpoint should be continued, the processing of the information processing device 319 proceeds to step S730 and subsequent steps. On the other hand, if the continuation determination unit 601 determines that the virtual viewpoint should not be continued, the processing of the information processing device 319 proceeds to step S420 and subsequent steps. The processing from step S420 onward is the same as that in the flowchart of FIG. 4, and therefore a description thereof will be omitted. However, in the second embodiment, the processing proceeds to step S460 after processing step S440.

[0052] In step S730, the relative relationship calculation unit 313 continuously calculates the viewpoint movement amount of the virtual viewpoint of the other user from the virtual viewpoint position and orientation information of the other user received by the receiving unit 314, and sends the calculated amount to the correction unit 312. In the next step S740, the correction unit 312 calculates the position and orientation of the user's virtual viewpoint by continuously adding the amount of viewpoint movement of the other user's virtual viewpoint calculated in step S730 to the position and orientation of the user's virtual viewpoint in world coordinates. This allows the position of the user's virtual viewpoint to move in accordance with the movement of the virtual viewpoint due to the other user's walk-through. After processing in step S740, the process proceeds to step S480, where an image of the mixed reality space is generated in the same manner as described above.

[0053] <Third embodiment> In the first and second embodiments described above, a movement trigger input via the operation unit 305 is used to determine whether or not to move the virtual viewpoint of the user to match that of the other user. In contrast, in the third embodiment, an example will be described in which the viewpoint positions and orientations of the user and the other user are used to determine whether or not to move the virtual viewpoint of the user to match that of the other user.

[0054] The information processing device of the third embodiment calculates the magnitude of the relative position and orientation between the virtual viewpoints of the user and another user (hereinafter referred to as the virtual viewpoint relative position and orientation). The information processing device of the third embodiment then acquires, as a movement trigger for moving the user's virtual viewpoint, that the magnitude of the virtual viewpoint relative position and orientation has exceeded a predetermined relative position and orientation threshold. That is, when the magnitude of the virtual viewpoint relative position and orientation of the user's and another user's virtual viewpoints exceeds the predetermined relative position and orientation threshold, the information processing device of the third embodiment executes processing to correct a mismatch between the virtual viewpoint of the other user created by the walk-through and the user's virtual viewpoint. In this embodiment, the comparison condition that the magnitude of the virtual viewpoint relative position and orientation exceeds the relative position and orientation threshold is assumed to be a case where the other user's virtual viewpoint has moved beyond a predetermined walk-through movement range. That is, in this embodiment, when the other user's virtual viewpoint has moved beyond a predetermined walk-through movement range, the user's virtual viewpoint is assumed to automatically follow the movement.

[0055] The information processing device of the third embodiment calculates the distance between the virtual viewpoints of the user and the other user as information representing the magnitude of the relative position and orientation of the virtual viewpoints between the user and the other user, and uses a predetermined distance threshold as the relative position and orientation threshold. When the information processing device receives, as a movement trigger, that the calculated distance has become larger than the predetermined distance threshold, it performs a calculation process of the relative position and orientation of the real viewpoints between the user and the other user, and further performs a correction process of the virtual viewpoint position and orientation of the user, as in the above-mentioned embodiments.

[0056] Fig. 8 is a diagram showing an example of the configuration of a system including an information processing device 319 according to the third embodiment. In the configuration of Fig. 8, the same functional units as those in Fig. 1 are given the same reference numerals, and their description will be omitted. Below, the configuration and processing different from the first embodiment will be described.

[0057] 8 does not include the movement determination unit 307 or the continuation determination unit 601 described above, and operation information acquired by the operation acquisition unit 306 is sent to the virtual viewpoint generation unit 310. The virtual viewpoint generation unit 310 generates the virtual viewpoint position and orientation of the own user in the same manner as described above, and sends the virtual viewpoint position and orientation information of the own user and the real viewpoint position and orientation information from the real viewpoint acquisition unit 309 to the transmission unit 315. In addition, in the case of the third embodiment, the virtual viewpoint position and orientation and real viewpoint position and orientation information of the own user output from the virtual viewpoint generation unit 310 are also sent to the relative relationship comparison unit 801. The relative relationship comparison unit 801 also receives as input the virtual viewpoint position and orientation information and real viewpoint position and orientation information of other users transmitted from the transmission unit 316 of the device 318 and received by the reception unit 314.

[0058] In addition to the functions of the relative relationship calculation unit 313 described above, the relative relationship comparison unit 801 has a function of comparing the magnitude of the relative position and orientation between the virtual viewpoints of the user and the other user with a threshold. Based on the virtual viewpoint position and orientation information of the user and the other user received by the receiving unit 314, the relative relationship comparison unit 801 calculates the distance between the virtual viewpoints as the magnitude of the relative position and orientation between the virtual viewpoints of the users. If the relative relationship comparison unit 801 determines that the calculated distance between the virtual viewpoints is greater than the distance threshold, it acquires this as a movement trigger. After detecting a movement trigger, the relative relationship comparison unit 801 performs a real viewpoint relative position and orientation calculation process, i.e., a process of calculating the relative position and orientation between the real viewpoint positions and orientations of the user and the other user, similar to the relative relationship calculation unit 313 in the above-described embodiment. Note that if the calculated distance between the virtual viewpoints is equal to or less than the distance threshold, the relative relationship comparison unit 801 sends the virtual viewpoint position and orientation information of the user generated by the virtual viewpoint generation unit 310 to the synthesis unit 311.

[0059] In the third embodiment, when the correction unit 312 receives real viewpoint relative position and orientation information from the relative relationship comparison unit 801, it executes processing to correct the mismatch between the virtual viewpoint obtained by the other user's walkthrough and the virtual viewpoint of the user himself / herself, in the same manner as described above. That is, the correction unit 312 adds the real viewpoint relative position and orientation between the user himself / herself and the other user to the virtual viewpoint position and orientation of the other user, thereby correcting the virtual viewpoint position and orientation of the user himself / herself so as to match the virtual viewpoint obtained by the other user's walkthrough or the like.

[0060] The synthesis unit 311 of the third embodiment generates a 3D image based on the received virtual viewpoint position and orientation, and superimposes and synthesizes the 3D image on the captured image of the real environment acquired by the image acquisition unit 308, thereby generating a mixed reality image in which the virtual image and the real environment image are synthesized. In the third embodiment, when the correction unit 312 executes a correction process, the mixed reality image generated by the synthesis unit 311 becomes an image that appears to match the virtual viewpoint position and orientation resulting from the walkthrough of the other user. On the other hand, when the relative relationship comparison unit 801 determines that the distance is equal to or less than the distance threshold, the mixed reality image generated by the synthesis unit 311 becomes an image that appears from the virtual viewpoint position and orientation resulting from the walkthrough of the own user.

[0061] Fig. 9 is a flowchart showing the flow of information processing performed in the information processing device 319 of the device 301 of the third embodiment. As with the example of the flowchart in Fig. 4, the processing of the flowchart in Fig. 9 is also performed in the information processing devices of the device 301 and device 318. In the flowchart in Fig. 9, the same processing steps as in Fig. 4 are assigned the same reference numerals as in Fig. 4, and their description will be omitted.

[0062] 9, after the processing of step S410, the process proceeds to the processing of step S430, and after the processing of step S440, the process proceeds to the processing of step S910. In step S910, the relative relationship comparison unit 801 calculates the relative position and orientation between the virtual viewpoint of the other user and the virtual viewpoint of the own user.

[0063] Next, in step S920, the relative relationship comparison unit 801 determines whether the magnitude of the virtual viewpoint relative position and orientation, which is the relative position and orientation between the virtual viewpoints of the own user and the other user, is greater than a predetermined relative position and orientation threshold set in advance. In this embodiment, the distance between the virtual viewpoints of the own user and the other user is calculated, and it is determined whether the calculated distance is greater than a predetermined distance threshold. If it is determined that the magnitude of the virtual viewpoint relative position and orientation between the own user and the other user is greater than the relative position and orientation threshold, the information processing device 319 proceeds to step S470; otherwise, it proceeds to step S480. The processes of steps S470 and S480 are the same as those described above, and therefore description thereof will be omitted.

[0064] Note that, in the determination process of step S920, it is determined whether the magnitude of the relative position and orientation of the virtual viewpoint is greater than a relative position and orientation threshold, but the present invention is not limited to this example. For example, the determination process of step S920 may be a process of determining whether the magnitude of the relative position and orientation of the virtual viewpoint is equal to or less than the relative position and orientation threshold, i.e., whether the distance between the virtual viewpoints of the user and another user is equal to or less than a predetermined distance threshold. In this case, when the virtual viewpoint of another user who has moved by walk-through and the user's virtual viewpoint become equal to or less than the distance threshold, that is, when the user's virtual viewpoint becomes close enough to be equal to or less than the distance threshold, it becomes possible to automatically align the user's viewpoint with the virtual viewpoint of the other user.

[0065] In the third embodiment, an example was given in which the presence or absence of a movement trigger was determined based on the distance between the virtual viewpoints of the own user and the other user, but the presence or absence of a movement trigger may also be determined based on the distance between the real viewpoints of the own user and the other user. In this case, the relative relationship comparison unit 801 calculates the distance between the real viewpoints of both users based on the real viewpoint position and orientation information of the own user and the real viewpoint position and orientation information of the other user, and acquires as a movement trigger the determination that the distance is greater than the distance threshold. Then, in response to the detection of the movement trigger, the relative relationship comparison unit 801 performs a calculation process for the relative position and orientation of the real viewpoints between the own user and the other user.

[0066] According to the third embodiment, it is possible to automatically reduce the positional discrepancy between the virtual viewpoints of both the user and another user without using a trigger operation input as in the first and second embodiments. As a result, in the third embodiment as well, in a walkthrough in MR or AR performed by multiple users, it is possible to eliminate inconsistencies in the relationship between the virtual viewpoints and real viewpoints of users during the walkthrough, and reduce confusion in communication between users.

[0067] The information processing device 319 of the third embodiment may include the movement determination unit 307 described in the first embodiment and the continuation determination unit 601 described in the second embodiment. In addition, in the first to third embodiments described above, an information processing device 319 is provided for each user, but the information processing device 319 may be a single device that communicates with each user's operation unit 305 and terminal 302 to perform the information processing described above.

[0068] <Hardware configuration example> FIG. 10 is a diagram showing an example of a hardware configuration in which the information processing device in each of the above-described embodiments can be implemented. In FIG. 10 , a CPU 1020 comprehensively controls each device connected via a bus 1010. The CPU 1020 reads and executes processing steps and programs stored in a read-only memory (ROM) 1030. An operating system (OS), information processing programs, device drivers, and the like according to the present embodiment are stored in the ROM 1030, temporarily stored in a random access memory (RAM) 1040, and appropriately executed by the CPU 1020. An input I / F 1050 inputs signals from external devices (e.g., the imaging unit 303 and operation unit 305 of the terminal 302, other devices 318, etc.) in a format processable by the information processing device of each embodiment. An output I / F 1060 outputs signals in a format processable by external devices (e.g., the display unit 304 of the terminal 302, other devices 318, etc.). These functional units are implemented by the CPU 1020 loading programs stored in the ROM 1030 into the RAM 1040 and executing processing according to the flowcharts of each embodiment described above. Furthermore, for example, when configuring hardware as an alternative to software processing using the CPU 1020, it is sufficient to configure a calculation unit or circuit that corresponds to the processing of each functional unit described here.

[0069] The functional units in the information processing apparatuses of the above-described embodiments may be implemented entirely by hardware, or partly by software. When implemented by software, the software is executed using the CPU 1020, ROM 1030, RAM 1040, etc. shown in FIG. 10.

[0070] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. The above-described embodiments are merely examples of specific implementations of the present invention, and should not be construed as limiting the technical scope of the present invention. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0071] 301, 318: device, 319: information processing device, 306: operation acquisition unit, 307: movement determination unit, 308: image acquisition unit, 309: real viewpoint acquisition unit, 310: virtual viewpoint generation unit, 311: synthesis unit, 312: correction unit, 313: relative relationship calculation unit, 314: reception unit, 315: transmission unit

Claims

1. a first acquisition means for acquiring a real viewpoint position and orientation and a virtual viewpoint position and orientation of a first user; A second acquisition means for acquiring a real viewpoint position and orientation and a virtual viewpoint position and orientation of a second user; a correction means for correcting a virtual viewpoint position and orientation of the first user based on a relative position and orientation between a real viewpoint position and orientation of the first user and a real viewpoint position and orientation of the second user and a virtual viewpoint position and orientation of the second user; 13. An information processing device comprising:

2. An information processing device as described in Claim 1, characterized in that it has a calculation means for calculating a relative position and attitude between the actual viewpoint position and attitude of the first user and the actual viewpoint position and attitude of the second user.

3. The information processing device described in Claim 1, characterized in that the first acquisition means generates a virtual viewpoint position and attitude of the first user based on viewpoint position operation information input by the first user and the actual viewpoint position and attitude of the first user.

4. A transmitting means for transmitting the real viewpoint position and the virtual viewpoint position and the virtual viewpoint position and the virtual viewpoint position of the first user to an information processing device of the second user, the second acquisition means includes a receiving means for receiving the real viewpoint position and orientation and the virtual viewpoint position and orientation of the second user transmitted from an information processing device of the second user; The information processing device according to claim 1, characterized in that the real viewpoint position and attitude and the virtual viewpoint position and attitude of the first user and the real viewpoint position and attitude and the virtual viewpoint position and attitude of the second user are shared between the information processing device of the first user and the information processing device of the second user.

5. The second acquisition means acquires the real viewpoint position and the virtual viewpoint position and the virtual viewpoint position for each of the second users, the calculation means calculates the relative position and orientation between an actual viewpoint position and orientation for each of the second users and an actual viewpoint position and orientation for the first user; The information processing apparatus according to claim 2 , wherein the correction means corrects the virtual viewpoint position and orientation of the first user based on the calculated relative position and orientation for any one of the second users and the virtual viewpoint position and orientation.

6. The information processing device according to claim 1, characterized in that the correction means corrects the virtual viewpoint of the first user to move into a coordinate system of the virtual viewpoint position and orientation of the second user.

7. The information processing device according to claim 6, characterized in that the correction means corrects the virtual viewpoint of the first user to move to the origin of a coordinate system of the virtual viewpoint position and orientation of the second user.

8. The information processing device described in Claim 7, characterized in that the correction means corrects the virtual viewpoint of the first user to move by adding the relative position and attitude to the virtual viewpoint position and attitude of the second user.

9. The information processing device described in Claim 8, characterized in that the correction means performs the correction to calculate the relative position and attitude of the first user's virtual viewpoint in the virtual viewpoint coordinate system of the second user by adding the relative position and attitude to a virtual viewpoint coordinate system whose origin is the virtual viewpoint position and attitude of the second user.

10. In response to detection of a predetermined trigger, the calculation means calculates the relative position and orientation, The information processing apparatus according to claim 2 , wherein the correction means corrects the virtual viewpoint position and orientation.

11. In response to detection of predetermined continuing information, the calculation means continues the process of calculating the relative position and orientation, 3. The information processing apparatus according to claim 2, wherein the correction means continues the process of correcting the virtual viewpoint position and orientation.

12. The calculation means continuously calculates a viewpoint movement amount of the virtual viewpoint of the second user based on the virtual viewpoint position and orientation of the second user, The information processing apparatus according to claim 11 , wherein the correction means continues the process of correcting the virtual viewpoint position and orientation by adding the continuously calculated viewpoint movement amount to the virtual viewpoint position and orientation of the first user.

13. The information processing device described in Claim 2, characterized in that the calculation means calculates the relative position and attitude between the real viewpoint position and attitude of the first user and the real viewpoint position and attitude of the second user in response to detection of a predetermined trigger based on the relative position and attitude between the virtual viewpoint position and attitude of the first user and the virtual viewpoint position and attitude of the second user.

14. The information processing device described in Claim 13, characterized in that the calculation means detects the specified trigger based on a comparison of the magnitude of the relative position and attitude between the virtual viewpoint position and attitude of the first user and the virtual viewpoint position and attitude of the second user with a specified threshold value.

15. The information processing device described in Claim 14, characterized in that the calculation means detects as the specified trigger a case in which the magnitude of the relative position and attitude between the virtual viewpoint position and attitude of the first user and the virtual viewpoint position and attitude of the second user is greater than the specified threshold value.

16. The information processing device described in Claim 14, characterized in that the calculation means detects as the specified trigger a case where the magnitude of the relative position and attitude between the virtual viewpoint position and attitude of the first user and the virtual viewpoint position and attitude of the second user is below the specified threshold.

17. The information processing device described in Claim 2, characterized in that the calculation means calculates the relative position and attitude between the real viewpoint position and attitude of the first user and the real viewpoint position and attitude of the second user in response to detection of a predetermined trigger based on comparison of the distance between the real viewpoint position of the first user and the real viewpoint position of the second user with a predetermined distance threshold.

18. The information processing device described in Claim 17, characterized in that the calculation means detects as the specified trigger a case in which the distance between the actual viewpoint position of the first user and the actual viewpoint position of the second user is greater than a specified distance threshold.

19. A first acquisition step of acquiring a real viewpoint position and a virtual viewpoint position and an orientation of a first user; a second acquisition step of acquiring an actual viewpoint position and a virtual viewpoint position and an orientation of a second user; a correcting step of correcting a virtual viewpoint position and orientation of the first user based on a relative position and orientation between a real viewpoint position and orientation of the first user and a real viewpoint position and orientation of the second user and a virtual viewpoint position and orientation of the second user; 13. An information processing method comprising:

20. A program for causing a computer to function as an information processing device described in any one of claims 1 to 18.