Virtual Reality Sharing Terminal

Abstract

To increase the freedom of choice of viewers' gaze in the MR space which the viewers share.SOLUTION: A virtual reality sharing system is composed of a VR server that creates a VR video by combining a plurality of camera-capturing-videos obtained by capturing the same real space from different perspectives, an AR server that draws an AR object (ARO), and a virtual reality sharing terminal (terminal) for viewers that receives the VR video and the ARO and superimposes and displays the ARO on the VR video, in which they are connected for communication via a network. When an input device of the terminal accepts a setting input operation of a viewer's perspective to take in the VR video, the terminal transmits data indicating the position and direction of the viewer's perspective to the VR server and the AR server, receives the VR video viewed from the viewer's perspective from the VR server, also receives ARO drawing data viewed from the viewer's perspective from the AR server, and superimposes and displays the AR object based on the drawing data on the VR video viewed from the viewer's perspective, on the terminal's display.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a virtual reality sharing terminal, and more particularly to a virtual reality sharing terminal that enables multiple people to share a mixed reality (MR) space composed of a real space and a virtual space (also referred to as an AR object: Argument Reality object).

Background Art

[0002] Patent Document 1 discloses a technique having "a first image acquisition step of acquiring a first stereo image based on a stereo video of a first stereo imaging unit worn by a first user and a virtual object image based on the position and orientation of the first stereo imaging unit; a second image acquisition step of acquiring a second stereo image based on a stereo video of a second stereo imaging unit installed in the space where the first user is located and a virtual object image based on the position and orientation of the second stereo imaging unit; a selection step of selecting an image according to an instruction of a second user from the first stereo image and the second stereo image; and a presentation step of presenting the selected image to the second user (abstract excerpt)".

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to Patent Document 1, in order for a viewer at a remote site to share the MR space, in addition to the HMD of the experiencer (operator), a fixed camera placed in the real space is used to select the viewer's line of sight from either the experiencer's HMD or the fixed camera, and the MR space can be shared from the selected line of sight.

[0005] However, in Patent Document 1, a viewer at a remote site can share the MR space only with a fixed line of sight of either the HMD of the experiencer or a fixed camera. Therefore, there is a problem that the viewer can share the MR space only with a limited line of sight.

[0006] The present invention has been made in view of the above problems, and aims to increase the degree of freedom of selection of the line of sight for a viewer to share the MR space.

Means for Solving the Problems

[0007] In order to solve the above problems, the present invention has the configuration described in the claims. As an example, it includes a display, an input device, a communicator, and a processor. When the processor receives a setting input operation of the viewer's perspective for capturing a VR video synthesized from a plurality of camera captured videos obtained by the input device capturing the same real space from different perspectives, it transmits data indicating the position and orientation of the viewer's perspective to a VR server that creates the VR video and an AR server that draws an AR object from the communicator. When receiving the VR video viewed from the viewer's perspective from the VR server and the drawing data of the AR object viewed from the viewer's perspective from the AR server via the communicator, it superimposes the AR object based on the drawing data on the VR video viewed from the viewer's perspective and displays it on the display. The plurality of camera captured videos include a first camera captured video and a second camera captured video. The first camera captured video includes metadata indicating the relative position of a real object that is a display trigger of the AR object as viewed from a first perspective. The second camera captured video includes metadata indicating the relative position of the real object as viewed from a second perspective. The processor receives, via the communicator, the VR video with the relative position viewed from the first perspective and the relative position viewed from the second perspective added from the VR server, compares the position of the real object captured in the VR video viewed from the viewer's perspective with the position of the AR object viewed from the viewer's perspective, and when the real object is closer than the AR object, at least a part of the AR object is hidden and displayed by the real object, and when the real object is farther than the AR object, at least a part of the real object is hidden by the AR object and displayed on the display. This is the feature.

Effect of the Invention

[0008] According to the present invention, the degree of freedom of choice of the line of sight for viewers to share the MR space can be increased. For purposes, configurations, and effects other than those described above, they will be clarified in the following embodiments.

Brief Explanation of Drawings

[0009]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the figures, the same components and steps are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] Adding an AR object created by CG (Computer Graphics) to the background image of the real space captured by a camera is used in contents such as games, telemedicine, and maintenance work. To add an AR object, an image of a real object called an AR trigger or marker is captured by a camera simultaneously with the background image, and the AR object associated with the AR trigger or marker is synthesized into the background image.

[0012] Particularly in mixed reality (MR), using a head-mounted display (HMD) in which a camera and a display are integrated, an AR object drawn according to the position and orientation of the HMD is overlaid on the background image captured by the camera equipped on the HMD or the background image seen through the HMD, and is displayed on the HMD as an image of the virtual space.

[0013] In this embodiment, among the images of the virtual space, a virtual reality (VR) that continuously displays images of an ultra-wide area (for example, 360°) space in accordance with the movement of a person wearing an HMD provides a pseudo-experience by the images, and the pseudo-experience is shared by a plurality of people.

[0014] [First Embodiment] (System Configuration and HMD) FIG. 1 is a schematic diagram of a virtual reality sharing system 100 according to the first embodiment.

[0015] The virtual reality sharing system 100 is a system that shares a virtual space including the MR space 6 among the experiencer 11 who has the operation right of the AR object 4b, the participants 12a, 12b, 12c, 12d, 12e who are in the same real space as the experiencer 11 and view the AR object 4b, and the viewers 13a, 13b, 13c who are in a real space (remote site) different from the experiencer 11 and the participants 12a to 12e. It is characterized in that not only the experiencer 11 and the participants 12a to 12e in the same real space but also the viewers 13a to 13c at the remote site can share and experience the virtual reality including the MR space 6.

[0016] In the MR space 6, an AR object 4b linked to the real object 4a as an AR marker is superimposed and displayed on the real object 4a. In FIG. 1, the real object 4a is a non-operating device of a new product, and the AR object 4b is an operating means. And it is an example in which the experiencer 11 is demonstrating and explaining the operation method of the new product with the AR object 4b.

[0017] Each of the experiencer HMD 1 worn by the experiencer 11 (hereinafter may be abbreviated as "HMD 1") and the participant HMDs 2a to 2e worn by each of the participants 12a to 12e (hereinafter may be abbreviated as "HMDs 2a to 2e") transmits and receives network signals 10a, 10b, 10c, 10d, 10e, 10f, 10g, 10h, 10i to and from access points 5a to 5c installed in the MR space 6. The experiencer HMD 1 and the participant HMDs 2a to 2e are virtual reality sharing terminals.

[0018] The viewer information terminal 3a operated by the viewer 13a (hereinafter may be abbreviated as "information terminal 3a") and the viewer HMDs 3b, 3c worn by each of the viewers 13b, 13c (hereinafter may be abbreviated as "HMDs 3b, 3c") transmit and receive network signals 10j, 10k, 10l, 10m, 10n to and from access points 5d, 5e. The viewer information terminal 3a and the viewers 13b, 13c are also virtual reality sharing terminals.

[0019] The access points 5a to 5c are connected to the network 7 outside the MR space 6 and serve as an intermediary medium for communication between the VR server 8, the AR server 9 placed on the network 7, and the HMDS 1, 2a to 2e. Also, on the network 7, access points 5d, 5e of a remote site are placed and serve as an intermediary medium for communication between the VR server 8, the AR server 9, the HMDS 3b, 3c for viewers, and the information terminal 3a.

[0020] Since the AR object 4b is three-dimensionally displayed on the HMDS 1, 2a to 2e, 3b, 3c, a realistic display reflecting the positional relationship such as the front and back between the real object 4a and the AR object 4b can be realized.

[0021] Fig. 2A is an external view of the HMD 1 for an experiencer and the HMDS 2a to 2e for participants, which are virtual reality sharing terminals, and Fig. 2B is an external view of the HMDS 3b, 3c for viewers, which are virtual reality sharing terminals.

[0022] First, the configuration common to the HMD 1 for an experiencer, the HMDS 2a to 2e for participants, and the HMDS 3b, 3c for viewers will be described. Each of the HMD 1 for an experiencer, the HMDS 2a to 2e for participants, and the HMDS 3b, 3c for viewers includes a left projector 22a, a right projector 22b, a screen 23, a nose pad 24, a processor 25, a speaker 26, a microphone 27, a left arm 28a, a right arm 28b, and a bridge 28c. The left arm 28a, the right arm 28b, and the bridge 28c are collectively referred to as the frame housing 28. The wearer wears the HMD on his / her face with the frame housing 28 and the nose pad 24.

[0023] Also, the HMD 1 for an experiencer and the HMDS 2a to 2e for participants further include a camera 21 and a distance measuring camera 29. The camera 21 is attached so as to photograph the background image, which is the real space in front of the line of sight of the HMD wearer, and the distance measuring camera 29 measures the distance to the real object 4a captured as part of the background image.

[0024] For the HMD1 for the experiencer and the HMDs 2a to 2e for the participants, the HMD wearer views the background image of the front real space through the screen 23 or views the background image captured by the mounted camera 21 projected onto the screen 23. Also, the left projector 22a and the right projector 22b project the AR object 4b onto the screen 23 as an image to be confirmed with the left eye and an image to be confirmed with the right eye, and display it stereoscopically as if the AR object 4b is at a predetermined distance in the real space. For the HMDs 3b and 3c for the viewers, the VR image described later and the AR object 4b are superimposed and displayed.

[0025] In the display on the HMD, based on the front-back relationship of the distances between the real object 4a and the AR object 4b, when a part of the real object 4a is in front of a part of the AR object 4b, occlusion processing is performed to process the drawing data of the AR object so that a part of the AR object is hidden behind a part of the real object, and a video of the MR space with enhanced realism is displayed.

[0026] The processor 25, the camera 21, the speaker 26, and the microphone 27 are arranged in the frame housing 28. Note that the arrangement location does not have to be as shown in FIGS. 2A and 2B.

[0027] (Block diagram of the three-dimensional virtual reality display device) FIG. 3 is a block diagram of the HMD1 for the experiencer. In FIG. 3, the processor 25 is the part surrounded by the dashed line, and each of the camera 21, the distance measuring camera 29, the left projector 22a, the right projector 22b, the screen 23, the speaker 26, the microphone 27, the azimuth sensor 250, the gyro sensor 251, and the acceleration sensor 252 is connected to the processor 25.

[0028] The processor 25 includes a CPU 254, a RAM 255, a video RAM 256, a FROM 257, a feature extraction processor 260, a distance calculation processor 261, an azimuth sensor 250, a gyro sensor 251, an acceleration sensor 252, and an internal bus 262, and each element is connected to each other via the internal bus 262. Note that the azimuth sensor 250, the gyro sensor 251, and the acceleration sensor 252 may be configured not to be included in the processor 25.

[0029] Each of the left projector 22a and the right projector 22b projects the left-eye image and the right-eye image onto the screen 23 independently to perform a three-dimensional display of the video. As another aspect, another display device capable of three-dimensional display such as an optical system using a holographic lens may be used.

[0030] The wireless communicator 253 selects an appropriate process from several communication processes such as mobile communication of 4G, 5G, etc. and wireless LAN, etc., and connects the HMD 1 to the network 7 via any one of the access points 5a, 5b, 5c.

[0031] The FROM 257 includes a basic program 258 and an MR experience program 259 as processing programs. These processing programs are developed and executed by the CPU 254 in the RAM 255. Further, the FROM 257 stores data necessary for executing the processing programs. The FROM 257 may be one memory medium as illustrated, or may be composed of a plurality of memory media. Furthermore, it may be a non-volatile memory medium other than Flash ROM.

[0032] Also, the video data sent to the left projector 22a and the right projector 22b is stored in the video RAM 256 by the CPU 254, and the CPU 254 reads it from the video RAM 256 and projects it from the left projector 22a and the right projector 22b onto the screen 23.

[0033] The feature extraction processor 260 extracts the outline (edges) of real objects from the background video captured by the camera 21, and performs processing using inflection points and vertices of the outline as feature points. Distance data obtained by the distance calculation processor 261 is combined with these feature points. The set of feature point data constituting the outline is associated with the background video.

[0034] The azimuth sensor 250, gyro sensor 251, and acceleration sensor 252 are used for tracing the position of the HMD 1 and the shooting direction of the camera (equal to the line of sight of the experiencer 11). Using access points 5a, 5b, and 5c with known arrangement positions described later, the position obtained is combined with the up-down and left-right line-of-sight directions obtained by the azimuth sensor 250 and acceleration sensor 252, and the movement of the HMD 1 is traced by the gyro sensor 251 or the like to detect changes in position and direction due to the movement of the wearer (experiencer 11) of the HMD 1.

[0035] Figure 4 is a block diagram of the participant HMDs 2a to 2e. In Figure 4, the basic program 258 and the MR participation program 263 are stored in the FROM 257. The HMDs 2a to 2e are the same as the HMD 1 in terms of hardware, and are different in that they store the MR participation program 263, which is an application program, instead of the MR experience program 259 of the HMD 1.

[0036] Figure 5 is a block diagram of the viewer HMDs 3b and 3c. In Figure 5, the camera 21 and the distance measurement camera 29 are omitted, and the basic program 265 and the MR viewing application 266 are stored in the FROM 264. Also, instead of the left projector 22a, right projector 22b, and screen 23, an immersive (non-transmissive / sheltering type) display 30 is provided.

[0037] Figure 6 is a block diagram of the VR server 8. In Figure 6, the VR server 8 includes a network interface (network IF) 81 such as a wired LAN, a CPU 82, a RAM 83, a storage 84, and an internal bus 87, and each component is connected to each other via the internal bus 87.

[0038] In addition to the Flash ROM, the storage 84 may be composed of a hard disk drive or the like. The storage 84 includes a VR creation program 85 as a processing program, which is expanded and executed by the CPU 82 in the RAM 83. Further, the storage 84 includes VR data 86 and stores data necessary for executing the processing program.

[0039] FIG. 7 is a block diagram of the AR server 9. In FIG. 7, the AR server 9 includes a network IF 91 such as a wired LAN, a CPU 92, a RAM 93, a storage 94, and an internal bus 97, and each component is connected to each other via the internal bus 97.

[0040] In addition to the Flash ROM, the storage 94 may be composed of a hard disk drive or the like. The storage 94 includes an AR drawing program 95 as a processing program, which is expanded and executed by the CPU 92 in the RAM 93. Further, the storage 94 includes AR data 96 and stores data necessary for executing the processing program. Also, as the AR data 96, AR objects created in the past may be archived and promoted for use by the experiencer during the execution of the processing program.

[0041] (Generation of VR video) FIG. 8 is a diagram for explaining an example of a method for detecting the positions of the HMDS 1, 2a to 2e worn by the experiencer 11 and the participants 12a to 12e. In FIG. 8, in order to detect the positions of the HMDS 1, 2a to 2e, access points 5a to 5c installed in the MR space 6 and having known installation positions are used. Since the position calculation methods of the HMDS 1, 2a to 2e are the same, the HMD 1 will be taken as an example for explanation below.

[0042] When the HMD1 is at the position of point P, it receives network signals 10g - i from access points 5a - 5c, and calculates the distances to the access points 5a - 5c from the intensities thereof. In FIG. 8, the distance to access point 5a is r1, the distance to access point 5b is r2, and the distance to access point 5c is r3. One point on the surface where the sphere with a distance of r1 from access point 5a, the sphere with a distance of r2 from access point 5b, and the sphere with a distance of r3 from access point 5c intersect is the point P, and the existence range within the MR space 6 is narrowed down. In addition to the information on these received intensities, the position of point P can be specified based on the height of the head on which the HMD1 is mounted, the image visible from the camera 21, and the information from various sensors.

[0043] When the HMD1 is at point P, the direction in which the HMD1 faces, that is, the line-of-sight direction of the experiencer 11, is determined by the azimuth sensor 250 for the left - right direction and by the acceleration sensor 252 for the up - down direction.

[0044] The experiencer 11 moves the position and moves the head to change the line of sight during the MR experience. These changes are traced by the acceleration sensor 252, the gyro sensor 251, etc. When it is detected that the HMD1 is in a stationary state after the movement, position detection etc. using the access points 5a - 5c may be performed again to reset the cumulative error in the tracing by the sensors.

[0045] The method for calculating the position and direction of the HMD1 is not limited to the above method. For example, instead of the access points 5a - 5c, video markers are used, and in addition to the equipped camera 21, a distance - measuring camera 29 is used in combination. A method of capturing the video marker with the camera 21 and obtaining the distance to the video marker with the distance - measuring camera 29, a method of calculating the movement from the change of the video marker captured by the camera 21, etc. may be used.

[0046] FIG. 9 is a diagram for explaining a method of generating a VR video.

[0047] The VR video is generated by pasting the videos captured by the cameras of the HMD 1 for the experiencer and the HMDs 2a to 2e for the participants onto the upper hemisphere with a radius R. Among the HMD 1 for the experiencer and the HMDs 2a to 2e for the participants, when any HMD located at point E is shooting towards point PE on the upper hemisphere, the camera-captured video is pasted onto the area centered at point PE. Similarly, when the HMD located at point F is shooting towards point PF on the upper hemisphere, the camera-captured video is pasted onto the area centered at point PF. In the overlapping part of the two areas, the areas are joined at the boundaries of the objects shown in the camera-captured video to form a continuous VR video. This process is performed for all of the HMD 1 for the experiencer and the HMDs 2a to 2e for the participants. As a result, even if it is not the entire upper hemisphere, a VR video with the video pasted onto the area where the experiencer 11 and the participants 12a to 12e have their lines of sight can be obtained.

[0048] Note that when joining the two overlapping areas, the position and direction of the HMD are finely adjusted so that the sizes and positions of the objects at the boundaries match, and the correction result is used as the position and direction data of the HMD, and the accuracy of the position and direction of the HMD can be improved.

[0049] Figure 10 is a diagram for explaining the normalization of the VR video.

[0050] Figure 10 shows the case of shooting from point H to point K and the case of shooting from point G to point K. In the two cases, since the shooting angle of view of the camera 21 is the same, the shooting area is narrower when shooting from the closer G point. Conversely, for the same subject, it appears larger in the shooting screen. Let the distance between G and K be d2 and the distance between H and K be d1. When pasting the video P2 captured from point G onto the area centered at point K, a VR video for the viewer is generated by converting it as if it were a video captured from point H on the upper hemisphere. The ratio of the two shooting ranges is d2 / d1, and the video P2 captured from point G is normalized by this ratio to obtain a normalized video. The display area of the normalized video P2 is smaller than the area P1 of the video actually captured from point H.

[0051] (MR Experience Program) FIG. 11 is a flowchart of the MR experience program 259.

[0052] The HMD 1 is activated and logs in to the user management process for sharing the MR space 6 (S101). Although the user management process is not shown in the system configuration of FIG. 1, a server may be provided independently, or the VR server 8 or the AR server 9 may be provided with that function. When logging in to the user management process, the user is registered as the experiencer 11 in the VR server 8 or the AR server 9.

[0053] The camera 21 and the distance measurement camera 29 of the HMD 1 start shooting (S102). The camera shooting video includes distance data to real objects. The camera shooting may, for example, capture a video at 30 fps (frames per second), and the subsequent steps may be executed in synchronization with the camera shooting cycle.

[0054] The HMD 1 detects the position of the HMD 1 in the MR space 6 (S103).

[0055] The HMD 1 detects the shooting direction of the camera 21 of the HMD 1 (S104). The position information of the HMD 1 and the detected shooting direction become metadata of the camera shooting video.

[0056] From S105 to S111, it is divided into two processes and processed in parallel. S105 to S110 are processes related to the processing with the AR server 9, and S111 to S113 are processes related to the processing with the VR server 8.

[0057] In S105, the data of the position and shooting direction of the HMD 1 are transmitted from the HMD 1 to the AR server 9.

[0058] The HMD 1 selects, for example, from a list stored in the AR server 9 the AR objects to be used in the MR experience and generates them as data to be displayed (S106).

[0059] The HMD1 receives the rendering data of the selected AR object and displays it on the screen 23 of the HMD1 (S107).

[0060] When the user 11 performs an operation on the displayed AR object 4b (S108), parameters such as the arrangement position, size, and orientation in the MR space 6 indicating the content of the performed operation are set. The parameters such as the arrangement position and direction are relative values based on the position and orientation of the HMD1. The HMD1 transmits the set parameters to the AR server 9, and the AR server 9 transmits the rendering data of the AR object 4b with the parameters reflected to the HMD1.

[0061] The HMD1 receives and displays the rendering data of the AR object 4b with the set parameters reflected (S109). When displaying, based on the distance relationship between the real object 4a and the AR object 4b, when a part of the real object 4a is in front of a part of the AR object 4b, the HMD1 performs a process of processing the rendering data of the AR object 4b so that a part of the AR object 4b is hidden behind a part of the real object 4a and appears that way.

[0062] The plurality of AR objects may be processed in one step, or a limited number of AR objects may be processed in one step, and the processing of all the AR objects may be performed over a plurality of cycles of the camera shooting period. Once an AR object is generated, it remains in the MR space 6 unless the user 11 cancels it. When the user 11 moves and the position and orientation change, the data of the rendered AR object received in S109 includes all of the already generated AR objects. It is determined whether the period (Time) of camera shooting arrives in S110. If it has not arrived (S110: No), the process returns to S106, and if it has arrived (S110: Yes), it is determined whether to end the program (S114).

[0063] In parallel with S105 to S110, the HMD1 transmits data on the position and shooting direction of the HMD1 (S111) and camera shooting video data to the VR server 8 (S112). The camera shooting video data is distance data between the background video shot by the camera 21 and the real object 4a measured by the distance measuring camera 29.

[0064] The HMD1 determines the camera shooting period (S113), waits for the arrival (S113: Yes), checks for the end of the program (S114), and if it has not ended (S114: No), continues the steps from S103 in the next camera period. If it ends (S114: Yes), a series of processes ends.

[0065] FIG. 12 is a flowchart of the MR participation program 263. The process of FIG. 12 is executed in the participant HMDs 2a to 2e. Hereinafter, the HMD2a will be taken as an example for explanation.

[0066] The participant 12a wearing the participant HMD2a logs in to the user management process for MR experience sharing (S121). When logging in to the user management process, the participant is registered with the VR server 8 and the AR server 9.

[0067] The HMD2a starts camera shooting with the camera 21 mounted on the HMD2a (S122). The subsequent steps may be performed in synchronization with the camera shooting period.

[0068] The HMD2a detects the position of the HMD2a within the MR space 6 (S123).

[0069] Also, the HMD2a detects the camera shooting direction (the participant's line of sight) of the HMD2a (S124) and associates it with the camera shooting video.

[0070] S125 to S127 are processes related to the processing with the AR server 9, and S128 to S129 are processes related to the processing with the VR server 8.

[0071] The HMD2a transmits the position and orientation data of the HMD2a to the AR server 9 (S125), receives the rendering data of the AR object generated by the user from the AR server 9 (S126), and displays it on the screen 23 of the HMD2a (S127).

[0072] The rendering data of the received AR object 4b is data rendered according to the participant's HMD2a based on the position and orientation data of the HMD2a transmitted in S125.

[0073] At the time of display, based on the distance relationship between the real object 4a and the AR object 4b, when a part of the real object 4a is in front of a part of the AR object 4b, the HMD2a performs a process of processing the rendering data of the AR object 4b so that a part of the AR object 4b appears hidden behind a part of the real object 4a.

[0074] FIG. 13 is a diagram for supplementary explanation regarding the position of the AR object 4b.

[0075] The user 11 is at the position T, and an AR object 4b is generated at the position U. The magnitude of the vector Vtu from point T to point U is the distance between the user 11 and the AR object 4b. At this time, assume that the participant 12a is at the position S and observes the AR object 4b. If the vector between the position T of the user 11 and the position S of the participant 12a is Vts, then the vector Vsu between the position S of the participant 12a and the position U of the AR object 4b is given by (vector Vtu - vector Vts). Therefore, the distance between the participant 12a and the AR object 4b is obtained by the magnitude of the vector Vsu, and in the display of the participant's HMD2a, the distance between the AR object 4b obtained by the magnitude of the vector Vsu and the distance to the real object obtained by the ranging camera 29 is evaluated for the front-back relationship.

[0076] In parallel with S125 to S127, the HMD2a transmits the position and shooting direction data of the HMD2a (S128) and the camera shooting video data to the VR server 8 (S129).

[0077] HMD2a determines the camera shooting cycle (S130), waits for arrival (S130: Yes), checks for program termination (S131), and if not terminated (S131: No), continues the steps from S123 in the next camera cycle. When the program is terminated (S131: Yes), a series of processes ends.

[0078] (MR viewing application) Figure 14 is a flowchart of the MR viewing application 266. The processes in Figure 14 are executed on the viewer information terminal 3a, the viewers' HMDs 3b, 3c. Hereinafter, HMD3b will be taken as an example for explanation.

[0079] The viewer 13b wearing the viewer's HMD3b logs in to the user management process for MR experience sharing (S141). When logging in to the user management process, the viewer is registered as a viewer in the VR server 8 and the AR server 9.

[0080] HMD3b receives the rendering data of the AR object 4b (initial data of the AR object 4b) from the AR server 9. In parallel, HMD3b receives the VR video data (initial data of VR) from the VR server 8 (S143).

[0081] HMD3b displays the AR object 4b and the VR video on the display 30 of HMD3b (S144).

[0082] The AR object 4b and the VR video in these steps are displayed, for example, with HMD3b as the default starting point, for example, on the upper hemisphere surface, at the position and direction facing the experiencer 11.

[0083] The VR video data includes, for each real object 4a, the metadata of the positions and directions of the photographed HMDs 1, 2a~2e and the distance data to the real object 4a. The VR server 8 calculates the distance to the real object 4a from the relationship between the positions of the photographed HMDs 1, 2a~2e and the viewpoint position of the viewer's viewpoint of the viewer's HMD3b (the viewpoint position is obtained by detecting the movement of the viewer's head).

[0084] Then, the AR server 9 corrects the distance to the AR object 4b based on the relationship among the position of the AR object 4b, the real object 4a, and the viewpoint position of the viewer's HMD 3b. At this time, when a part of the real object 4a is in front of a part of the AR object 4b, the AR server 9 performs processing on the drawing data of the AR object 4b so that a part of the AR object 4b is hidden by a part of the real object 4a and is visible.

[0085] Note that the default position and direction of the HMD 3b may be changed to the position and direction set by the viewer 13b, and when starting the MR viewing application, the changed position and direction may be used as the starting point.

[0086] While viewing the VR video, the viewer 13b moves the head to move the viewpoint and line of sight. When the HMD 3b detects that the head has been rotated to the left, the VR server 8 relatively shifts the VR video to the right. When the HMD 3b detects that the head has been moved forward, the HMD 3b enlarges and displays the VR video. Thereby, an MR experience as if the viewer 13b has approached can be produced.

[0087] The HMD 3b detects the movement of the viewer 13b's head from an azimuth sensor 250, a gyro sensor 251, an acceleration sensor 252, etc. The detected movement information is transmitted to the AR server 9 (S146) and also transmitted to the VR server 8 (S148).

[0088] Based on the transmitted movement information, the HMD 3b receives updated data of the AR object 4b as seen from the position and direction of the viewpoint of the viewer 13b's HMD 3b (S147). On the other hand, the HMD 3b receives updated data of the VR video as seen from the position and direction of the viewpoint of the viewer 13b's HMD 3b from the VR server 8 (S149).

[0089] Similar to S144, the HMD 3b superimposes and displays the AR object 4b and the VR video in consideration of the distance relationship between the real object 4a and the AR object 4b (S150). Thereby, the viewer 13b performs viewing of the MR experience.

[0090] Determine the display period from the update unit (frame rate) of the image frame (S151), wait for arrival (S151: Yes), check for the end of the MR viewing application (S152), and if it has not ended (S152: No), continue the steps from S145 in the next display period. If the MR viewing application is to be terminated (S152: Yes), end the series of processes.

[0091] Figure 15 is a flowchart of the VR creation program 85.

[0092] Various processes are provided for the information terminals 3a and HMDs 3b, 3c of the registered experiencer 11, participants 12a to 12e, and viewers 13a to 13c. The processing by the VR creation program 85 consists of a reception process from S161 to S165 and a transmission process from S165 to S171.

[0093] The reception process is provided by the VR server 8 for the experiencer 11 and the HMDs 1, 2a to 2e of the participants 12a to 12e.

[0094] The VR server 8 receives the camera shooting video data of the HMDs 1, 2a to 2e (S161), and also receives the position and direction data of the HMDs that have generated each camera shooting video (S162).

[0095] The VR server 8 normalizes the received camera shooting video data (S163), pastes it on the upper hemisphere surface, and appends it to the VR video data (S164).

[0096] The VR server 8 switches to the HMD for which there is a reception request for the camera shooting video data (S165: Yes) and returns to S161. If there is no HMD making a reception request (S165: No), the VR server 8 performs an end determination (S172).

[0097] The transmission process is provided by the VR server 8 for the information terminals 3a and HMDs 3b, 3c of the viewers 13a to 13c.

[0098] The VR server 8 determines whether the transmission and provision target is a new device (S166). If it is new (S166: Yes), it obtains the starting data of the transmission and provision target and transmits the initial data of the VR video (S167).

[0099] If it is not new (S166: No) or after transmitting the initial data of the VR video (S167), the VR server 8 receives movement data indicating the movement of the transmission and provision target HMD from the transmission and provision target HMD (S168).

[0100] The VR server 8 calculates the position and orientation of the transmission and provision target HMD, etc. based on the movement data, cuts out a part of the region from the VR video data on the upper hemisphere surface (S169), and transmits the cut-out VR video data (VR video update data) to the transmission and provision target HMD (S170).

[0101] If there are other HMDs or information terminals requesting the transmission of VR video data (S171: Yes), the VR server 8 switches to another transmission and provision target HMD (or information terminal) and returns to S166. If there are no other HMDs or information terminals requesting the transmission (S171: No), the VR server 8 makes an end determination (S172). If there is an end instruction (for example, when receiving the end of the MR experience from the HMD1 of the experiencer 11, etc.) (S172: Yes), the processing in the VR server 8 ends.

[0102] If there is no end instruction (S172: No), it returns immediately after the start of the processing of the VR server 8 and continues the processing between S161 and S166.

[0103] (AR Drawing Program) Figure 16 is a flowchart of the AR drawing program 95.

[0104] The AR server 9 provides various processes for the registered experiencer 11, participants 12a to 12e, and viewers 13a to 13c with respect to the HMDS 1, 2a to 2e, 3b, 3c, and information terminals 3a. The processes performed by the AR server 9 mainly include the process provided to the experiencer's HMD 1 in S181 to S187, the process provided to the participants' HMDS 2a to 2e in S188 to S191, and the process provided to the viewers' HMDS 3b, 3c, and information terminal 3a in S192 to S197.

[0105] In S181, the AR server 9 determines the presence and type of requests from the experiencer's HMD 1. If there is no request (S181: No), it waits for a request.

[0106] In the case of a new request (S181: New), the AR server 9 receives the selection data of the AR object from the HMD 1 (S182) and transmits the default rendering data of the selected AR object (S183). Then, it proceeds to step S187.

[0107] In the case of an operation request (S181: Operation), the AR server 9 receives the operation parameter data from the HMD 1 (S184), the AR server 9 redraws the AR object according to the operation parameters (S185), and transmits the data of the redrawn AR object to the HMD 1 (S186). Then, it proceeds to step S187.

[0108] In S187, the AR server 9 determines whether there is a next request. If there is a next request (S187: Yes), it continues; otherwise (S187: No), it proceeds to the end determination (S198).

[0109] In S188, among the participants' HMDS 2a to 2e, the position and orientation data of one registered unit (the HMD for the target participant to be transmitted) are received. Then, the AR object 4b selected by the experiencer 11's HMD 1 is redrawn according to the position and orientation data received in S188 (S189), and the data of the redrawn AR object is transmitted to the HMD for the target participant to be transmitted (S190).

[0110] In S191, if there are unprocessed registered participant HMDs 2a to 2e remaining, or if there are changes in the position or orientation of the registered participants (S191: Yes), it returns to S188. If all the registered participant HMDs 2a to 2e have been processed (S191: No), it proceeds to the end determination (S198).

[0111] In S192, the AR server 9 determines whether the transmission and provision target is the information terminal 3a of viewer 13a, or whether the viewer HMDs 3b or 3c are new (S192). If they are new (S192: Yes), it transmits the initial data of the AR object 4b (S193). The initial data is, similar to the case of VR video, for example, data drawn assuming a position and direction facing the HMD 1 of the experiencer 11 on the upper hemisphere spherical surface.

[0112] If they are not new (S192: No) or after transmitting the ARO drawing data (initial) (S193), the AR server 9 receives the movement information data of the viewer terminal for the transmission and provision target, for example, HMD 3b (S194), and calculates the position and orientation of the viewer terminal for the transmission and provision target. Then, the AR server 9 redraws the AR object 4b according to the received movement data (S195), and transmits the data of the redrawn AR object to the viewer terminal for the transmission and provision target (S196).

[0113] The AR server 9 determines whether there are remaining unprocessed viewer terminals for the transmission and provision target. If there are remaining (S197: Yes), it returns to S192. If the processing for all the viewer terminals for the transmission and provision target has been completed (S197: No), it proceeds to the end determination (S198).

[0114] The AR server 9 performs the end determination (S198). If there is an end instruction (S198: Yes), it ends the series of processes by the AR server 9. If there is no end instruction (S198: No), it returns to S180 and continues the processing.

[0115] In the above description, based on the distance relationship between the real object 4a and the AR object 4b, when a part of the real object 4a is in front of a part of the AR object 4b, the process of processing the rendering data of the AR object 4b so that a part of the AR object 4b appears hidden behind a part of the real object 4a has been described for each of the HMDS 1, 2a to 2e, 3b, 3c, and the information terminal 3a side. However, it may be performed on the AR server 9 side. In this case, in addition to the position and orientation data, each of the HMDS 1, 2a to 2e, 3b, 3c, and the information terminal 3a transmits video data created from the perspective of how the real object 4a appears in the composite video of a large number of captured data using the data of the real object 4a included in the VR video data obtained from the VR server 8, more specifically, the position data and orientation data based on each HMD, to the AR server 9. A step of receiving these data from each of the HMDS 1, 2a to 2e, 3b, 3c, and the information terminal 3a is added to the AR server 9.

[0116] FIG. 17 is a diagram showing an example of a method for selecting the position (viewer's perspective) of the viewer 13a to 13c viewing the VR video. In particular, by determining the perspective for observing the real object 4a and the AR object 4b, the real object 4a and the AR object 4b can be observed from the viewer's preferred perspective.

[0117] The three-dimensional arrows 301 and 302 are an example of pointers for selecting a viewpoint position and a line-of-sight direction. By operating the three-dimensional arrows 301 and 302, it is possible to select from which position in the MR space 6 to view the VR video. At this time, it is also possible to specify the line-of-sight direction by using a dashed line. Further, by rotating the MR space 6 itself, the positions and directions of the three-dimensional arrows 301 and 302 can be determined three-dimensionally. By using a pointer such as the three-dimensional arrows 301 and 302 that can select free positions and directions, it becomes possible to select a viewpoint position and a line-of-sight direction. To select the process of free positions and directions in two steps like the three-dimensional arrows 301 and 302, first, to determine the positions of the three-dimensional arrows 301 and 302, and then, to change the line-of-sight directions of the three-dimensional arrows 301 and 302, for example, the direction may be determined by grasping and moving the arrow part at the tip of the dotted line.

[0118] For the viewers' HMDs 3b and 3c or the information terminal 3a, when an input device, for example, a microphone 27 or a switch (not shown in FIG. 5) in the case of voice input, displays a pointer and enters a mode for selecting a line-of-sight position and a line-of-sight direction, a three-dimensional arrow is displayed. When that mode ends, the three-dimensional arrow is erased, and instead, it switches to the video from the selected line-of-sight position and line-of-sight direction.

[0119] According to the virtual reality sharing system 100 of the first embodiment, the viewers 13a to 13c at the remote site can view the MR experience by using the VR video generated from the camera-captured videos of the experiencer 11 and the participants 12a to 12e. Therefore, since the VR video can be generated based on the camera-captured videos viewed from various angles, the viewers 13a to 13c can continuously and approximately arbitrarily select the line of sight.

[0120] Also, the virtual reality sharing terminals (HMDs 1, 2a to 2e) for the experiencer 11 and the participants 12a to 12e contribute to the generation of the VR video, and the virtual reality sharing terminals (information terminal 3a and HMDs 3b, c) for the viewers 13a to 13c can share the MR experience by the VR video.

[0121] [Second Embodiment] FIG. 18 is a schematic diagram of the virtual reality sharing system 100a according to the second embodiment. The difference from the virtual reality sharing system 100 according to the first embodiment shown in FIG. 1 is that a fixed camera 14 is arranged. The captured video of the fixed camera 14 is provided to the VR server 8 by the network signal 10p, and becomes video data for creating a VR video together with the camera captured videos captured by the virtual reality sharing terminals for the experiencer 11 and the participants 12a to 12e.

[0122] As the fixed camera 14, for example, a 360° camera is used to obtain an upper hemisphere video, so that videos of areas that cannot be covered by the virtual reality sharing terminals for the experiencer 11 and the participants 12a to 12e can be provided. Thereby, since the generated VR video can be generated as a seamless upper hemisphere video, the VR video without a break can be observed as a background video from the virtual reality sharing terminals of the viewers 13a to 13c. Also, when the virtual reality sharing terminals for the experiencer 11 and the participants 12a to 12e can capture partially high-definition images compared to the 360° camera, it is possible to provide a high-quality VR video by combining those images.

[0123] FIG. 19 is a schematic diagram of a modification of the virtual reality sharing system 100b. The difference between the virtual reality sharing system 100b and the virtual reality sharing system 100a is that instead of the fixed camera 14a consisting of a 360° camera, fixed cameras 14b and 14c with limited viewing angles are arranged.

[0124] In FIG. 19, only the fixed cameras 14b and 14c are shown, but preferably they are arranged at the four corners of the MR space 6 to cover the shooting of the entire MR space. Or, the fixed cameras are arranged so that the entire object to be photographed can be photographed from multiple aspects. The captured videos of the fixed cameras 14b and 14c are provided to the VR server 8 by the network signals 10q and 10r respectively, and become video data for creating a VR video together with the camera captured videos captured by the virtual reality sharing terminals for the experiencer 11 and the participants 12a to 12e.

[0125] According to the virtual reality sharing system 100b, a VR video with relatively high image quality can be obtained compared to the virtual reality sharing system 100a.

[0126] Also, in the virtual reality sharing system 100a, a video is obtained as if the video is pasted inside the upper hemispherical surface, and the viewers 13a to 13c can obtain a video with the inside as the viewpoint. In contrast, in the virtual reality sharing system 100b, it is possible to obtain a video with the outside as the viewpoint centered on the AR object 4b.

[0127] As described above, according to the second embodiment, with a small addition of equipment, it has the same effect as the first embodiment, and a VR video with the video pasted on the entire upper hemispherical surface and the inside as the viewpoint, or a VR video with the outside viewpoint centered on the object can be obtained, making it easier for the virtual reality sharing terminals for the viewers 13a to 13c to grasp the entire MR space 6.

[0128] [Third Embodiment] FIG. 20 is a block diagram of the participant's HMD 2f of the virtual reality sharing system according to the third embodiment. In the third embodiment, in the virtual reality sharing systems 100, 100b, and 100c described in FIGS. 1, 18, and 19, the participant's HMD 2f is used instead of the participant's HMDs 2a to 2e of the first and second embodiments.

[0129] The screen 23 of the participant's HMD 2f is different from the participant's HMDs 2a to 2e of the first and second embodiments in that it includes a shutter 231 and a half mirror 232.

[0130] When the shutter 231 is controlled to be in the open state, the participant's HMD 2f functions as a transmissive HMD. That is, on the half mirror 232, it is possible to see the background video of the real space and the AR object 4b projected from the projector 22 (a general term for the left projector 22a and the right projector 22b).

[0131] On the other hand, when the shutter 231 is controlled to be closed, the AR object 4b and the VR video are projected from the projector 22 in a superimposed manner. In this case, the participant HMD 2f functions as an immersive HMD. As a result, the participants 12a to 12e can have an MR viewing experience in which they can select a free viewpoint position and a line-of-sight direction in the same way as the viewers 13a to 13c.

[0132] Figure 21 is a flowchart of the MR participation program 263a.

[0133] After activation, the HMD 2f logs in to the user management process for MR experience sharing (S211). When logging in to the user management process, it is registered as a participant in the VR server 8 and the AR server 9.

[0134] The HMD 2f starts camera shooting (S212). The subsequent steps may be performed in synchronization with the camera shooting cycle.

[0135] The HMD 2f detects the position of the HMD in the MR space (S213), detects the camera shooting direction of the HMD 2f (S214), and associates them as metadata of the camera shooting video.

[0136] In S218, the HMD 2f transmits the position and direction data of the HMD 2f to the AR server 9. The HMD 2f receives the drawing data of the AR object 4b generated by the experiencer 11 (S219).

[0137] On the other hand, if the shutter 231 is closed (S215: Yes), since it is synonymous with using the HMD 2f as an immersive HMD, the HMD 2f transmits the movement information of the HMD 2f to the VR server 8 (S216), receives VR video data matching the movement information from the VR server (S217), and proceeds to step S222.

[0138] If the shutter 231 is open (S215: No), since it is synonymous with using the HMD 2f as a transmissive HMD, VR video data is not required. Therefore, it proceeds to step S222.

[0139] In S222, the AR object 4b is superimposed on the VR video received in S217, or when used as a transmissive HMD, only the AR object 4b is projected onto the screen 23 and is superimposed on the background video seen through the shutter 231 and the half mirror 232. Then, it proceeds to step S223.

[0140] Here, the HMD 2f determines whether it is inside or outside the MR space 6 using the distances to the access points 5a to 5c. When the HMD 2f is inside the MR space 6, it is possible to select either immersive or transmissive, and the user performs a selection operation from the input device. Also, when it is determined that the HMD 2f is outside the MR space 6, it automatically switches to the immersive type (closes the shutter 231).

[0141] In S220, the data on the position and orientation of the HMD 2f (S220) and the camera captured video data (S221) are transmitted to the VR server 8. Then, it proceeds to step S223.

[0142] The HMD 2f determines the camera capture period (S223), waits for arrival (S223: Yes), checks for the end of the MR participation program, and if it has not ended (S224: No), continues the steps from S213 in the next camera period. If it ends (S224: Yes), the processing of the MR participation program ends.

[0143] As described above, according to the third embodiment, it has the same effects as the first and second embodiments, and the participants 2a to 2e can switch between AR viewing and VR viewing in the MR space 6 in the same form as the viewers 13a to 13c.

[0144] [Fourth Embodiment] FIG. 22 is a schematic diagram of the virtual reality sharing system 100c of the fourth embodiment.

[0145] In the virtual reality sharing system 100c of FIG. 22, the experiencers 11a to 11f and the participants 12f to 12i exist in the MR space 6.

[0146] The experiencers 11a to 11f and the participants 12f to 12i are wearing HMDS on their heads, and network signals are being transmitted and received from the HMDS. However, for the sake of simplicity of the drawings, the illustration is omitted.

[0147] In the virtual reality sharing system 100c, there are a plurality of experiencers 11a to 11f, who are divided into the experiencers 11a to 11c on the left side and the experiencers 11d to 11f on the right side. Each of the experiencers 11a to 11f holds an operation controller 15a to 15f, respectively, and uses each operation controller 15a to 15f to generate and operate AR objects 4c to 4e. Each of the operation controllers 15a to 15f has a built-in sensor for detecting its movement, and according to its movement, AR objects 4c to 4e are generated and moved.

[0148] In the virtual reality sharing system 100c, the AR objects 4c to 4e generated by the operation controllers 15a to 15f are virtual attack objects for attacking the experiencers 11a to 11c or 11d to 11f on the opposite side. By hitting the attack object on the experiencers 11a to 11c or 11d to 11f on the opposite side, the hit experiencers 11a to 11f are made to leave the field, and it is an e-sport that uses the MR space 6 to compete for the number of experiencers who have left the field within a certain time.

[0149] When the participants 12f to 12i use the participant HMD 2f of the third embodiment and use the worn HMD as a transmissive type, while capturing the experiencers 11a to 11f on both sides in the MR space 6, the AR objects 4c to 4e can be viewed in an overlapping manner. When the worn HMD is used as an immersive type, it is possible to view the VR video and the AR objects in an overlapping manner as if one were the experiencers 11a to 11f themselves. Also, similar to the previous embodiments, the viewers 13a, 13b, and 13c can freely select the viewpoint position and the line-of-sight direction to view inside the MR space 6.

[0150] Figure 23 is a flowchart of the AR painting program 95.

[0151] The AR server 9 executes a process for providing to the HMD 1 for the experiencer in S231 to S236, a process for providing to the HMDs 2a to 2f for the participants in S188 to S191 shown in FIG. 16, and a process for providing to the HMDs 3b and 3c for the viewers and the information terminal 3a in S192 to S197.

[0152] The AR server 9 discriminates the presence or absence and the type of a request from any one of the operation controllers 15a to 15f (transmission request terminals) held by the HMD 1 for the experiencer. When there is no request (S231: No), it waits for a request.

[0153] When a setting request is made (S231: setting), at least one setting data of the AR objects 4c to 4e is received from the transmission request terminal (S232), and one of the AR objects 4c to 4e generated by one of the operation controllers 15a to 15f held by the person wearing the HMD 1 for the experiencer and the parameters are set. When the operation controllers 15a to 15f are moved in a predetermined pattern in the "AR object setting" mode, the AR objects 4c to 4e are automatically generated.

[0154] When an operation request is made (S231: operation), the movement data of the operation controllers 15a to 15f is received (S233), one of the AR objects 4c to 4e is drawn in accordance with the movement data (S234), and the data of one of the drawn AR objects 4c to 4e is transmitted (S235).

[0155] The AR server 9 determines whether there is the following request, and decides whether to continue (S236: Yes) or proceed to the end determination S198 (S236: No). The loop indicated by the broken lines in S234 and S235 is the repetition of drawing and transmitting the AR objects 4c to 4e. The AR objects 4c to 4e generated by operating the operation controller have parameters for self-running. Once generated as attack objects, the AR objects 4c to 4e self-run toward the other party's experiencer in a determined speed pattern. When the AR objects 4c to 4e hit the experiencer, reach a predetermined range, or go out of the MR space 6, they automatically disappear.

[0156] In S198, an end determination is made. If there is an end instruction, the process ends (S199: Yes). If there is no instruction (S199: No), the process returns to S231, S188, and S192 to continue the process.

[0157] As described above, according to the fourth embodiment, it has the same effects as the first to third embodiments, and a virtual reality sharing system with a plurality of experiencers can also be shared.

[0158] As described above, the above-described embodiments explained with reference to FIGS. 1 to 23 are not limited to these. For example, in the above, a VR video is created by stitching together the camera shooting videos of a plurality of virtual reality sharing terminals existing in the MR space 6. However, a VR video may be created by stitching together the camera shooting videos taken by one virtual reality sharing terminal at different positions and viewpoints at different times.

[0159] Also, all of the plurality of virtual reality sharing terminals existing in the MR space 6 may be experiencer HMDs.

[0160] Furthermore, if a VR video is created at one end and stored in the VR server 8, the viewers 13a to 13c can not only share the virtual reality with the experiencer 11 and the participants 12a to 12e in real time, but also read the VR video created later and share the virtual reality.

[0161] Also, in the present invention, a part of the configuration of one embodiment can be replaced with that of another embodiment. Also, the configuration of another embodiment can be added to the configuration of one embodiment. All of these belong to the scope of the present invention. Further, the numerical values, messages, etc. appearing in the text and drawings are merely examples, and using different ones does not impair the effects of the present invention.

[0162] Also, functions of the invention, etc. may be implemented in hardware by designing part or all of them, for example, with an integrated circuit. Also, they may be implemented in software by a microprocessor unit, a CPU, etc. interpreting and executing an operation program. Also, the scope of software implementation is not limited, and hardware and software may be used in combination.

Description of Reference Numerals

[0163] 1: HMD for Experiencer 2a~2f: HMD for Participants 3a: Information Terminal for Viewer 3b, 3c: HMD for Viewer 4a: Real Object 4b~4d: AR Object 5a~5e: Access Point 6: MR Space 7: Network 8: VR Server 9: AR Server 10a~10r: Network Signal 11, 11a~11f: Experiencer 12a~12i: Participant 13a~13c: Viewer 14, 14a~14c: Fixed Camera 15a~15f: Operation Controller 21: Camera 22: Projector 22a: Left Projector 22b: Right Projector 23: Screen 24: Nose Pad 25: Processor 26: Speaker 27: Microphone 28: Frame housing 28c: Bridge 29: Distance measuring camera 30: Display 82: CPU 83: RAM 84: Storage 85: VR creation program 86: VR data 87: Internal bus 91: Network IF 92: CPU 93: RAM 94: Storage 95: AR drawing program 96: AR data 97: Internal bus 100, 100a~100c: Virtual reality sharing system 231: Shutter 232: Half mirror 250: Azimuth sensor 251: Gyro sensor 252: Acceleration sensor 253: Wireless communicator 254: CPU 255: RAM 256: Video RAM 258, 265: Basic program 259: MR experience program 260: Feature extraction processor 261: Distance calculation processor 262: Internal bus 263, 263a: MR participation program 266: MR viewing app 301, 302: Stereo arrow P1: Area P2: Video

Claims

1. A display and An input device; A communication device, A processor; Equipped with The processor, When the input device receives a setting input operation of a viewer's viewpoint for capturing a VR image obtained by synthesizing images captured by a plurality of cameras obtained by capturing the same real space from different viewpoints, data indicating a position and a direction of the viewer's viewpoint is transmitted from the communication device to a VR server that creates the VR image and an AR server that creates an AR object; receiving the VR video as seen from the viewer's viewpoint from the VR server via the communication device, and receiving rendering data of the AR object as seen from the viewer's viewpoint from the AR server; The AR object based on the drawing data is superimposed on the VR image seen from the viewer's viewpoint and displayed on the display; The plurality of camera images include a first camera image and a second camera image, The first camera captured image includes metadata indicating a relative position of a real object that is a display trigger of the AR object as viewed from a first viewpoint, the second camera captured image includes metadata indicating a relative position of the physical object as viewed from a second viewpoint; The processor, receiving the VR image to which the relative position as seen from the first viewpoint and the relative position as seen from the second viewpoint have been added from the VR server via the communication device; a position of the real object captured in the VR video as seen from the viewer's viewpoint is compared with a position of the AR object as seen from the viewer's viewpoint, and when the real object is closer than the AR object, at least a part of the AR object is hidden by the real object and displayed on the display, and when the real object is farther away than the AR object, at least a part of the real object is hidden by the AR object and displayed on the display. A virtual reality sharing terminal characterized by:

2. The virtual reality sharing terminal according to claim 1, the processor displays a pointer indicating a viewpoint position and a line of sight direction of the viewer viewpoint on the display, and the input device accepts an input operation using the pointer. A virtual reality sharing terminal characterized by:

3. The virtual reality sharing terminal according to claim 2, The processor executes either a viewpoint setting mode or a viewing mode, In the viewpoint setting mode, the pointer is displayed on the display; In the viewing mode, the pointer is hidden from the display and the AR object is displayed superimposed on the VR image viewed from the viewer's viewpoint. A virtual reality sharing terminal characterized by:

4. The virtual reality sharing terminal according to claim 1, The processor, receiving, from the VR server via the communication device, the VR image obtained by combining the first camera captured image generated by capturing the real space from a first viewpoint and the second camera captured image generated by capturing the real space from a second viewpoint at the same time as when the first camera captured image was captured; A virtual reality sharing terminal characterized by:

5. The virtual reality sharing terminal according to claim 1, The processor, receiving, from the VR server via the communication device, the VR image obtained by combining the first camera captured image generated by capturing the real space from a first viewpoint and the second camera captured image generated by capturing the real space from a second viewpoint at a time different from when the first camera captured image was captured; A virtual reality sharing terminal characterized by:

Images