3D virtual reality display system

Abstract

To provide a technique capable of more accurately recognizing a three-dimensional AR object.SOLUTION: A three-dimensional virtual reality display system comprises a server (9) and a head mount display HMD (2a) connected in radio communication with the server. The HMD measures three-dimensional real space image data obtained by capturing a real object and the distance from an observer of the real space to a real object, and transmits the distance data to the server. The server uses a shooting direction of three-dimensional real space image data as a visual line of the observer. When a real object exists on the visual line on which the observer observes the three-dimensional virtual reality object, the server compares the distance from the observer to the place where the three-dimensional virtual reality object is displayed with the distance from the observer indicated in the distance data to the real object. When the real object overlaps the three-dimensional virtual reality object, the server displays the three-dimensional virtual realty object on the visual line and generates display image data having been subjected to overlapping elimination display processing which does not display the real object image on the visual line. The HMD displays the image based on the display image data.SELECTED DRAWING: Figure 7A

Description

[Technical Field]

[0001] The present invention relates to a three-dimensional virtual reality display system, and more particularly to a technology for experiencing mixed reality (MR) that includes a real space and a virtual reality object (AR object). [Background technology]

[0002] Patent Document 1 discloses a technology in which "an information processing device outputs a composite image, which is a composite of a real space image and a virtual object image, to a display device, identifies the position of the real object, determines whether the real object is moving based on the information on the identified position of the real object, and if the real object is moving, outputs the composite image to be displayed on the display device so that the real object can be recognized" (abstract excerpt). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-122392 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, when a 3D AR object overlaps with a real object in real space in the line of sight from the HMD, the display of the 3D AR object is restricted by making the 3D AR object semi-transparent or by hiding the 3D AR object near the real object, etc. This may prevent accurate recognition of the 3D AR object and prevent an MR experience.

[0005] The present invention has been made in view of the above-mentioned problems, and aims to provide a technique that enables more accurate recognition of three-dimensional AR objects. [Means for solving the problem]

[0006] In order to solve the above problems, the present invention has the configuration described in the claims. For example, the present invention provides a 3D virtual reality display system, comprising: a server; and a head-mounted display connected to the server via wireless communication, the server comprising a first processor, the head-mounted display comprising: a camera that captures an image of a real object present in a real space and outputs 3D real space image data including the real object image; a distance sensor that measures a distance from an observer in the real space to the real object and outputs distance data; a display; and a second processor, the head-mounted display transmits the 3D real space image data and the distance data to the server, the server receives the 3D real space image data and the distance data, the first processor uses the capturing direction of the 3D real space image data as the observer's line of sight, and, when the real object is present in the line of sight along which the observer observes the 3D virtual reality object, compares the distance at which the 3D virtual reality object is displayed from the observer with the distance from the observer to the real object indicated in the distance data. and determining that the physical object overlaps with the three-dimensional virtual object if, as a result of the comparison, the distance at which the three-dimensional virtual object is displayed from the observer and the distance from the observer to the physical object indicated in the distance data are the same in the same line of sight of the observer; When the real object overlaps the 3D virtual reality object, display image data is generated that has undergone overlap elimination display processing in which the 3D virtual reality object is displayed in the line of sight and the real object image is not displayed in the line of sight, the display image data is transmitted from the server to the head-mounted display, the head-mounted display receives the display image data, and the second processor displays an image including the 3D virtual reality object on the display based on the display image data. [Effects of the Invention]

[0007] According to the present invention, it is possible to more accurately recognize a three-dimensional AR object. Objects, configurations, and effects other than those described above will become apparent from the following embodiments. [Brief explanation of the drawings]

[0008] [Figure 1]Schematic diagram of a 3D virtual reality display system according to a first embodiment. [Figure 2] External view of an HMD as an example of a 3D virtual reality display device [Figure 3] HMD block diagram [Figure 4] VR service server block diagram [Figure 5] MR support server block diagram [Figure 6A] A diagram showing an example of a conventional 3D virtual reality display (showing a 3DAR object overlapping with a real object) [Figure 6B] FIG. 1 is a diagram showing a 3D virtual reality display (first overlap-eliminating display example) according to the first embodiment; [Figure 6C] FIG. 10 is a diagram showing a 3D virtual reality display (second overlap-eliminating display example) according to the first embodiment; [Figure 6D] FIG. 10 is a diagram showing a 3D virtual reality display (third overlap-eliminating display example) according to the first embodiment; [Figure 7A] Flowchart of the MR experience program according to the first embodiment [Figure 7B] An example of an algorithm for determining the degree of volume overlap. [Figure 8] Flowchart of the VR service server according to the first embodiment [Figure 9] Flowchart of the MR support server according to the first embodiment [Figure 10A] A diagram showing an example of a conventional 3D virtual reality display (showing a real object in front of the 3DAR object) [Figure 10B] FIG. 10 is a diagram showing a 3D virtual reality display according to the second embodiment (fourth overlap-eliminating display example); [Figure 11] Flowchart of MR experience program according to the second embodiment [Figure 12A] A diagram showing an example of a conventional 3D virtual reality display (showing a 3DAR object overlapping with a real object) [Figure 12B] FIG. 10 is a diagram showing an example of a three-dimensional virtual reality display according to the third embodiment (example of processing a transparent portion); [Figure 12C]FIG. 10 is a diagram showing an example of a three-dimensional virtual reality display according to the third embodiment (an example of replacement with a VR image); [Figure 13] Diagram showing the 3DAR object table [Figure 14] Flowchart of MR experience program according to the third embodiment [Figure 15] FIG. 10 is a diagram showing another example of a three-dimensional virtual reality display according to the third embodiment. [Figure 16] Block diagram of the MR support server according to the fourth embodiment [Figure 17] 10 is a flowchart showing the flow of a 3D virtual reality display process according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all the drawings, the same components and steps are designated by the same reference numerals, and redundant explanations will be omitted.

[0010] In this embodiment, a three-dimensional real space image (hereinafter referred to as a "real space image") captured by a distance measuring camera is combined with a three-dimensional virtual reality object (hereinafter referred to as a "3DAR object" and written as "3D-ARO" in the drawings) created by CG (Computer Graphics) and displayed. In this embodiment, the camera and a distance sensor that measures distance are integrated by using the distance measuring camera 20, but the camera may also be combined with a separate distance sensor such as an ultrasonic range finder.

[0011] When a 3DAR object is superimposed on a 3D real-space image obtained by capturing real space, an area of ​​the image that should be displayed farther from the viewpoint is obscured by the image that should be displayed closer to the viewpoint. Occlusion is an image processing technique that represents this obscured area.

[0012] Mixed reality (MR) images, in which an AR image is synthesized with a background image of real space, are used in content such as games, maintenance work, and sales promotions. To synthesize an AR image, for example, an object called an AR marker is captured from a real space image, and an AR image associated with the AR marker is superimposed on the object area. Since an HMD (head-mounted display) in which a camera and a display are integrated is often used as hardware for displaying three-dimensional virtual reality, the following describes an embodiment in which the present invention is implemented in an HMD.

[0013] [First embodiment] The first embodiment will be described with reference to FIGS.

[0014] (Configuration of 3D virtual reality display system) FIG. 1 is a schematic diagram of a 3D virtual reality display system according to the first embodiment.

[0015] 1, an MR user 2 wears an HMD 2a on his / her head and views an MR space 1. Similarly, an MR user 3 wears an HMD 3a on his / her head and views the MR space 1.

[0016] The HMDs 2a and 3a are connected to the access point 1a via wireless communication by transmitting and receiving wireless LAN signals 1b, 2b, and 3b, respectively.

[0017] The access point 1a is placed in the MR space 1. The access point 1a is connected to a network 7 outside the MR space 1, and allows the HMDs 2a and 3a to communicate with a VR service server 8 and an MR support server 9 placed on the network 7. VR (Virtual Reality) refers to a virtual reality space.

[0018] In the real space, there are two people 4a and 4b who have not experienced MR. A vase 5a and a window 5b are present as part of the background of the real space. A 3DAR object 6 is a three-dimensional AR object of a car.

[0019] The MR space 1 is a space intended for car sales promotion, and is not limited to a single specific MR experiencer. Multiple people, such as MR experiencers 2 and 3, may simultaneously experience the MR experience. The MR experiencer 3 may be a product clerk who is observing the same 3DAR object 6 as the MR experiencer 2 from a different direction, or may be an independent visitor observing a different 3DAR object from the MR experiencer 2. By presenting a car for sale as the 3DAR object 6, the promoter does not need to exhibit an expensive real object (car), nor does it need a large space to exhibit multiple cars. In addition to the MR experiencers 2 and 3, the MR space 1 may also include visitors who have not experienced MR 4a and 4b. The non-MR experiencers 4a and 4b may be family members of the MR experiencers 2 and 3, or visitors who are waiting to experience the MR experience.

[0020] The non-MR users 4a and 4b do not observe the 3DAR object 6, and there are no restrictions on their movement within the MR space 1. For this reason, it is possible for them to be in the same position as the 3DAR object 6, as is the case with the non-MR user 4b.

[0021] FIG. 2 is an external view of HMDs 2a and 3a as an example of a three-dimensional virtual reality display device. The HMD 2a is a video see-through type HMD. The HMD 3a has the same configuration as the HMD 2a, so a description thereof will be omitted. The HMD 2a is equipped with a distance measuring camera 20 with parallax and a display 22. The HMD 2a captures an image of the foreground with the distance measuring camera 20, and displays a 3DAR object 6 rendered by CG or the like on the display 22 placed in front of the MR user 2 by combining the image captured by the distance measuring camera 20 with the image.

[0022] The distance measuring camera 20 includes a left camera 20a and a right camera 20b, and is a distance measuring camera for measuring the distance to an object being photographed. The display 22 is a flat display with a shutter 23 provided inside. When images for the left eye and images for the right eye are alternately displayed on the display 22, the shutter 23 opens and closes in synchronization with the display on the display 22. That is, when an image for the left eye is displayed, the left half of the display 22 opens and the right half closes, and when an image for the right eye is displayed, the left half of the display 22 closes and the right half opens. This enables the HMD 2a to support three-dimensional display. The MR user 2 alternately views the displayed image with only one eye in synchronization with the displayed image.

[0023] The HMD 2a further includes a processor 24 and wearing housings 25a and 25b. The wearing housings 25a and 25b allow the HMD 2a to be worn on the head.

[0024] The real space image of the front taken by the left camera 20a and the right camera 20b is displayed on the display 22, and the MR experience person 2 views the real space image. The display 22 also displays the 3DAR object 6 superimposed on the real space image. At this time, an image of an AR object for the left eye is superimposed on the image taken by the left camera 20a, and an image of an AR object for the right eye is superimposed on the image taken by the right camera 20b, and the images are displayed on the display 22, so that the 3DAR object 6 is displayed stereoscopically (three-dimensionally) as if it were at a predetermined distance in real space.

[0025] The display on the HMD 2a reflects the front-to-back relationship of the distance between real objects in real space, in Figure 1, the non-MR user 4b, the vase 5a, the window 5b, and the 3DAR object 6. For example, when a part of the real object (non-MR user 4b) is in front of a part of the 3DAR object 6, occlusion processing is performed to process the rendering data of the 3DAR object 6 so that a part of the 3DAR object 6 appears to be hidden by a part of the real object (non-MR user 4b).

[0026] (Block diagram of a 3D virtual reality display device) Figure 3 is a block diagram of the HMD 2a. In Figure 3, the same components as in Figure 2 are assigned the same numbers. In Figure 3, the processor 24 is the part surrounded by a dashed line, and the left camera 20a, right camera 20b, display 22, shutter 23, speaker 26, and microphone 27 are connected to the processor 24.

[0027] The processor 24 includes a camera processor 240, a direction sensor 241, a gyro sensor 242, an acceleration sensor 243, a wireless communication device 244, a CPU 245 (corresponding to the main processor), a RAM 246, an image RAM 247, a Flash ROM (FROM) 248, and an internal bus 249, and each element is connected to each other via the internal bus 249.

[0028] The wireless communication device 244 selects an appropriate process from several communication processes such as mobile communication such as 4G and 5G, and wireless LAN, and connects the HMD 2a to the network 7 via the access point 1a.

[0029] The FROM 248 includes a basic program 250 and an MR experience program 251. The CPU 245 loads these processing programs into the RAM 246 and executes them. Furthermore, the FROM 248 stores data necessary for executing the processing programs. The FROM 248 may be a non-volatile memory medium other than a Flash ROM.

[0030] The CPU 245 also stores image data to be sent to the display 22 in the image RAM 247 and then reads it out.

[0031] Camera processor 240 calculates the distance to the subject (corresponding to a real object) of the real space image based on the images captured by left camera 20a and right camera 20b, and adds distance data to the subject of the real space image to the real space image. In this specification, "real space image" means only the image, and data with distance data added to it is called "real space image data."

[0032] A group of sensors such as a direction sensor 241, a gyro sensor 242, and an acceleration sensor 243 are used to know the position of the HMD 2a and the imaging direction of the distance measuring camera 20 (used as the line of sight of the MR user 2 wearing the HMD 2a).

[0033] The HMD 2a may include some or all of the processes executed by the VR service server 8 and the MR support server 9 described below.

[0034] 4 is a block diagram of the VR service server 8. The VR service server 8 includes a network interface (network IF) 81 such as a wired LAN, a CPU 82, a RAM 83, and a storage 84, which are connected to each other via an internal bus 85.

[0035] The storage 84 may be a Flash ROM or may be a combination of a hard disk drive, etc. The storage 84 stores a VR service program 86. The CPU 82 loads the VR service program 86 into the RAM 83 and executes it.

[0036] Furthermore, the storage 84 stores VR data 87 such as 3DAR objects. The VR data 87 is data necessary for executing the VR service program 86.

[0037] The VR data 87 may include 3DAR objects as well as VR (Virtual Reality) image data. The VR image data is an image that replaces the entire real-space image of the MR experiencers 2 and 3, allowing the MR experiencers 2 and 3 to experience observing the 3DAR object 6 while feeling as if they are in a different space provided by the VR image data.

[0038] 5 is a block diagram of the MR support server 9. The MR support server 9 includes a network IF 91 such as a wired LAN, a CPU 92, a RAM 93, and a storage 94, which are connected to each other via an internal bus 95.

[0039] The storage 94 may be a Flash ROM or may be a combination of a hard disk drive, etc. The storage 94 includes an MR assistance program 96 as a processing program. The CPU 92 loads the MR assistance program 96 into the RAM 93 and executes it.

[0040] Furthermore, the storage 94 stores a background object image 97 and a real object image 98. These are data necessary for executing the MR assistance program 96.

[0041] The background object image 97 and the real object image 98 are data for the user to have an MR experience, and when corresponding to multiple users, separate data exists for each user.

[0042] The real object image 98 is data obtained by detecting a moving area from the real space image received from the HMD 2a using time difference or the like, and recognizing a block of area as a real object. Furthermore, it may be possible to detect what the real object is, for example, whether it is a person, based on the shape of the real object or the like.

[0043] The background object image 97 is data of a background image obtained by removing the area of ​​a real object from the real space image, and is data of a motionless area of ​​the real space image. The removed area of ​​the real object is filled with data from a real space image going back in time when the real object did not appear in that area, to obtain a background image. More specifically, since the background object image 97 is located behind the real object image 98, the background object located further behind the real object image 98 is not captured at a certain point in time, i.e., in the same frame (target frame) of a 3D real space image consisting of multiple frames. Therefore, the MR assistance program 96 recognizes the real object image 98 from the target frame and extracts the background object image 97 from other frames in which it does not appear, thereby generating the background object image 97.

[0044] (Image from 3D virtual reality display) 6A to 6D are diagrams illustrating a 3D virtual reality display method. FIG. 6A is a diagram illustrating an example of a conventional 3D virtual reality display (a diagram illustrating a state in which a 3DAR object and a real object overlap), FIG. 6B is a diagram illustrating a 3D virtual reality display according to the first embodiment (a first example of a display in which overlapping is resolved), FIG. 6C is a diagram illustrating a 3D virtual reality display according to the first embodiment (a second example of a display in which overlapping is resolved), and FIG. 6D is a diagram illustrating a 3D virtual reality display according to the first embodiment (a third example of a display in which overlapping is resolved). FIGS. 6A to 6C correspond to the area (AREA) surrounded by a dashed line in FIG. 1. As shown in FIG. 1, the MR non-experiencer 4b and the 3DAR object 6, which are real objects, are located at approximately the same distance from each other and overlap in the line of sight of the HMD 2a. The "distance relationship" here refers to the distance relationship in the same line of sight direction of the MR experiencer 2, who is wearing the HMD 2a, with the HMD 2a as the base point. This does not include distance relationships where the line of sight is different even if the distance from MR experiencer 2 is the same.

[0045] 6A shows an image in which occlusion processing is performed on the overlapping non-MR user 4b (real object) and 3DAR object 6. The upper part of the non-MR user 4b in front of the 3DAR object 6 is displayed, but the lower part of the non-MR user 4b behind the 3DAR object 6 is not displayed. As a result, the image shows the non-MR user 4b appearing from within the 3DAR object 6 (as if the upper half of a person's body is sitting on or growing on the hood of a car), resulting in an unnatural image in which the non-MR user 4b is obstructing the recognition of the 3DAR object 6. For this reason, in this embodiment, processing shown in FIG. 6B or FIG. 6C is performed.

[0046] In Figures 6B and 6C, occlusion processing is not performed between the real object (non-MR user 4b) and the 3DAR object 6, which are located at approximately the same distance. Instead, a background object image 10 cut out from the background image is inserted into the area of ​​the real object (non-MR user 4b), and occlusion processing is performed between the background object image 10 and the 3DAR object 6. The background object image 10 is usually located at a greater distance in the same line of sight than the 3DAR object 6, so the 3DAR object 6 can be displayed without being cut off, and the background (e.g., vase 5a) that was hidden by the real object (non-MR user 4b) is also displayed, allowing for a natural MR experience. This completes the processing of Figure 6B (first overlap resolution display).

[0047] In FIG. 6C , in addition to FIG. 6B , the image of the real object (non-MR user 4b) is moved (evacuated) to a location that does not interfere with the recognition of the 3DAR object 6 (second overlap resolution display). Image 11 in FIG. 6C shows the real object (non-MR user 4b) after the movement. In cases where the real object (non-MR user 4b) is an entity that requires constant attention, such as a child of the MR users 2 and 3, the real object is not hidden, allowing the 3DAR object 6 and the real object (non-MR user 4b) to be simultaneously visible, and the immersive feeling of the MR experience is maintained without displaying unnatural images due to occlusion. When the real object (non-MR user 4b) is moved, the distance from the same line of sight to the MR user 2 may vary, such as from a distant location to a nearby location. In such cases, the scale of the real object (non-MR user 4b) may be scaled down or up depending on the distance from the MR user 2, allowing it to be viewed at a natural size.

[0048] FIG. 6D is an example (third overlap resolution display example) in which a real space image is replaced with VR image data 6a. When the VR image data 6a is behind the 3DAR object 6, the 3DAR object 6 is displayed with the VR image 6a as the background. Also, in a car sales promotion scene, switching from the image of FIG. 6B or 6C to a display like that of FIG. 6D can display an image combining a variety of backgrounds with the 3DAR object 6, allowing the viewer to view the 3DAR object 6 in a variety of scenes. When switching from the image of FIG. 6B or 6C to the background image of FIG. 6D, the images may be gradually composited and changed, or processing such as a wipe may be performed.

[0049] (flowchart) 7A is a flowchart of the MR experiencing program 251 according to the first embodiment. The MR experiencer 2 starts the MR experiencing program 251 stored in the HMD 2a (S101) and logs in to the VR service server 8 and the MR support server 9. The operation of the HMD 2a while the MR experiencing program 251 is being executed will be described below in order of steps. In addition, the following description will be given taking as an example the processing when the non-MR experiencer 4b as a real object overlaps or does not overlap the 3DAR object 6 of a car.

[0050] The HMD 2a starts camera shooting (S102). Distance data to real objects is attached to images captured by the distance measuring camera 20. The camera may capture a moving image at, for example, 30 fps (frames per second) to generate 3D real space image data in which multiple frames are arranged in time series, and capture the captured image. The subsequent steps may be executed in synchronization with the camera shooting cycle.

[0051] The HMD 2a transmits the three-dimensional real space image data to the MR support server 9 via the wireless communication device 244 (S103). The MR support server 9 separates a real object image (the MR non-experiencer 4b) and a background object image (e.g., a vase 5a and a window 5b) from the real space image, as will be described later.

[0052] Furthermore, the HMD 2a transmits a transmission request for the rendering data (included in the VR data 87) of the 3DAR object 6 to the VR service server 8 (S104).

[0053] The HMD 2a receives at least one, preferably all, real object image data (including the real object image and its distance data) extracted from the real space image from the MR support server 9 (S105), and receives drawing data and VR image data of the 3DAR object (3DAR object 6 in this example) from the VR service server 8 (S106).

[0054] The HMD2a compares the three-dimensional overlap between each real object (non-MR user 4b) and the 3DAR object 6, in other words, the distance between the real object image from the HMD2a (including the image of the non-MR user 4b and distance data up to that point) and the 3DAR object 6 from the HMD2a on the same line of sight when the HMD2a is used as the reference.

[0055] When a real object and a three-dimensional 3DAR object 6 are at the same distance on the same line of sight, the volume of the real object and the volume of the 3DAR object overlap. Therefore, if occlusion is performed without considering this volume overlap, even if the occlusion process is successful for the area in front of the 3DAR object 6 and the surface of the real object, the distance relationship between the area behind the 3DAR object 6 and the surface of the real object will not be processed appropriately, and an unnatural display may result in a real object suddenly appearing from within the 3DAR object 6.

[0056] In this embodiment, the conventional occlusion process or overlap resolution display process is selected depending on the degree of overlap between the volume of the real object and the volume of the 3DAR object 6.

[0057] Therefore, if the volume of the real object is at a distance that does not overlap with the volume of the 3DAR object 6 (S107: Separate), the HMD 2a executes occlusion processing between the real object and the 3DAR object (S108).

[0058] On the other hand, when the volume of the real object (the non-MR user 4b) is at a distance where it overlaps with the volume of the 3DAR object (S107: overlapped), the HMD 2a performs overlap resolution display processing.

[0059] An example of the determination algorithm in step S107 will be described with reference to FIG. 7B. FIG. 7B is a diagram showing an example of an algorithm for determining the degree of volume overlap. For ease of explanation, the display surface of the HMD 2a is assumed to be a plane parallel to the vertical direction. Then, a point on the display 22, for example the upper left corner of the display 22, is set as the origin, and real three-dimensional coordinates are defined using the two-dimensional coordinate xy plane of the screen and the z axis perpendicular to it. Therefore, the zx plane is a horizontal plane, and the z axis indicates the distance in the depth direction of the line of sight as seen from the HMD 2a.

[0060] In the example of Fig. 7B, when a real object is located in front of the HMD 2a, the value of the z axis corresponds to the distance from the HMD 2a to the non-MR user 4b. Since the non-MR user 4b is visually recognized by the distance measuring camera 20, the position P R (x R ,z R ) can be expressed as the intersection of the line of sight L and the surface facing the HMD 2a of the non-MR user 4b.

[0061] On the other hand, the shape of the 3DAR object 6 is defined by the three-axis coordinates (s, t, u) of the three-dimensional image system. When an AR marker appears in the real space, the 3DAR object 6 is displayed superimposed on it. Therefore, the origin (s0, t0, u0) of the 3DAR object 6 is defined by the three-dimensional coordinates (x l ,y m ,z n ), then (s0,t0,u0) is (x l ,y m ,z n For ease of explanation, there is no deviation in the rotation direction of each axis between the stu coordinate system and the xyz coordinate system, and the s axis coincides with the x axis, the t axis coincides with the y axis, and the u axis coincides with the z axis.

[0062] If there is only one point that constitutes the 3DAR object 6 on the line of sight L of the HMD 2a, the processor 24 sets that point as the farthest point P n , multiple points, e.g., P1,...,P n-2 , P n-1 , P n If there is a point, the point farthest from the HMD2a, i.e., the point with the largest z-axis value, is called the farthest point P n The point P1 with the smallest z-axis value is the closest point.

[0063] And the intersection point P of the line of sight L and the MR non-experienced person 4b R 3D coordinates (x R , y R , z R ) and the farthest point P of 3DAR object 6 n coordinates (x ARn , y ARn , z ARn ) (However, in this example, x R =x ARn , y R =y ARn ) and z R >z ARn If so, it is determined that there is no overlapping volume between the real object and the 3DAR object 6 (state 1). R If ≦zar, it is determined that there is an overlapping area in volume between the real object and the 3DAR object 6 (state 2).

[0064] Therefore, the HMD 2a requests data of a background object image (corresponding to the background object image 10 in FIG. 6B) corresponding to the area of ​​the real object (the non-MR user 4b) from the MR support server 9 (S109) and receives it (S110). After receiving the data, the HMD 2a hides the real object (the non-MR user 4b) with the background object image 10 and the 3DAR object 6 (first overlap dissolution display). The HMD 2a also cuts out the real object (the non-MR user 4b) and displays it in a location that does not overlap with the 3DAR object 6, and fits the background object image 10 into the area where the real object (the non-MR user 4b) actually exists (second overlap dissolution display).

[0065] If background VR image data is received together with the rendering data of the 3DAR object 6 in S106, the background object image among the real objects is replaced with a VR image in steps S107 and S108, and synthesis processing such as occlusion processing and movement of the real object is performed between the background VR image, the 3DAR object, and the real object in S108. Here, in this example, the processing for synthesizing the display image of the HMD 2a is shown as being performed within the HMD 2a, but the location where the synthesis processing is performed is not limited thereto, and as will be described later, the processing may be performed on a server connected via a network, or on a smartphone or tablet connected in cooperation with the HMD 2a.

[0066] The HMD 2a checks whether steps S107 to S111 have been executed for all real objects that overlap with the 3DAR object 6, and if any real objects remain (S112: No), the process returns to S107. On the other hand, if the processing has been completed for all real objects (S112: Yes), the HMD 2a displays the processed image on the display 22 of the HMD 2a (S113).

[0067] If the MR experience program 251 of the HMD 2a has not ended, the steps from S103 are continued in the next camera cycle (S114: No). If the MR experience program 251 of the HMD 2a has ended (S114: Yes), the above process ends.

[0068] 8 is a flowchart of the VR service program 86. When a login request from a registered MR experiencer 2 is received, login processing is executed (S121).

[0069] When the VR service server 8 receives a request to send rendering data of the 3DAR object 6 from the HMD 2a (S122), it creates rendering data of the requested 3DAR object (S123). The rendering data of the 3DAR object is data (object file) that renders the 3DAR object 6 in 3D according to the distance between the HMD 2a and the 3DAR object, the line of sight direction of the HMD 2a, and other information included in the request to send rendering data of the 3DAR object, and the rendering data is updated according to movement of the HMD 2a and changes in the line of sight. In addition, the effects of reflections and shadows from the direction of light sources such as the sun and lighting may be added as images to create the rendering data.

[0070] The VR service server 8 transmits the created drawing data to the HMD 2a (S124).

[0071] The VR service server 8 continues the processing from steps S122 to S124 until the end condition of the VR service program 86 is met (S125: No), such as the MR experiencer 2 logging out or the MR experience program 251 ending.

[0072] When the termination condition of the VR service program 86 is met (S125: Yes), the VR service server 8 terminates the above series of processes.

[0073] FIG. 9 is a flowchart of the MR assistance program 96.

[0074] The MR support server 9 processes a login request from the registered MR experiencer 2 (S131).

[0075] The MR support server 9 receives the real space image data from the HMD 2a (S132), recognizes the real object image (S133), extracts the real object image data, and obtains a background image. The background image is updated every time a real space image is received (S134).

[0076] The MR support server 9 transmits the real object image data to the HMD 2a (S135). When the MR support server 9 receives a request to transmit a background object image (S136), it transmits the background object image data to the HMD 2a (S137).

[0077] The MR support server 9 continues the processing from steps S132 to S137 until a termination condition of the MR support program 96 is met, such as the MR experiencer 2 logging out or the MR experience program 251 ending (S138: No).

[0078] When the termination condition of the MR assistance program 96 is met (S138: Yes), the MR assistance server 9 terminates the above series of processes.

[0079] According to this embodiment, when a real object and the 3DAR object 6 overlap in the same line of sight of the MR user, if the volumes of the real object and the 3DAR object 6 are too far apart to overlap, occlusion is performed, but if the volumes of the real object and the 3DAR object 6 are too close to overlap, no occlusion is performed and overlap resolution processing is performed, thereby preventing the real object and the 3DAR object 6 from being displayed unnaturally overlapping. This can enhance the sense of immersion in the MR experience.

[0080] Furthermore, according to this embodiment, even in an open space where a third party (non-MR experiencer) who is not experiencing MR is present, the shape of the 3DAR object 6 is not damaged by the third party, so the MR experiencer can accurately recognize the 3DAR object 6 and can experience MR.

[0081] [Second embodiment] A second embodiment of the present invention will be described with reference to Fig. 10A, Fig. 10B, and Fig. 11. Fig. 10A is a diagram showing an example of a conventional three-dimensional virtual reality display (a diagram showing a state in which a real object is located in front of a 3DAR object 6). Fig. 10B is a diagram showing a three-dimensional virtual reality display according to the second embodiment (a fourth example of overlap resolution display).

[0082] As shown in Figure 10A, the non-MR user 4b and the 3DAR object 6, which are real objects, are located at approximately the same distance, and another non-MR user 4c, which is also a real object, is located in front of the 3DAR object 6, and both overlap with the line of sight of the HMD 2a, hindering the observation of the 3DAR object 6.

[0083] In FIG. 10B , the non-MR user 4b, who is behind the 3DAR object 6, is replaced with a background object image 10, as in the first embodiment. Meanwhile, the other non-MR user 4c is deleted and the 3DAR object 6 is placed therein. Furthermore, for the remaining area from which the non-MR user 4c has been deleted and where the 3DAR object 6 does not overlap, an image corresponding to the remaining area is extracted from the panoramic image of another frame to generate a foreground image, which is then embedded in the remaining area. The 3DAR object 6 and background object image 10 correspond to the non-MR user 4b, while the 3DAR object 6 and foreground image 11a correspond to the other non-MR users 4c. A process is performed in which the background object image 10, foreground image 11a, and 3DAR object 6 overwrite and hide the real object images of the non-MR user 4b and the other non-MR users 4c (fourth overlap resolution display example). As a result, the entire 3DAR object 6 becomes observable.

[0084] FIG. 11 is a flowchart of the MR experience program 251 according to the second embodiment.

[0085] The difference from the flowchart in Figure 7 is the comparison of the distances of steps S150.

[0086] In S150, the distance between the real object and the 3DAR object 6 is determined as either "located behind and far away" or "close or in front." In the former case, occlusion processing is performed on the real object and the 3DAR object 6 (S108). In the latter case, a request to send a background object image and a foreground image is made (S151) and these are received (S152). Then, the real object image is hidden by the background object image, the 3DAR object 6, and the foreground image (S153). In the above example, processing is performed so that the non-MR user 4b and another non-MR user 4c appear as if they are not present.

[0087] As described above, the second embodiment has the same features as the first embodiment, and can remove a real object that interferes with observation of the 3DAR object 6 even if the real object is in front of the 3DAR object 6.

[0088] [Third embodiment] The third embodiment will be described with reference to FIGS. 12A to 15B.

[0089] FIG. 12A is a diagram showing an example of a conventional 3D virtual reality display (a diagram showing a state in which a 3DAR object and a real object overlap). FIG. 12B is a diagram showing an example of a 3D virtual reality display according to the third embodiment (an example of processing a transparent portion). FIG. 12C is a diagram showing an example of a 3D virtual reality display according to the third embodiment (an example of replacement with a VR image). In FIGS. 12A to 12C, an MR experiencer 2 experiences a situation as if he or she were sitting in the driver's seat of a car. A dashboard 60, a front window 61, a rearview mirror 62, a steering wheel 63, and the like are displayed as 3DAR objects. Non-MR experiencers 4d and 4e (real objects) overlap with the front window 61 and are visible through the front window 61. The dashboard 60, the front window 61, the rearview mirror 62, and the steering wheel 63 are each parts that make up a 3DAR object 6 of the car. In this example, a 3DAR object 6, which is a single virtual reality object, is divided into multiple virtual reality object parts, and a flag specifying the type of occlusion processing is added to each part. The dashboard 60, the rearview mirror 62, and the steering wheel 63 are non-transparent areas of the 3DAR object 6, and therefore a non-transparent area flag is added to each of them. On the other hand, the front window 61 is a transparent area of ​​the 3DAR object 6, and therefore a transparent area flag is added to each of them.

[0090] FIG. 12A shows how the 3DAR object 6 appears when no occlusion processing is performed, and the non-MR users 4d and 4e overlap with the 3DAR object 6 in an unnatural manner.

[0091] Fig. 12B shows the result of occlusion processing. In the third embodiment, an occlusion flag described in Fig. 13 is set in the rendering data of the 3DAR object 6 on the front window 61, and processing to replace a real object overlapping the front window 61 with the 3DAR object 6 is prohibited, and occlusion processing is performed between the real object and the 3DAR object 6. As a result, the non-MR users 4d and 4e are observed through the front window 61. In this case, if the distance between the non-MR user 4d and the 3DAR object 6 is short and the non-MR user 4d is displayed behind the front window 61 in the same way as the non-MR user 4b (real object) in Fig. 6A, the non-MR user 4d (real object) is reduced in size and displayed as if it were farther away, like the non-MR user 4d in Fig. 12B. In addition, since the non-MR user 4e is farther away than the dashboard (3DAR object) 60, the steering wheel 63 (3DAR object), and the car body 3DAR object 6, the windshield 61 is treated as transparent, and occlusion processing is performed based on the distance relationship with other objects.

[0092] 12C shows the case where the background image of the real space is replaced with a VR image 6b. The dashboard 60, steering wheel 63, and VR image 6b are occluded, but the windshield 61 is a transparent or semi-transparent 3DAR object, and the VR image 6b behind the line of sight can be observed, providing an experience that makes one feel as if they are in the virtual place provided by the VR image.

[0093] 13 is a table 100 of an example of a 3DAR object. A 3DAR object is identified by a "CONTENTS ID," and multiple 3DAR objects (such as AR Objects 1 to 7) can be grouped together as related objects. Each 3DAR object includes a "Data ID" that is unique to the 3DAR object, a "Title" to make it easy to understand for users such as MR users, an "Occlusion Flag," and "3D Image Data."

[0094] "Occlusion Flag" defines "00", "01", and "10". When the value of "Occlusion Flag" is "00", occlusion processing is performed according to the distance between the real object and the 3DAR object. A flag with an "Occlusion Flag" value of "00" corresponds to a non-transparent part flag.

[0095] Also, if the value of "Occlusion Flag" is "01", and the distance between the real object and the 3DAR object is short, the real object is replaced with a background object to prevent the 3DAR object from being hidden. A flag with the value of "Occlusion Flag" being "01" corresponds to a non-transparent part flag.

[0096] When the value of "Occlusion Flag" is "10", the 3DAR object is treated as transparent regardless of the distance, like the windshield 61 in FIG. 12B, and occlusion processing is performed between the 3DAR object beyond the windshield 61 and the real object. A flag with an "Occlusion Flag" value of "10" corresponds to a transparent part flag. The 3DAR object beyond the windshield 61 may be part of the 3DAR object 6 of the automobile, such as the hood, or may be another 3DAR object different from the 3DAR object 6 of the automobile, such as the 3DAR object of another automobile.

[0097] FIG. 14 is a flowchart of the MR experience program 251 according to the third embodiment.

[0098] The difference from the flowchart in FIG. 7 is that steps S160 and S161 have been added.

[0099] In S160, the HMD 2a checks the "Occlusion Flag" of the 3DAR object, and performs different processing depending on the "Occlusion Flag", as described with reference to FIG.

[0100] If the "Occlusion Flag" is "00", the HMD 2a performs occlusion processing in S108 according to the distance relationship between the real object and the 3DAR object. (This processing will be described with reference to FIG. 15.)

[0101] If the "Occlusion Flag" is "10", the HMD 2a performs processing in S161 to treat the object as a transparent object like the windshield 61 shown in Fig. 12, so that the object can be seen through even if it is closer than the real object. Then, the process proceeds to S107.

[0102] If the "Occlusion Flag" is "01", the HMD 2a compares the distance between the real object and the 3DAR object in S107, and performs different processing depending on whether the real object and the 3DAR object are far apart, close to each other, or overlapping with each other.

[0103] FIG. 15 is an example of an image to which the "Occlusion Flag" is "00." FIG. 15 shows a situation in which a real object 4f is playing at the beach (there are 3DAR objects of a sandy beach 64 and a sea surface 65). In FIG. 15, the "Occlusion Flag" of the sea surface 65 is set to "00." There is no sense of incongruity even if only half of a person, which is real object 4f, appears from the sea surface 65. For this reason, occlusion processing should be performed based on the distance between the sea surface 65 and real object 4f.

[0104] As described above, the third embodiment has the same features as the first embodiment, and can also apply occlusion processing according to the features of the 3DAR object.

[0105] [Fourth embodiment] The fourth embodiment will be described with reference to Fig. 16 and Fig. 17. Fig. 16 is a block diagram of an MR support server 9 according to the fourth embodiment. The MR support server 9 of the fourth embodiment stores 3DAR object & VR image data 900, real space image data 901, and display image data 902 in a storage 94.

[0106] The MR support server 9 of the fourth embodiment is instructed by the HMD 2a and holds 3DAR object & VR image data 900 received from the VR service server 8, and real space image data 901 received from the HMD 2a.

[0107] Furthermore, the MR support server 9 recognizes and extracts, from the physical space image data 901, an object located on the far side (far side) in the depth direction on the line of sight of the physical object image 98 as a background object, and generates a background object image 97.

[0108] Furthermore, the MR support server 9 performs occlusion processing or the like on the real object image 98 and the 3DAR object & VR image data 900 to obtain display image data 902 that is synthesized with the real space image. The display image data 902 is transmitted to the HMD 2a and displayed on the display 22 of the HMD 2a. When background VR image data is received together with the image data of the 3DAR object, it is stored in the 3DAR object & VR image data 900.

[0109] The MR support server 9 includes a program for replacing a background image of a real object with VR image data, and for performing synthesis processing such as occlusion processing and movement of the real object between the background VR image, the 3DAR object, and the real object.

[0110] FIG. 17 is a flowchart of the MR assistance program 96 according to the fourth embodiment.

[0111] The MR support server 9 processes a login request from the registered MR user 2 (S131). Furthermore, the MR support server 9 receives real space image data from the HMD 2a (S132).

[0112] The MR support server 9 transmits a request for transmission of the rendering data of the 3DAR object to the VR service server (S140). The requested 3DAR object is determined by the user using the HMD 2a to determine what kind of AR content to synthesize, and receives an instruction to send the rendering data of the 3DAR object accordingly.

[0113] The MR support server 9 recognizes a real object from the real space image data received in S132 (S133), and updates the background image from the received real space image data (S134).

[0114] The MR support server 9 receives the rendering data of the 3DAR object and the VR image data (the VR image may not be received) (S141).

[0115] The MR support server 9 detects an overlap between the real object and the 3DAR object and compares the distance between the two objects (S142). If the two objects are far enough apart that their volumes do not overlap (S142: far apart), the MR support server 9 performs occlusion processing between the real object and the 3DAR object (S143).

[0116] On the other hand, if there is overlapping (S142: overlapping), the MR support server 9 generates data of a background object image from the background image (S144), and performs processing to overwrite and hide the real object with the background object image and the 3DAR object (S145).

[0117] The MR support server 9 checks whether steps S142 to S145 have been executed for all real objects that overlap with the 3DAR object, and if any real objects remain (S146: No), the process returns to S142.

[0118] If the processing is completed for all the physical objects (S146: Yes), the MR support server 9 transmits the processed image to the HMD 2a as display image data (S147).

[0119] The MR support server 9 checks whether the program has ended, and if it has not ended (S138: No), it continues the steps from S142 onward, whereas if it has ended (S138: Yes), it ends the series of processes.

[0120] The fourth embodiment has the same features as the first embodiment, and also has the advantage of allowing for flexible implementation, such as by executing most of the MR experience processing on the high-performance MR support server 9 and reducing the processing load on the HMD 2a.

[0121] The present invention is not limited to the embodiments described above with reference to Figures 1 to 17, and it is possible to replace part of the configuration of one embodiment with another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. These all fall within the scope of the present invention, and the numerical values, messages, etc. appearing in the text and figures are merely examples, and the use of different ones does not impair the effects of the present invention.

[0122] Furthermore, some or all of the functions of the invention may be implemented in hardware, for example, by designing an integrated circuit. They may also be implemented in software by a microprocessor unit, CPU, etc., interpreting and executing an operating program. Furthermore, the scope of software implementation is not limited, and both hardware and software may be used. [Explanation of symbols]

[0123] 1:MR space 1a: Access point 1b:Wireless LAN signal 2, 3: MR experience 2a, 3a: HMD 2b, 3b: Wireless LAN signal 4a, 4b, 4c, 4d, 4e: MR non-experienced 4f: Real objects 5a: vase 5b: Window 6: 3DAR object 6a: VR image data 6b: VR image 7: Network 8: VR service server 9: MR support server 10: Background object image 11: Image 11a:Foreground image 20: Range finding camera 20a: Left camera 20b: Right camera 22: Display 23: Shutter 24: Processor 25a, 25b: Mounting housing 26: Speaker 27:Mike 60: Dashboard 61: Front window 62: Rearview mirror 63: Handle 64: Sandy Beach 65: Sea level 82 :CPU 83:RAM 84: Storage 85: Internal bus 86:VR Service Program 87:VR data 91: Network Interface 92:CPU 93:RAM 94: Storage 95: Internal bus 96: MR Support Program 97, 98: Background object image 100: Table 240: Camera processor 241: Orientation sensor 242: Gyro sensor 243: Acceleration sensor 244: Radio communication device 245 :CPU 246:RAM 247: Image RAM 249: Internal bus 250: Basic Program 251:MR Experience Program 900: 3DAR object & VR image data 901: Real-space image data 902: Indicates portrait L: line of sight PR: Intersection Pn: Farthest point P1: Closest point

Claims

1. 1. A three-dimensional virtual reality display system, comprising: A server and a head-mounted display wirelessly connected to the server, The server a first processor; The head-mounted display includes: a camera that captures an image of a real object present in a real space and outputs three-dimensional real space image data including the real object image; a distance sensor that measures a distance from an observer in the real space to the real object and outputs distance data; The display and a second processor, transmitting the three-dimensional real space image data and the distance data from the head-mounted display to the server; the server receives the three-dimensional real space image data and the distance data; The first processor a photographing direction of the three-dimensional real space image data is used as a line of sight of the observer; If the physical object is present in the line of sight of the observer observing the three-dimensional virtual reality object, the distance at which the three-dimensional virtual reality object is displayed from the observer is compared with the distance from the observer to the physical object indicated in the distance data, and if the comparison shows that the distance at which the three-dimensional virtual reality object is displayed from the observer to the physical object in the same line of sight of the observer is the same as the distance from the observer to the physical object indicated in the distance data, it is determined that the physical object overlaps the three-dimensional virtual reality object, and if the physical object overlaps the three-dimensional virtual reality object, display image data that has been subjected to overlap resolution display processing is generated, in which the three-dimensional virtual reality object is displayed in the line of sight and the physical object image is not displayed in the line of sight; transmitting the display image data from the server to the head mounted display; the head-mounted display receives the display image data; the second processor displays an image including the three-dimensional virtual reality object on the display based on the display image data. A three-dimensional virtual reality display system.

2. 2. The three-dimensional virtual reality display system of claim 1, The first processor selects points that form the three-dimensional virtual reality object along the line of sight. The physical object is located at the same distance as or closer to the farthest point in the depth direction. If so, it is determined that the real object overlaps with the three-dimensional virtual object. A three-dimensional virtual reality display system.

3. 3. The three-dimensional virtual reality display system according to claim 2, The first processor determines whether the physical object is located farther on the line of sight than the farthest point. When placing the virtual object, the real object image is provided with an option for displaying the real object behind the virtual object. performing an exclusion process to generate the display image data; A three-dimensional virtual reality display system.

4. 3. The three-dimensional virtual reality display system according to claim 2, The first processor determines whether the physical object is at the same distance as the farthest point or closer than the farthest point. If the image of the real object exists, the image of the real object is deleted from the three-dimensional real space image, and the deleted area is the display image data in which a background object image generated based on the three-dimensional real space image is embedded. Generate A three-dimensional virtual reality display system.

5. 5. The three-dimensional virtual reality display system according to claim 4, The first processor is configured to position the object in the three-dimensional real space image at a position away from the line of sight. generating the display image data by moving the deleted physical object image; A three-dimensional virtual reality display system.

6. 3. The three-dimensional virtual reality display system according to claim 2, The first processor determines whether the physical object is positioned at the same distance as the farthest point or closer than the farthest point. When placing the object, the three-dimensional virtual reality object is superimposed on a three-dimensional virtual space image prepared in advance. generating display image data; A three-dimensional virtual reality display system.

7. 3. The three-dimensional virtual reality display system according to claim 2, The first processor selects points that form the three-dimensional virtual reality object along the line of sight. The real object is located at the same distance as the closest point in the depth direction or even closer to the closest point. When the real space exists, the real space is converted into a three-dimensional real space image captured by the camera. The object image is deleted, and the 3D virtual reality is added to the 3D real space image from which the real object has been deleted. The object is superimposed, and the remaining area is obtained by deleting the real object image from the three-dimensional real space image. the display image data in which a foreground image generated based on the three-dimensional real space image is embedded in the area; Generate A three-dimensional virtual reality display system.

8. 2. The three-dimensional virtual reality display system according to claim 1, The three-dimensional virtual reality object is configured to control an occlusion process for the three-dimensional virtual reality object. A flag is added to control The first processor selects the three-dimensional virtual reality object and the real object image in accordance with the flag. Controlling the execution of occlusion processing between the image and the A three-dimensional virtual reality display system.

9. 9. The three-dimensional virtual reality display system according to claim 8, the three-dimensional virtual reality object includes a non-transparent area and a transparent area; The flags include a non-transparent area flag attached to the non-transparent area and a transparent area flag attached to the transparent area. a transparent region flag; The first processor performs the process of generating a non-transparent region flag of the three-dimensional virtual reality object. The part executes the overlap resolution display process, and the three-dimensional image to which the transparent area flag is added is displayed. The transparent area of ​​the virtual reality object is transparent regardless of the distance between the transparent area and the real object. Another part or part of the three-dimensional virtual reality object is on the line of sight that passes through the area and observes the real object. If there is another three-dimensional virtual reality object and the real object, the three-dimensional virtual reality object a display according to the distance between another part of the three-dimensional virtual reality object or the real object, or Or, if only the physical object exists on the line of sight, a display according to the distance to the physical object generating the display image data for displaying the image data; A three-dimensional virtual reality display system.

Images