3D virtual reality display system

Abstract

This system provides a 3D virtual reality (AR) display system that more accurately recognizes 3D virtual reality (AR) objects. [Solution] The 3D virtual reality display system comprises a server and a head-mounted display (HMD) connected to the server via wireless LAN. The HMD captures 3D real-space image data of real objects and measures the distance from the observer in real space to the real objects, transmitting distance data to the server. The server uses the direction in which the 3D real-space image data is captured as the observer's line of sight. If a real object exists in the line of sight of the observer viewing the 3D AR object, the server compares the distance from the observer to the display of the 3D AR object with the distance from the observer to the real object indicated in the distance data. If the real object overlaps with the 3D AR object, the server generates display image data that performs overlap resolution display processing, displaying the 3D AR object in the line of sight while not displaying the real object image in the line of sight. The HMD then displays an image based on the display image data.

Description

[Technical Field]

[0001] This invention relates to a three-dimensional virtual reality display system, and more particularly to a system that displays real space and virtual reality objects (AR). Mixed reality (including objects: Argument Reality Objects) This concerns the technology used to experience MR (Mixed Reality). [Background technology]

[0002] Patent Document 1 describes a "display device that combines a real-world image with a virtual object image to create a composite image." The information processing device that outputs to the location identifies the location of the real object, and the information of the identified location of the real object Based on the information, it is determined whether the real object is moving, and if it is moving, the presence of the real object is determined. The technology outputs the composite image so that it can be displayed on the display device. (Summary excerpt) This has been disclosed. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2016-122392 [Overview of the project] [Problems that the invention aims to solve]

[0004] Patent Document 1 describes how a 3D AR object and a real object in real space are viewed from the line of sight of the HMD. If they overlap, make the 3D AR object semi-transparent, or in the vicinity of the real object. Restrictions can be placed on the display of 3D AR objects, such as hiding 3D AR objects. Therefore, it is possible to accurately recognize 3D AR objects and perform MR experiences. It may not be possible.

[0005] This invention has been made in view of the above problems, and aims to make 3D AR objects more accurate The objective is to provide recognition technology. [Means for solving the problem]

[0006] To solve the above problems, the present invention has the configuration described in the claims. To give an example, the present invention is a 3D virtual reality display system comprising a server and a head-mounted display wirelessly connected to the server, wherein the server comprises a first processor, the head-mounted display comprises a camera that captures images of real objects existing in real space and outputs 3D real space image data including images of real objects, a distance sensor that measures the distance from the observer in real space to the real object and outputs distance data, a display and a second processor, the head-mounted display transmits the 3D real space image data and the distance data to the server, the server receives the 3D real space image data and the distance data, the first processor divides a single 3D virtual reality object into multiple parts and holds 3D virtual reality object data with a non-transparent area flag or a transparent area flag attached to each part, and sets the shooting direction of the 3D real space image data to the observer's line of sight The system is used in such a way that, if the real object is in the line of sight of the observer observing the 3D virtual reality object, the distance from the observer to where the 3D virtual reality object is displayed is compared with the distance from the observer to the real object as shown in the distance data. If the 3D virtual reality object is in front of the real object, an overlap resolution display process is performed to display the portion of the 3D virtual reality object to which the non-transparent area flag is attached, and to display the real object image in the portion that overlaps with the portion of the 3D virtual reality object to which the transparent area flag is attached, and display image data is generated with the real object image displayed in the portion that overlaps with that portion. The display image data is transmitted from the server to the head-mounted display, the head-mounted display receives the display image data, and the second processor displays an image including the 3D virtual reality object on the display based on the display image data. [Effects of the Invention]

[0007] According to the present invention, 3D AR objects can be recognized more accurately. Other than the above The composition and effects will be clarified in the following embodiments.

Brief Description of Drawings

[0008] [Figure 1] Schematic diagram of a three-dimensional virtual reality display system according to the first embodiment [Figure 2] External view of an HMD as an example of a three-dimensional virtual reality display device [Figure 3] Block diagram of the HMD [Figure 4] Block diagram of the VR service server [Figure 5] Block diagram of the MR support server [Figure 6A] Diagram showing a conventional three-dimensional virtual reality display example (diagram showing a state where a 3DAR object and a real object overlap) [Figure 6B] Diagram showing three-dimensional virtual reality display according to the first embodiment (first overlap elimination display example) [Figure 6C] Diagram showing three-dimensional virtual reality display according to the first embodiment (second overlap elimination display example) [Figure 6D] Diagram showing three-dimensional virtual reality display according to the first embodiment (third overlap elimination display example) [Figure 7A] Flowchart of the MR experience program according to the first embodiment [Figure 7B] Diagram showing an example of an algorithm for judging the degree of volume overlap [Figure 8] Flowchart of the VR service server according to the first embodiment [Figure 9] Flowchart of the MR support server according to the first embodiment [Figure 10A] Diagram showing an example of a conventional three-dimensional virtual reality display (diagram showing a state where a real object is in front of a 3DAR object) [Figure 10B] Diagram showing three-dimensional virtual reality display according to the second embodiment (fourth overlap elimination display example) [Figure 11] Flowchart of the MR experience program according to the second embodiment [Figure 12A]This diagram shows an example of conventional 3D virtual reality display (a diagram showing a state where a 3DAR object and a real-world object overlap). [Figure 12B] Figure showing an example of a 3D virtual reality display according to the third embodiment (example of processing of transparent areas) [Figure 12C] Figure showing an example of 3D virtual reality display according to the third embodiment (example of replacement with VR image) [Figure 13] Diagram showing a 3DAR object table. [Figure 14] Flowchart for the MR experience program according to the third embodiment [Figure 15] A diagram showing another example of a three-dimensional virtual reality display according to the third embodiment. [Figure 16] Block diagram of the MR support server in the fourth embodiment [Figure 17] Flowchart showing the flow of 3D virtual reality display processing in the fourth embodiment [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. The same symbols are used for the components and steps, and redundant explanations are omitted.

[0010] In this embodiment, a 3D real-space image (hereinafter referred to as "real-space image") is captured in real space by a distance measuring camera. A 3D virtual image created with CG (Computer Graphics) on an interim image. Real-world objects (hereinafter referred to as "3DAR objects," and written as "3D-AR" in drawings) .) is combined and displayed. In this embodiment, by using the distance measuring camera 20, the camera and The distance sensor for measuring distance was integrated into the system, but the camera and, for example, the ultrasonic distance meter... It may also be configured by combining it with a separate distance sensor.

[0011] 3D AR objects are overlaid and displayed on a 3D real-world image obtained by capturing the real space. When doing so, the image that should be displayed on the side farther from the viewpoint will be displayed on the side closer to the viewpoint. This creates an occluded area. Occlusion is an image processing technique used to represent this occluded area. ru.

[0012] Mixed reality (MR) images, created by combining AR images with real-world background images, are used in games and maintenance. It is used in content such as work and sales promotion. To synthesize AR images For example, an AR marker is captured from a real-world image, and this subject area is used. AR images linked to AR markers are superimposed. Hardware that displays 3D virtual reality A is an HMD (Head-Mounted Display) which integrates a camera and a display. Since ) is often used, the following describes embodiments for implementing the present invention in an HMD. I will explain.

[0013] [First Embodiment] The first embodiment will be described with reference to Figures 1 to 9.

[0014] (Configuration of a 3D virtual reality display system) Figure 1 is a schematic diagram of a three-dimensional virtual reality display system according to the first embodiment.

[0015] In Figure 1, MR participant 2 wears HMD2a on their head and views the MR space 1. MR participant 3 wears HMD3a on their head and views MR space 1.

[0016] HMD2a and 3a respectively use access point 1a and wireless LAN signals 1b and 2b. 3b is transmitted and received to establish a wireless communication connection.

[0017] Access point 1a is located in MR space 1. It connects to network 7 outside MR space 1, and HMDs 2a, 3a and network 7 It communicates with the VR service server 8 and the MR support server 9 located at [location]. VR (Virtual Reality) refers to a virtual reality space.

[0018] In the real world, there are MR non-experienced individuals 4a and 4b. Also, as part of the background of the real world... There is a vase 5a and a window 5b. 3DAR object 6 is a 3D AR object of a car. be.

[0019] MR Space 1 is a space designed for car sales promotion, and is intended for a specific individual MR It's not a space limited to just one person experiencing it; multiple people, like MR experiencers 2 and 3, can experience MR simultaneously. The trial may be conducted. MR participant 3 is a product presenter and is the same 3DA as MR participant 2. It's also possible that R object 6 is being observed from a different direction, or from a different perspective than MR participant 2. It may also be an independent visitor observing a 3DAR object. By presenting it as a 3DAR object 6, the promoter can show expensive real-world objects (cars). There is no need to display the car itself, nor is there a need for a large space to display multiple cars. MR Space 1 In addition to MR experience participants 2 and 3, there may also be attendees 4a and 4b who have not experienced MR. Participants 4a and 4b are family members of MR participants 2 and 3, or are waiting to participate in the MR experience. These are visitors, etc.

[0020] Non-MR users 4a and 4b did not observe 3DAR object 6, and were not in MR space 1. Movement is not restricted. Therefore, like MR non-experienced person 4b, 3DAR object It's possible to be in the same position as 6.

[0021] Figure 2 shows external views of HMD2a and 3a as examples of 3D virtual reality display devices. The MD2a is a video-see-through type HMD. The HMD3a has the same configuration as the HMD2a. Therefore, the explanation will be omitted. The HMD2a has a parallax-equipped distance measuring camera 20 and a display It is equipped with 22. The HMD2a captures the foreground with the distance measuring camera 20 and places it in front of the MR experiencer 2. The 3D AR image, which is rendered using CG or other methods on the image captured by the distance measuring camera 20, is displayed on the placed display 22. Combine and display object 6.

[0022] The distance measuring camera 20 includes the left camera 20a and the right camera 20b, and the object being photographed This is a distance measuring camera for measuring the distance to [the object]. Display 22 is a flat-panel display. A shutter 23 is located inside it. The display 22 shows the image for the left eye and the image for the right eye. When these are displayed alternately, the shutter 23 opens and closes in sync with the display 22. That is, when the image for the left eye is displayed, the left half of display 22 opens and the right half closes. When an image for the eyes is displayed, the left half of display 22 closes and the right half opens. Therefore, HMD2a supports 3D display. MR participant 2 sees the displayed image. Synchronized, the display alternates between viewing the image with only one eye at a time.

[0023] Furthermore, the HMD2a includes a processor 24 and mounting housings 25a and 25b. Wearable housings 25a and 25b are used to attach the HMD2a to the head.

[0024] Images of the real space in front, captured by the left camera 20a and the right camera 20b, are displayed on the screen. Display 22 shows the real-world image to MR participant 2. Display 22 also displays 3DA The R object 6 is superimposed onto the real-world image. This is captured by the left camera 20a. The image for the left eye is added to the image taken with the right camera 20b, and the image for the right eye is added to the image taken with the right camera 20b. The image of the R object is superimposed and displayed on the display 22, and the 3DAR object 6 It displays the object three-dimensionally, as if it were at a predetermined distance in real space.

[0025] The HMD2a displays real objects in the real world, such as the non-MR user 4b and the vase 5a in Figure 1. The relative distance between window 5b and 3DAR object 6 is reflected. For example, a real object ( When a part of MR non-experienced person 4b) is in front of a part of 3DAR object 6 A portion of the 3DAR object 6 is hidden within a portion of the real object (MR non-experiencer 4b). To make it visible, occlusion processing is performed to modify the rendering data of 3DAR object 6. cormorant.

[0026] (Block diagram of a 3D virtual reality display device) Figure 3 is a block diagram of HMD2a. In Figure 3, the same numbers are used for components identical to those in Figure 2. In Figure 3, the processor 24 is the area enclosed by the dashed line, and the processor 24 is Left camera 20a, right camera 20b, display 22, shutter 23, speaker 26, m Iku27 is connected.

[0027] Processor 24 includes a camera processor 240, a compass sensor 241, and a gyroscope sensor 2 42, Accelerometer 243, Wireless communication device 244, CPU 245 (equivalent to the main processor) (to do), RAM246, image RAM247, Flash ROM(FROM)248, It includes an internal bus 249, and each element is connected to one another via the internal bus 249.

[0028] The wireless communication device 244 is used for several types of communication, including 4G, 5G, and wireless LAN. Select the appropriate processing from the available processing options to connect HMD2a to the network via access point 1a. Connect to twerk7.

[0029] FROM248 includes the basic program 250 and the MR experience program 251. 245 loads these processing programs into RAM246 and executes them. Furthermore, FROM2 48 stores the data necessary to execute the processing program. FROM248 is, Other non-volatile memory media besides Flash ROM may also be used.

[0030] The CPU 245 also sends the image data to the display 22 to the image RAM 247. After storing, read it.

[0031] The camera processor 240 processes the images captured by the left camera 20a and the right camera 20b. Based on this, the process of calculating the distance to the subject (corresponding to a real object) in the real-space image is performed, Distance data to the subject in the real-space image is added to the real-space image. "Real-space image" refers only to the image, while data with distance data added to it is called "real-space image". It is referred to as "image data."

[0032] The sensor group, including the compass sensor 241, gyro sensor 242, and acceleration sensor 243, is HM The position of D2a and the shooting direction of the distance measuring camera 20 (the line of sight of MR experiencer 2 wearing HMD2a and It is used to learn how it is used.

[0033] HMD2a runs on the VR service server 8 and MR support server 9 described below. This may include some or all of the processing.

[0034] Figure 4 is a block diagram of the VR service server 8. The VR service server 8 is wired L Network interface (network IF) 81, CPU 82, RAM It includes 83 and storage 84, which are connected to each other via an internal bus 85.

[0035] Storage 84 combines Flash ROM with hard disk drives, etc. It may be a modified version. Storage 84 stores the VR service program 86. PU82 deploys and executes the VR service program 86 in RAM83.

[0036] Furthermore, storage 84 stores VR data 87 such as 3DAR objects. Data 87 is the data required to execute the VR service program 86.

[0037] VR data 87 includes 3DAR objects as well as VR (Virtual Reality) (i) Image data may also be included. VR image data should include the real-world images of 2 or 3 MR participants. With an image that replaces the entire picture, MR participants 2 and 3 are transported to a different space provided by VR image data. This allows for an experience of observing 3DAR object 6 as if it were actually present. .

[0038] Figure 5 is a block diagram of the MR support server 9. The MR support server 9 is connected to a wired LAN, etc. It includes network IF91, CPU92, RAM93, and storage94, and these are internal They are connected to each other via bus 95.

[0039] Storage 94 combines Flash ROM with hard disk drives and other components. It may also be a MR support program. Storage 94 is a processing program, This includes 96. CPU92 loads the MR support program 96 into RAM93 and executes it.

[0040] Furthermore, storage 94 stores background object images 97 and real object images 98. These are M This is the data required to run the R support program 96.

[0041] The background object image 97 and the real-world object image 98 are data for the user to experience MR. When dealing with multiple users, separate data exists for each user.

[0042] The real object image 98 is obtained from the real space image received from HMD2a, by time difference, etc. This data detects areas of movement and recognizes a single area as a real object. This may involve detecting what a real object is, for example, a person, based on its shape and other characteristics. .

[0043] Background object image 97 is data of a background image obtained by removing the region of real objects from a real-space image. This is data from the motionless region of the real-space image. This involves supplementing data from real-world images taken in the past, specifically focusing on times when no real objects appeared in that region. Then the background image is obtained. More specifically, the background object image 97 is behind the real object image 98. Therefore, at a certain point in time, that is, the same frame of a 3D real space image consisting of multiple frames In frame 98 (the target frame), background objects that are further behind the real object image are not captured. Therefore, the MR support program 96 recognizes the real object image 98 from the target frame, By extracting the background object image 97 from other frames in which it is not visible, the background object image Generate image 97.

[0044] (Image displayed using 3D virtual reality) Figures 6A to 6D illustrate a method for displaying 3D virtual reality. Figure 6A shows a conventional 3D virtual reality display method. This figure shows an example of dimensional virtual reality display (showing a state where a 3DAR object and a real-world object overlap). Figures 6B and 6B show a 3D virtual reality display (first overlap-reducing display example) according to the first embodiment. Figure 6C shows a 3D virtual reality display (second overlap resolution display example) according to the first embodiment. Figure 6D shows a 3D virtual reality display (example of third overlap resolution display) according to the first embodiment. The diagrams are shown. Figures 6A to 6C are shown in relation to the area enclosed by the dashed line in Figure 1. They are responding. As shown in Figure 1, the real object is the MR non-experienced person 4b and the 3DAR object Effect 6 is located at roughly the same distance and overlaps in the line of sight of HMD2a. The "distance relationship" referred to here is the relationship between the HMD2a as the base point and the wearer of the HMD2a, M. This is the distance relationship along the same line of sight as R experiencer 2. However, this does not include distance relationships where the direction of gaze is different.

[0045] Figure 6A shows the overlapping MR non-experienced person 4b (real object) and 3DAR object 6. This is a cruising-processed image. It shows MR non-experienced person 4b in front of 3DAR object 6. The top part is displayed, but the bottom part of MR non-experienced person 4b behind 3DAR object 6 is not It does not appear. As a result, among the 3DAR object 6, MR non-experienced person 4b appears. The image shows (a person's upper body sitting on or growing out of the hood of a car). As a result, the unnatural image was created, which was hindering the recognition of 3DAR object 6 by person 4b who had not experienced MR. Therefore, in this embodiment, the process shown in Figure 6B or Figure 6C is performed.

[0046] Figures 6B and 6C show a real object (MR non-experiencer 4b) and a 3 object located at approximately the same distance. Do not perform occlusion processing with DAR object 6. Instead, the real object (M In the area of ​​R non-experienced user 4b), insert background object image 10 extracted from the background image, background object Occlusion processing is performed on image 10 and 3DAR object 6. Background object image 10 is These are typically located at a greater distance on the same line of sight than 3DAR object 6. It can display 3DAR object 6 without any missing parts, as well as real-world objects (MR non-experience). The background (e.g., vase 5a) that was hidden behind object 4b) is also displayed, enabling a natural MR experience. This completes the process shown in Figure 6B (first overlap elimination display).

[0047] In Figure 6C, in addition to Figure 6B, images of real objects (MR non-experiencer 4b) are displayed as 3D AR objects. Move (remove) the object 6 to a location that does not interfere with its recognition (second overlap elimination display). Figure 6C Image 11 shows the real object after movement (MR non-experienced person 4b). Real object (MR non-experienced person 4 b) In cases where the person is a child of two or three MR patients, or someone who should always be kept attentive, Without hiding the real object, 3DAR object 6 and the real object (MR non-experienced person 4b) It allows for simultaneous viewing and provides an MR experience without displaying unnatural images due to occlusion. It has the characteristic of maintaining a sense of immersion. Here, a real object (MR non-experiencer 4b) is moved. When doing so, the distance from the same line of sight as MR user 2 may differ, such as when the object is far away or nearby. In that case, the scale of the real object (non-MR participant 4b) will be adjusted according to the distance from MR participant 2. By processing it to be smaller or larger, it becomes possible to make it visible at a natural size. Yes.

[0048] Figure 6D shows an example where a real-world image is replaced with VR image data 6a (third example of overlay resolution). ) If the VR image data 6a is on the back of the 3DAR object 6, the 3DAR object Project 6 is displayed with VR image 6a as the background. Also, car sales promotion Depending on the scene, the display may switch from the video in Figure 6B or Figure 6C to the display shown in Figure 6D. This allows for the display of images combining various backgrounds and 3DAR objects, and the body The test subject will be able to see the 3DAR object 6 in various scenarios. (Figure 6B) Alternatively, when switching from the video in Figure 6C to the background video in Figure 6D, the images can be gradually combined. You can also change the image or apply processing such as a wipe effect.

[0049] (flowchart) Figure 7A is a flowchart of the MR experience program 251 according to the first embodiment. R participant 2 starts the MR experience program 251 stored in HMD2a (S101), Log in to VR service server 8 and MR support server 9. The following describes how HMD2a will perform an MR experience. The operation of program 251 during execution will be explained step by step. If MR non-experiencer 4b, as a real object, overlaps with or is overlapped with the car's 3DAR object 6 Let's explain using an example of what to do if it doesn't exist.

[0050] HMD2a starts taking pictures with its camera (S102). The image captured by the rangefinder camera 20 includes Distance data to the real object is attached. Camera shooting is, for example, 30fps (frames per second). A video is shot (per second) and multiple frames are arranged chronologically in a 3D real space Intermediate image data may be generated and captured images may be taken, and the following steps are: It may also be executed in sync with the camera's shooting cycle.

[0051] HMD2a transmits 3D real-world image data via wireless communication device 244 to MR-assisted servers. Send to B9 (S103). MR support server 9, as described below, uses real-space images. The images of real objects (4b, non-MR users) and background objects (e.g., vase 5a, window 5b) are separated. Let go.

[0052] Furthermore, HMD2a sends rendering data of 3DAR object 6 to VR service server 8. Send a request to send the VR data (included in VR data 87) (S104).

[0053] HMD2a receives at least one image extracted from the real-world space from the MR support server 9. Preferably, all real-world object image data (including real-world object images and their distance data) is received. S105, the VR service server 8 sends a 3DAR object (in this example, 3DAR The drawing data and VR image data of object 6) are received (S106).

[0054] HMD2a displays 3D objects (4b) and 3DAR objects (6) in relation to each real-world object (4b, who has not experienced MR). A similar overlap, or in other words, the same line of sight as when using HMD2a as the reference point. Images of real objects from (including images of MR non-experienced person 4b and distance data up to that point) and HMD Compare the distance from 2a to 3DAR object 6.

[0055] In a field where a real object and a 3D 3DAR object 6 are at the same distance from each other on the same line of sight... In this case, the volume of the real object and the volume of the 3DAR object overlap. Therefore, If occlusion is performed without considering the overlap of these volumes, for example, 3DAR objects Even if occlusion processing is successful between the area in front of the object 6 and the surface of the real object, 3DAR In the depths of Object 6, the distance relationship with the surface of the real object is not handled properly, 3 Sometimes, a real-world object may suddenly appear from within DAR object 6, resulting in an unnatural display.

[0056] In this embodiment, the degree of volume overlap between the real object and the 3DAR object 6 is determined accordingly. Choose between using the previous occlusion or the overlap resolution display process.

[0057] Therefore, HMD2a determines a distance where the volume of the real object does not overlap with the volume of the 3DAR object 6. If separated (S107: Separated), the real object and the 3DAR object will be separated. Execute John processing (S108).

[0058] On the other hand, with HMD2a, the volume of a real object (MR non-experiencer 4b) is the same as that of a 3D AR object. If the distance overlaps with the volume (S107: overlapping), HMD2a displays the overlap resolved. Perform the process.

[0059] Refer to Figure 7B to illustrate an example of the decision algorithm in step S107. Figure 7B This is a diagram illustrating an example of an algorithm for determining the degree of volume overlap. For the sake of explanation, HMD Assume that the display surface of 2a is a plane parallel to the vertical direction. And one point on display 22, for example The upper left corner of the display 22 is taken as the origin, and the two-dimensional coordinate system of the screen consists of the xy plane and the z plane perpendicular to it. We define the real 3D coordinates using the axes. Therefore, the zx plane is the horizontal plane, and the z axis Indicates the distance in the depth direction of the line of sight as seen from HMD2a.

[0060] In the example of FIG. 7B, when a real object is located in front of HMD2a, the value of the z-axis corresponds to the distance from HMD2a to MR non-experience person 4b. Since the distance measuring camera 20 visualizes MR non-experience person 4b the position P on the line of sight L from the observer of MR non-experience person 4b m , , n , , , n , l , n- ,

[0062] , n , , , , n , l , m , , , , , n , ,

[0061] , n-2 , (x R ,z R ) can be represented by the intersection of the surface facing HMD2a and the line of sight L in MR non-experience person 4b.

[0061] On the other hand, it is assumed that the shape of the 3DAR object 6 is defined by the three-axis coordinates (s, t, u) of the three-dimensional image system. When an AR marker appears in the real space, the 3DAR object 6 is overlaid and displayed thereon. Therefore, if the origin (s0, t0, u 0) of the 3DAR object 6 is overlaid on the three-dimensional coordinates (x , y l , z m , z n ) of the AR marker, then (s 0, t0, u0) can be converted to (x l , y m , z n ) for the sake of convenience of explanation, it is assumed that there is no deviation in the rotational direction of each axis between the stu coordinate system and the xyz coordinate system, and the s-axis coincides with the x-axis, the t-axis coincides with the y-axis, and the u-axis coincides with the z-axis .

[0062] If there is only one point constituting the 3DAR object 6 on the line of sight L of HMD2a, the processor 24 designates that point as the farthest point P , if there are a plurality of points, for example, P1, ···, P n , P n-2 , P n- 1, P n if there are, the point farthest from HMD2a, that is, the point with the largest value of the z-axis is designated as the farthest point P nSelect it as such. Note that point P1, which has the smallest z-axis value, is the nearest point.

[0063] Then, point P is the intersection of the line of sight L and the non-MR experiencer 4b. R 3D coordinates (x R , y R , z R ) and the farthest point P of 3DAR object 6 n coordinates (x ARn , y ARn , z ARn )(However And in this example, x R =x ARn , y R =y ARn ) Compare with z R >z ARn If so It is determined that there is no volume overlap between the real object and 3DAR object 6 (State 1). . z R If ≤ zar, there is an overlapping volume between the real object and the 3DAR object 6. It is determined that this is the case (State 2).

[0064] Therefore, HMD2a displays a background object image corresponding to the region of a real object (MR non-experienced person 4b) The data (corresponding to the background object image 10 in Figure 6B) is requested from the MR support server 9 (S109 ), receive (S110). After receiving, HMD2a displays background object image 10 and 3DAR object. The 6th unit hides real-world objects (MR non-experiencer 4b) (first overlap resolution display). Also, HMD2 'a' is a real object (MR non-experiencer 4b) cut out and placed in a position that does not overlap with 3DAR object 6. In addition to displaying it in the location, background objects are placed in the area where real objects (MR non-experiencers 4b) actually exist. Insert body image 10 (second overlap resolution display).

[0065] Furthermore, in S106, along with the rendering data of 3DAR object 6, the background VR image data If a signal is received, steps S107 and S108 are used to select a background object image from the real objects. Replace with R image, and with S108, the background VR image, 3DAR object and real object are used. This process performs compositing, including cruising and the movement of real objects. In this example, we are using HMD2a. The configuration was shown to perform the image compositing process within the HMD2a, but the compositing process is performed The locations are not limited to these, and as will be explained later, they can also be servers connected to a network, and collaborative locations. You can also process it using a connected smartphone or tablet.

[0066] HMD2a uses S1 for all real-world objects that overlap with 3DAR object 6. Check if steps 07-S111 have been executed, and if a real object remains, (S112: No), return to S107. Meanwhile, HMD2a for all real objects If processing is complete (S112: Yes), the processed image will be displayed on the HMD2a's display. Display on 22 (S113).

[0067] If the HMD2a MR experience program 251 has not finished, then S will be displayed in the next camera cycle. Continue from step 103 (S114: No). HMD2a MR experience program If 251 is finished (S114:Yes), the above process will terminate.

[0068] Figure 8 is a flowchart of the VR service program 86. Registered MR bodies Upon receiving a login request from test subject 2, the login process is executed (S121).

[0069] The VR service server 8 receives rendering data of the 3DAR object 6 from the HMD2a. Upon receiving a request (S122), create drawing data for the requested 3DAR object. (S123). The drawing data of the 3DAR object is the drawing of the 3DAR object. The distance between the HMD2a and the 3DAR object, and the HMD2a's field of view, are included in the data transmission request. Data (object files) obtained by 3D rendering of 3DAR object 6 according to line direction, etc. The rendering data is updated according to the movement of the HMD2a and changes in the user's gaze. The effects of reflection and shadows, based on the direction of light sources such as the sun and artificial lighting, are added to the image as rendering data. You can.

[0070] The VR service server 8 sends the created rendering data to the HMD2a (S124). .

[0071] VR service server 8 logs out MR user 2 and terminates MR experience program 251. Until the termination conditions of the VR service program 86 are met (S125: No), Continue processing steps S122 through S124.

[0072] When the termination condition of the VR service program 86 is met, the VR service server 8 will terminate (S1 25: Yes), terminate the above series of processes.

[0073] Figure 9 is a flowchart of the MR support program 96.

[0074] The MR support server 9 processes the login request of the registered MR user 2 (S13 1).

[0075] The MR support server 9 receives real-world spatial image data from the HMD2a (S132), and The object image is recognized (S133), real object image data is extracted, and a background image is obtained. The image is updated each time a real-space image is received (S134).

[0076] The MR support server 9 transmits real-world object image data to the HMD2a (S135). When the MR support server 9 receives a request to transmit background object images (S136), it transmits the background object images The data is sent to the HMD2a (S137).

[0077] MR support server 9 handles things like MR user 2 logging out and MR experience program 251 ending. However, until the completion conditions for MR support program 96 are met, steps S132 to S137 are completed. Continue processing (S138: No).

[0078] When the termination condition of the MR support program 96 is met, the MR support server 9 will terminate (S138:Y es) Then, terminate the above series of processes.

[0079] According to this embodiment, the real object and the 3DAR object 6 are on the same line of sight as the MR user. If they overlap, the distance is such that the volume of the real object and the volume of 3DAR object 6 do not overlap. In some cases, occlusion is performed, and if the two are close enough to overlap, occlusion is not performed. By performing overlap resolution processing, the real object and the 3DAR object 6 will not overlap unnaturally. This message is never displayed. Therefore, it can enhance the immersion of the MR experience.

[0080] Furthermore, according to this embodiment, there is a third party who does not participate in the MR experience (a person who has not experienced MR). Even in an open space, the shape of 3DAR object 6 will not be altered by a third party. This allows the MR user to accurately recognize 3DAR object 6 and perform the MR experience. .

[0081] [Second Embodiment] A second embodiment of the present invention will be described with reference to Figures 10A, 10B, and 11. Figure 10A shows an example of a conventional 3D virtual reality display (more than 3DAR object 6). This is a diagram showing a state where a real object is in the foreground. Figure 10B is a 3D hypothetical representation according to the second embodiment. This figure shows the representation of perceived reality (an example of the fourth overlap resolution representation).

[0082] As shown in Figure 10A, the real object is the MR non-experienced person 4b and the 3DAR object. 6 is located at approximately the same distance, and is also a real object, and 4c is a non-MR experiencer in 3DA. It is positioned in front of R object 6, and both overlap from the viewpoint of HMD2a. This is interfering with the observation of 3DAR object 6.

[0083] In Figure 10B, the non-MR participant 4b, who is behind the 3DAR object 6, is in the first implementation. Similar to the state, it is replaced with background object image 10. On the other hand, the other non-MR experiencers 4c are deleted and 3 Place DAR object 6. Furthermore, from the area where MR non-experienced person 4c was removed, 3 The remaining area where DAR object 6 does not overlap is obtained from the full-view image of other frames, and the remaining area is then... The corresponding image is extracted to generate a foreground image, which is then embedded in the remaining area. 3 DAR object 6 and background object image 10 are shown to other MR non-experienced participants 4c as 3DAR objects Object 6 corresponds to foreground image 11a. Background object image 10, foreground image 11a, and 3DAR Object 6 overwrites the real-world object images of MR non-experienced person 4b and other MR non-experienced person 4c. The process of hiding is performed (fourth example of overlap resolution display). As a result, 3DAR object 6 The entire structure becomes observable.

[0084] Figure 11 is a flowchart of the MR experience program 251 according to the second embodiment.

[0085] The difference from the flowchart in Figure 7 lies in the distance comparison of the steps in S150.

[0086] In S150, the distance between a real object and a 3DAR object is "behind and away from". The determination is made based on whether the object is "present" or "in a nearby position or in front," and in the former case, the real object Occlusion processing is performed on the body and 3DAR object 6 (S108), and in the latter, background objects A request is made to transmit the body image and foreground image (S151), and they are received (S152). Then, the background The real object image is obscured by the object image, 3DAR object 6, and foreground image (S153). In the example above, it is as if person 4b, who has not experienced MR, and another person 4c, who has not experienced MR, do not exist. Perform processing that makes it appear as if it were.

[0087] As explained above, the second embodiment has the same characteristics as the first embodiment, and Even if there is a real-world object in front of 3DAR object 6, 3DAR object 6 This allows for the removal of real-world objects that would interfere with the observation.

[0088] [Third Embodiment] A third embodiment will be described with reference to Figures 12A to 15B.

[0089] Figure 12A shows an example of a conventional 3D virtual reality display (3DAR object and real object). This is a diagram showing a state where bodies are overlapping. Figure 12B is a 3D virtual reality according to the third embodiment. This is a diagram showing an example of actual display (an example of processing of transparent areas). Figure 12C is a three-dimensional representation according to the third embodiment. This is a diagram showing an example of virtual reality display (an example of replacement with a VR image). Figures 12A to 12C show MR. Participant 2 is experiencing a situation similar to sitting in the driver's seat of a car. Dashboard 60 The front windshield 61, rearview mirror 62, steering wheel 63, etc. are 3DAR objects. It is displayed as [something]. For those who have not experienced MR, 4D and 4E (real objects) are displayed on the front window. It overlaps with 61 and is visible through the front windshield 61. Dashboard 60, front Each of the windows 61, rearview mirror 62, and steering wheel 63 is a 3D AR object of a car. This is a component of part 6. In this example, it is a 3DAR object, which is a single virtual reality object. Divide 6 into multiple parts of a virtual reality object and define the type of occlusion processing for each part. A flag is added. The dashboard 60, rearview mirror 62, and steering wheel 63 are all non-transparent areas of the 3DAR object 6, so a non-transparent area flag is added. On the other hand, the front window 61 is a transparent area of ​​the 3DAR object 6, so a transparent area flag is added.

[0090] Figure 12A shows the appearance without occlusion processing, and is for those who have not experienced MR. Objects 4d and 4e are unnaturally overlapping with 3DAR object 6.

[0091] Figure 12B shows the result after occlusion processing. In the third embodiment, The rendering data for the 3DAR object 6 on the front window 61 is as explained in Figure 13. The Occlusion Flag is set, and reality overlaps with the front windshield 61. Regarding objects, the process of replacing them with 3DAR object 6 is prohibited, and the real object and 3D Occlusion processing is performed on AR object 6. As a result, front window 6 Observe non-MR users 4d and 4e through 1. At this time, non-MR users 4d and 3DAR are observed. The distance to Ject 6 is close, and the MR non-experience 4d is displayed behind the front windshield 61. Therefore, if the appearance is the same as that of non-MR user 4b (real object) in Figure 6A, then the non-MR user The 4D (real object) is scaled down, so that it appears as if it is far away, like the MR non-experienced 4D in Figure 12B. Display. Note that MR non-experience users (4e) will see approximately 60 dashboard (3DAR objects). The handle 63 (3DAR object), and the car body 3DAR object 6 Because it is far away, the front windshield 61 is treated as transparent, and the other Occlusion processing is performed based on the distance relationship between objects.

[0092] Figure 12C shows the case where the background image of the real space is replaced with VR image 6b. The board 60, handle 63, and VR image 6b undergo occlusion processing, but the floor The view window 61 is a transparent or semi-transparent 3DAR object, and is visible from the viewer's perspective. The VR image 6b located there can be observed, and it is as if you are in the virtual location given by the VR image. You can have an eel-eating experience.

[0093] Figure 13 is Table 100 of an example of a 3DAR object. The object is identified by its "CONTENTS ID" and contains multiple 3DAR objects (AR Ob Objects 1-7, etc., can be grouped together as related items. The object has a unique "Data ID" specific to the 3DAR object, and information such as the MR user's information. In addition to "Title" to make it easier for users to understand, there is also "Occlusion Fla Includes "g" and "3D Image Data".

[0094] The “Occlusion Flag” defines “00”, “01”, and “10”. If the value of “Occlusion Flag” is “00”, then the real object and the 3DAR object Occlusion processing is performed according to the distance of the effect. A flag with a value of "00" corresponds to the non-transparent area flag.

[0095] Also, if the value of “Occlusion Flag” is “01”, then the real object and 3DAR When objects are close together, the real object is replaced with a background object, and the 3D AR object is displayed. Process the application so that it is not hidden. The value of "Occlusion Flag" is "01". This flag corresponds to the non-transparent area flag.

[0096] If the value of “Occlusion Flag” is “10”, then the front wall in Figure 12B Like the NdW61, it treats 3DAR objects as transparent regardless of distance, Occlusion processing is performed between the 3DAR object in front of the front window 61 and the real object. To perform the action. A flag with a value of "10" for "Occlusion Flag" is a transparent area flag. This corresponds to the 3DAR object in front of the front windshield 61, which is the 3DAR of the car. Part of R object 6, for example, the hood, or the 3D AR object of the car. Other 3DAR objects different from 6, for example, other car 3DAR objects That's fine.

[0097] Figure 14 is a flowchart of the MR experience program 251 according to the third embodiment.

[0098] The difference from the flowchart in Figure 7 is that steps S160 and S161 have been added. It is located there.

[0099] In S160, HMD2a performs "Occlusion F" on 3DAR objects. Check the “lag” and, as explained in Figure 13, the “Occlusion Flag” The processing will vary accordingly.

[0100] If the “Occlusion Flag” is “00”, HMD2a will send S108 Occlusion processing is performed according to the distance relationship between the real object and the 3DAR object. This process is explained in Figure 15.

[0101] If the “Occlusion Flag” is “10”, HMD2a will proceed to S161. Therefore, it is treated as a transparent object like the front window 61 shown in Figure 12, and is not a real object. The process is performed to allow the image to be seen through even at close range. After that, the program proceeds to S107.

[0102] If the “Occlusion Flag” is “01”, HMD2a will perform S107 The distance between real-world objects and 3DAR objects is compared, and the distances are different depending on whether they are far apart or close together. Different processes are performed depending on whether the condition is met or not.

[0103] Figure 15 shows an example of an image where the case where “Occlusion Flag” is “00” is applied. Yes. Figure 15 shows a beach (with 3DAR objects of sandy beach 64 and sea surface 65), and the real object Body 4f is in a state of idleness. Figure 15 shows the “Occlusion Flag” at sea surface 65. " is set to "00". Half of a person, which is a real object at 4f, emerges from the sea surface at 65. This does not create any sense of incongruity. Therefore, occlusion is determined by the relationship between the sea surface (65) and the distance of the real object (4f). You just need to make it so that it can be processed.

[0104] As explained above, the third embodiment has the same characteristics as the first embodiment, and Furthermore, occlusion processing can be applied according to the characteristics of the 3DAR object.

[0105] [Fourth Embodiment] A fourth embodiment will be described with reference to Figures 16 and 17. Figure 16 shows the fourth embodiment. This is a block diagram of the MR support server 9 related to the fourth embodiment. Range 94 contains 3DAR object & VR image data 900, and real-world image data 901. It also holds the display image data 902.

[0106] In the fourth embodiment, the MR support server 9 is instructed by the HMD2a and the VR service server 8 3DAR object & VR image data 900 received from and received from HMD2a It holds real-space image data 901 and

[0107] Furthermore, the MR support server 9 uses the real-space image data 901 to determine the line of sight of the real-world object image 98. Objects located towards the back (farthest side) in the depth direction are recognized and extracted as background objects. Generate body image 97.

[0108] Furthermore, the MR support server 9 displays real object images 98 and 3DAR object & VR image data. Display image data 90 is created by applying occlusion processing to the image data 900 and then compositing it onto a real-world image. Obtain 2. Display image data 902 is sent to HMD2a and displayed on HMD2a's display 2. It is displayed in 2. In addition, along with the image data of the 3DAR object, the background VR image data When receiving data, it is stored in 3DAR object & VR image data 900.

[0109] The MR support server 9 replaces the background image of real-world objects with VR image data, and the background V Occlusion processing and movement of real objects are performed when comparing R images, 3DAR objects, and real objects. It includes a program that performs synthesis processing.

[0110] Figure 17 is a flowchart of the MR support program 96 according to the fourth embodiment.

[0111] The MR support server 9 processes the login request of the registered MR user 2 (S13 1). Furthermore, the MR support server 9 receives real-world image data from the HMD2a (S1 32).

[0112] The MR support server 9 receives requests to send rendering data for 3DAR objects from the VR service Send to the server (S140). The requested 3DAR object is sent to the user on the HMD2a. It decides what kind of AR content to synthesize and then creates a 3D AR object accordingly. I received instructions to send the drawing data.

[0113] The MR support server 9 recognizes real-world objects from the real-world image data received in S132. S133) The background image is updated using the received real-world image data (S134).

[0114] The MR support server 9 receives rendering data of 3DAR objects and VR images (VR images are received). (Sometimes this does not happen) Data is received (S141).

[0115] The MR support server 9 detects the overlap between real-world objects and 3DAR objects, and , compare the distances between the two objects (S142). If they are far enough apart that their volumes do not overlap, (S142: Separate), occlusion processing between real objects and 3DAR objects Execute (S143).

[0116] On the other hand, if they overlap (S142: overlapping), the MR support server 9 will... Data of the background object image is generated from the image (S144), and the background object image and 3DAR object The system performs a process that overwrites and hides the real object (S145).

[0117] MR support server 9 will analyze all real-world objects that overlap with the 3DAR object. Check if steps S142 to S145 have been executed, and if a real object remains, (S146: No), return to S142.

[0118] MR support server 9 will complete processing for all real-world objects (S146:Ye s) The processed image is sent to the HMD2a as data for the display image (S147).

[0119] MR support server 9 checks for program termination, and if it does not terminate, (S138: No ), continue the steps from S142. If you want to finish (S138: Yes), The process will now terminate.

[0120] According to the fourth embodiment, it has the same characteristics as the first embodiment, and the MR experience processing is large Implementation methods include running a portion of the process on the high-performance MR support server 9 to reduce the processing load on the HMD2a. It has the characteristic of being flexible.

[0121] The present invention is not limited to the embodiments described in Figures 1 to 17 above, but is not limited to any particular embodiment. It is possible to replace some of the configuration of the implementation with that of other embodiments. It is also possible to add configurations from other embodiments to this configuration. All of these fall within the scope of the present invention. These are merely examples of what belongs to a particular category, and the numbers and messages that appear in the text and diagrams are also just examples. Using different materials will not impair the effects of the present invention.

[0122] Furthermore, the functions of the invention may be partially or entirely implemented by designing them, for example, using an integrated circuit. It can also be implemented in hardware. Furthermore, a microprocessor unit, CPU, etc., can operate. It may also be implemented in software by interpreting and executing the program. This does not limit the scope of implementation of the wearer; hardware and software can be used together. . [Explanation of Symbols]

[0123] 1:MR space 1a: Access point 1b:Wireless LAN signal 2, 3: MR experience participants 2a, 3a: HMD 2b, 3b: Wireless LAN signal 4a, 4b, 4c, 4d, 4e: MR non-experienced 4f: Real object 5a: vase 5b: Window 6: 3DAR Objects 6a: VR image data 6b: VR image 7: Network 8: VR Service Server 9: MR support server 10: Background object image 11: Image 11a:Foreground image 20: Rangefinder camera 20a: Left camera 20b: Right camera 22: Display 23: Shutter 24: Processor 25a, 25b: Mounting housing 26: Speaker 27: Microphone 60: Dashboard 61: Front Window 62: Rearview Mirror 63: Steering Wheel 64: Sandy Beach 65: Sea Surface 82: CPU 83: RAM 84: Storage 85: Internal Bus 86: VR Service Program 87: VR Data 91: Network IF 92: CPU 93: RAM 94: Storage 95: Internal Bus 96: MR Support Program 97, 98: Background Object Image 100: Table 240: Camera Processor 241: Azimuth Sensor 242: Gyro Sensor 243: Acceleration Sensor 244: Wireless Communicator 245: CPU 246: RAM 247: Image RAM 249: Internal Bus 250: Basic Program 251: MR Experience Program 900: 3DAR Object & VR Image Data 901: Real Space Image Data 902: Display Image Data L: Line of Sight PR: Intersection Point Pn: Farthest Point P1: Nearest Point

Claims

1. A 3D virtual reality display system, The system comprises a server and a head-mounted display wirelessly connected to the server, The aforementioned server, Equipped with a first processor, The aforementioned head-mounted display is A camera that captures images of real objects existing in real space and outputs 3D real-space image data including images of real objects, A distance sensor that measures the distance from the observer in the real space to the real object and outputs distance data, The display and A second processor is provided, The head-mounted display transmits the three-dimensional real-space image data and the distance data to the server. The server receives the three-dimensional real-space image data and the distance data, The first processor is, A single 3D virtual reality object is divided into multiple parts, and 3D virtual reality object data is stored with either a non-transparent area flag or a transparent area flag attached to each part. The direction in which the three-dimensional real-space image data is captured is used as the observer's line of sight. If the real object is located in the line of sight of the observer observing the 3D virtual reality object, the distance from the observer to where the 3D virtual reality object is displayed is compared with the distance from the observer to the real object as indicated in the distance data. If the 3D virtual reality object is in front of the real object, an overlap resolution display process is performed in which the part of the 3D virtual reality object to which the non-transparent area flag is attached is displayed, and the image of the real object is not displayed in the line of sight. For the part of the 3D virtual reality object to which the transparent area flag is attached, display image data is generated in which the image of the real object is displayed in the part that overlaps with that part. The server transmits the display image data to the head-mounted display. The head-mounted display receives the display image data, The second processor displays an image containing the three-dimensional virtual reality object on the display based on the display image data. A three-dimensional virtual reality display system characterized by the following features.

2. A three-dimensional virtual reality display system according to claim 1, The first processor determines that the real object overlaps the three-dimensional virtual reality object if the real object is located at the same distance as or closer to the furthest point in the depth direction along the line of sight among the points constituting the three-dimensional virtual reality object, A three-dimensional virtual reality display system characterized by the following features.

3. In the three-dimensional virtual reality display system according to claim 2, The first processor generates the display image data by performing an occlusion process on the image of the real object to display it behind the three-dimensional virtual reality object, if the real object is located further away in the line of sight than the furthest point. A three-dimensional virtual reality display system characterized by the following features.

4. In the three-dimensional virtual reality display system according to claim 2, The first processor, when the real object is at the same distance as or closer than the furthest point, deletes the real object image from the three-dimensional real space image captured by the camera, and generates display image data by inserting a background object image generated based on the three-dimensional real space image data into the deleted area. A three-dimensional virtual reality display system characterized by the following features.

5. In the three-dimensional virtual reality display system according to claim 4, The first processor generates the display image data by moving the real object image that has been removed to a position away from the line of sight in the three-dimensional real space image captured by the camera in the real space. A three-dimensional virtual reality display system characterized by the following features.

6. In the three-dimensional virtual reality display system according to claim 2, The first processor generates display image data by superimposing the three-dimensional virtual reality object onto a pre-prepared three-dimensional virtual space image when the real object is located at the same distance as or closer than the furthest point. A three-dimensional virtual reality display system characterized by the following features.

7. In the three-dimensional virtual reality display system according to claim 2, The first processor, when a real object exists at the same distance as or further in front of the nearest point in the depth direction along the line of sight among the points constituting the three-dimensional virtual reality object, deletes the real object image from the three-dimensional real space image captured by the camera, superimposes the three-dimensional virtual reality object onto the three-dimensional real space image from which the real object has been deleted, and generates display image data by inserting a foreground image generated based on the three-dimensional real space image into the remaining area from which the real object image was deleted. A three-dimensional virtual reality display system characterized by the following features.

8. In the three-dimensional virtual reality display system according to claim 1, The real object displayed behind the area to which the transparent region flag is attached will be displayed in a reduced size. A three-dimensional virtual reality display system characterized by the following features.

Images