Processing method of a display system, an imaging device, and a display system

Abstract

To provide an imaging apparatus which displays the video of a composite real space where the delay amount of the video of a real space is small and a display device which can display the video of the composite real space where the delay amounts of the video of the real space and the video of a virtual space are matched.SOLUTION: The imaging apparatus generates the first frame of the video of a first virtual space on the basis of the position attitude of the imaging apparatus based on the first frame of the video of the real space and composites a frame different from the first frame of the video of the real space and the first frame of the video of the first virtual space. The display device generates the first frame of the video of a second virtual space on the basis of the position attitude of the imaging apparatus based on the first frame of the video of the real space and composites the first frame of the video of the real space and the first frame of the video of the second virtual space.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a display system, an imaging device, a method for processing a display system, and a method for processing an imaging device.

Background Art

[0002] In recent years, as a technology for seamlessly integrating the real world and the virtual world in real time, there is a mixed reality (MR) system. According to the mixed reality system, an object can be superimposed and displayed in the real world as a CG (computer graphics) video. And it can be confirmed from a free viewpoint.

[0003] In the mixed reality system, as one of the video display devices, a head-mounted display (hereinafter referred to as HMD: head-mounted display) in which an imaging device such as a video camera and a display are integrated is used. As one form of the HMD, there is a video see-through method. This is a method of superimposing and displaying an image of a virtual space generated according to the position and orientation of the HMD on an image of the real space captured by the imaging device mounted on the HMD. Examples of images of the virtual space include, for example, virtual objects and character information drawn by CG (Computer Graphics).

[0004] In the design process of the manufacturing industry, reviews participated by many people are often held. In the review using the mixed reality system, in order for many people to experience the mixed reality space (a space synthesized from a virtual space and a real space), in addition to the HMD, a system is configured by combining a portable information terminal such as a tablet. This system distributes and displays the video of the mixed reality space viewed by the wearer of the HMD to other HMDs and a plurality of tablet screens at the same time. Thereby, the vision of the wearer of the HMD can be shared by a plurality of people at the same time, and they can participate in the design and discussion.

[0005] In generating images in the virtual space, there is a delay in displaying an image that reflects the current HMD position and orientation due to the calculation of the HMD's position and orientation, and the rendering of CG according to that position and orientation. If the delay is large in MR, the user will feel a sense of unease due to the discrepancy between their actions and the display, which can cause motion sickness. Therefore, even if the delay amount differs from that of the virtual space image, the real-world image will be displayed with as little delay as possible.

[0006] Patent Document 1 discloses a technique for reducing the delay of virtual space images by reacquiring the position and orientation of the HMD after generating an image of the virtual space, and then shifting the generated virtual space image according to the difference between the position and orientation used when generating the virtual space image. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2008-299669 [Overview of the project] [Problems that the invention aims to solve]

[0008] However, when streaming and displaying the mixed reality video seen by the HMD wearer, the movement in the shared video does not reflect the viewer's own movements. Therefore, the sense of unease due to the delay before the HMD wearer's movements are reflected is small, and the sense of unease due to the difference in delay between the video in the real world and the video in the virtual world is perceived as much greater.

[0009] The purpose of this disclosure is to enable the imaging device to display images of a mixed reality space with a low delay in the images of the real space, and the display device to display images of a mixed reality space where the delays of the images of the real space and the virtual space are matched. [Means for solving the problem]

[0010] The display system is a display system having an imaging device and a display device, wherein the imaging device has imaging means for imaging images of real space, a first position and orientation calculation means for calculating the position and orientation of the imaging device based on the images of real space, a first generation means for generating images of a first virtual space based on the position and orientation of the imaging device, a first image synthesis means for generating images of a first mixed reality space by combining the images of real space captured by the imaging means and the images of the first virtual space, and for displaying the images of the first mixed reality space, and a first communication means for transmitting at least the images of real space captured by the imaging means to the display device, wherein the first position and orientation calculation means calculates the position and orientation of the imaging device based on a first frame of the images of real space, and the first generation means generates images of the first virtual space based on the position and orientation of the imaging device based on a first frame of the images of real space The first image synthesis means generates a first frame of the real space image, and synthesizes a frame different from the first frame of the real space image with the first frame of the first virtual space image. The display device includes a second communication means for receiving the real space image from the imaging device, a second generation means for generating the second virtual space image based on the position and orientation of the imaging device, and a second image synthesis means for generating and displaying the second mixed reality image by synthesizing the real space image received by the second communication means with the second virtual space image. The second generation means generates a first frame of the second virtual space image based on the position and orientation of the imaging device based on the first frame of the real space image, and the second image synthesis means synthesizes the first frame of the real space image with the first frame of the second virtual space image. [Effects of the Invention]

[0011] According to this disclosure, the imaging device can display images of a mixed reality space with a low delay in the images of the real space, and the display device can display images of a mixed reality space where the delays of the images of the real space and the virtual space are matched. [Brief explanation of the drawing]

[0012] [Figure 1] This is a diagram showing an example of the display system configuration. [Figure 2] This is a block diagram showing an example of the functional configuration of a display system. [Figure 3] This flowchart shows an example of HMD processing. [Figure 4] This is a flowchart showing an example of processing for a display device. [Figure 5] This diagram illustrates video latency. [Figure 6] This block shows an example of the functional configuration of an HMD. [Figure 7] This flowchart shows an example of HMD processing. [Modes for carrying out the invention]

[0013] (First embodiment) The embodiments will be described in detail below with reference to the drawings. Figure 1 is a diagram showing an example configuration of a display system 500 according to the first embodiment. The display system 500 includes an HMD (head-mounted display) 100, a display device 200, and a CG (computer graphics) data management server 300. The HMD 100 is an example of a head-mounted display device, and is a device that presents a mixed reality image, which is a composite of images of the real world and images of the virtual world, in front of the eyes of a user wearing the HMD 100 on their head. The CG data management server 300 is a device that holds the CG data necessary to construct the virtual world. The HMD 100 and the display device 200 access the CG data management server 300 via a network, acquire CG data of the virtual world, generate images of the virtual world, and display the images of the virtual world, which are a composite of the virtual world images and images of the real world.

[0014] The HMD 100 and the display device 200 communicate with each other via a network, and can transfer the video of the real space captured by the HMD 100 and the information for generating the virtual space from the HMD 100 to the display device 200. The display device 200 can use the transferred information to generate a video of the composite reality space, and share and display the view of the wearer of the HMD 100 on the display device 200.

[0015] Note that the HMD 100 may be a system separated into a head-mounted part and an image processing device such as a personal computer (PC). Also, the HMD 100 may be a device such as a smartphone that displays the video of the real space captured in the hand on a display.

[0016] The display device 200 may be an HMD, or may be a terminal such as a PC (personal computer) or a tablet that can synthesize and display the video of the real space and the video of the virtual space. Also, there may be a plurality of display devices 200, and the video may be shared from the HMD 100 to all of them.

[0017] The HMD 100 and the display device 200 may be arranged in one space where each user can directly see each other, or may be arranged distantly where they cannot directly see each other. Also, the CG data management server 300 may be integrated with the HMD 100.

[0018] In this embodiment, the HMD 100 captures the video of the real space and calculates the position and orientation of the HMD 100. Then, the HMD 100 uses the CG data acquired from the CG data management server 300 to generate the video of the virtual space so as to be the video seen from the calculated position and orientation, and generates the video of the composite reality space by synthesizing the captured video of the real space and the generated video of the virtual space. Then, the HMD 100 displays the generated video of the composite reality space on the display unit of the HMD 100, and transmits the captured video of the real space and the position and orientation of the HMD 100 to the display device 200.

[0019] The display device 200 uses the CG data acquired from the CG data management server 300 to generate an image of a virtual space so as to be an image seen from the received position and orientation of the HMD 100, and synthesizes and displays the image of the virtual space and the received image of the real space. In this way, the HMD 100 and the display device 200 can share the image of the composite reality space.

[0020] FIG. 2 is a block diagram showing a functional configuration example of the display system 500. The display system 500 includes the HMD 100, the display device 200, and the CG data management server 300.

[0021] The HMD 100 includes an imaging unit 101, a position and orientation calculation unit 102, a CG data acquisition unit 103, a CG rendering unit 104, an image synthesis unit 105, an image display unit 106, and a communication unit 107.

[0022] The imaging unit 101 captures an image of the real space with the camera mounted on the HMD 100 for each frame. The camera can capture an image corresponding to the position of the eyes of the wearer of the HMD 100, and can capture an image for the right eye and an image for the left eye respectively. Note that the images and positions and orientations generated below are generated for each of the right eye and the left eye. The imaging unit 101 captures images of the real space for the right eye and the left eye.

[0023] The position and orientation calculation unit 102 calculates the position and orientation for the right eye and the left eye of the HMD 100 for each frame. Here, the position and orientation are the position in three dimensions and the orientation in three dimensions. There are various methods for calculating the position and orientation, and any method can be adopted. In the present embodiment, the position and orientation calculation unit 102 calculates the position and orientation for the right eye and the left eye for each frame from the images of the real space for the right eye and the left eye captured by the imaging unit 101 by a technique called VISUAL SLAM. VISUAL SLAM is a technique for calculating the position and orientation of a camera by detecting feature points from an image of the real space and tracking the temporal change of those points.

[0024] The CG data acquisition unit 103 acquires CG data for constructing a virtual space from an external CG data management server 300.

[0025] The CG rendering unit 104 receives the CG data acquired by the CG data acquisition unit 103. Then, for each frame, the CG rendering unit 104 renders the CG data using the position and orientation for the right and left eyes calculated by the position and orientation calculation unit 102 as the viewpoint, and generates virtual space images for the right and left eyes. The field of view during rendering is matched to the field of view of the image captured by the imaging unit 101.

[0026] The image synthesis unit 105 synthesizes, frame by frame, the images of the real space for the right eye and the left eye captured by the imaging unit 101, and the images of the virtual space for the right eye and the left eye generated by the CG rendering unit 104, to generate images of the mixed reality space for the right eye and the left eye.

[0027] The video display unit 106 displays the mixed reality space images for the right and left eyes, generated by the video synthesis unit 105, on the screens in front of the right and left eyes of the HMD 100, frame by frame.

[0028] The communication unit 107 transmits, frame by frame, the images of the real space for the right and left eyes captured by the imaging unit 101, and the position and orientation of the right and left eyes calculated by the position and orientation calculation unit 102 to the display device 200. The HMD 100 is the source of the sharing, and the display device 200 is the recipient.

[0029] The display device 200 includes a communication unit 201, a video delay unit 202, a CG data acquisition unit 203, a CG rendering unit 204, a video synthesis unit 205, and a video display unit 206.

[0030] The communication unit 201 receives, frame by frame, images of the real world for the right and left eyes transmitted from the HMD 100, as well as positional orientations for the right and left eyes.

[0031] The video delay unit 202 buffers the real-world video for the right and left eyes received by the communication unit 201 for each frame, and holds the frames of the real-world video up to the present. The video delay unit 202 can adjust the amount of delay in the real-world video by selecting the frames of the real-world video to be used for synthesis according to the number of frames that the video synthesis unit 205 (described later) wants to delay.

[0032] The CG data acquisition unit 203 acquires CG data for constructing a virtual space from an external CG data management server 300.

[0033] The CG rendering unit 204, for each frame, uses the position and orientation for the right and left eyes received by the communication unit 201 as the viewpoint and renders the CG data acquired by the CG data acquisition unit 203 to generate virtual space images for the right and left eyes. The field of view during rendering is matched to the field of view of the image captured by the imaging unit 101 of the HMD 100.

[0034] The video synthesis unit 205 synthesizes, frame by frame, the real-world images for the right and left eyes held by the video delay unit 202 with the virtual-world images for the right and left eyes generated by the CG rendering unit 204 to generate mixed-reality images for the right and left eyes.

[0035] The video display unit 206 displays the mixed reality space images for the right eye and left eye, generated by the video synthesis unit 205, on the screens in front of the right and left eyes of the display device 200.

[0036] Figure 3 is a flowchart showing the processing method of the HMD100, illustrating an example of processing for each frame of the HMD100. Figure 4 is a flowchart showing the processing method of the display device 200, illustrating an example of processing for each frame of the display device 200. Figure 5 is a diagram illustrating the delays at each processing stage in the HMD100 and the display device 200. The following describes the processing flow of the HMD100 and the display device 200, and the delays at each processing stage, following the flowcharts in Figures 3 and 4. The HMD100 starts processing according to the flowchart in Figure 3 at a predetermined timing each frame. The display device 200 starts processing according to the flowchart in Figure 4 at a predetermined timing each frame.

[0037] Here, delay refers to the time it takes for changes in the real world, such as camera movement or changes in the subject, to be reflected in the image. In the case of real-world images, delay is the time it takes for the captured image frame to be displayed. In the case of virtual-world images, delay is the time from the calculated position and orientation time until the virtual-world image reflecting that position and orientation is displayed.

[0038] The horizontal axis in Figure 5 represents time (frame). The upper part of Figure 5 shows the image of the real world captured by the HMD100, the position and orientation calculated by the HMD100, the image of the virtual world generated by the HMD100, and the image of the mixed reality world generated by the HMD100. The lower part of Figure 5 shows the image of the real world received by the display device 200, the position and orientation received by the display device 200, the image of the virtual world generated by the display device 200, and the image of the mixed reality world generated by the display device 200.

[0039] The timestamps displayed on the HMD100 and display device 200 for the real-world image, position and orientation, the virtual-world image, and the mixed-reality image indicate which time frame the generated image corresponds to.

[0040] Here, let ts be the time it takes for the HMD100 to calculate its position and orientation from the image in real space. Let tr be the rendering time for the CG rendering units 104 and 204, respectively. Let tn be the time it takes for the HMD100 to transmit the image in real space and its position and orientation to the display device 200. Figure 5 shows an example where ts=1 frame, tr=2 frames, and tn=1 frame.

[0041] In step S101 in Figure 3, the imaging unit 101 captures images of the real space for the right eye and the left eye for each frame. At time t in Figure 5, the frame of the real space image at time t is captured.

[0042] In step S102, the position and orientation calculation unit 102 calculates the position and orientation of the right and left eyes of the HMD 100 based on the real-world images for the right and left eyes captured by the imaging unit 101 for each frame. For example, the position and orientation calculation unit 102 calculates the position and orientation at time t based on the real-world image frame at time t. Since the calculation of the position and orientation takes time ts, the position and orientation at time t is calculated at time t+ts. Here, ts = 1 frame.

[0043] In step S103, the communication unit 107 transmits the video frame of the real space at time t and its position and orientation at time t to the display device 200. Transmission takes time tn. At time t+ts in Figure 5, the communication unit 107 transmits the video frame of the real space at time t and its position and orientation at time t to the display device 200. At time t+ts+tn, the communication unit 201 of the display device 200 receives the video frame of the real space at time t and its position and orientation at time t. Here, tn = 1 frame.

[0044] In step S104, the CG data acquisition unit 103 acquires CG data from the CG data management server 300.

[0045] In step S105, the CG rendering unit 104 renders the CG data acquired by the CG data acquisition unit 103, using the position and orientation for the right and left eyes calculated by the position and orientation calculation unit 102 as viewpoints for each frame, and generates virtual space images for the right and left eyes. For example, the CG rendering unit 104 renders the CG data using the position and orientation at time t as the viewpoint, and generates a virtual space image at time t. Since rendering takes tr time, a frame of the virtual space image at time t is generated at time t + ts + tr. Here, tr = 2 frames.

[0046] In step S106, the video synthesis unit 105 synthesizes, frame by frame, the real-world images for the right and left eyes captured by the imaging unit 101 and the virtual-world images for the right and left eyes generated by the CG rendering unit 104. The video synthesis unit 105 generates mixed-reality images for the right and left eyes through frame by frame synthesis. The video display unit 106 displays the mixed-reality images for the right and left eyes generated by the video synthesis unit 105 on screens in front of the right and left eyes of the HMD 100, frame by frame.

[0047] For example, the video synthesis unit 105 combines a video frame from the real space at time t+3 with a video frame from the virtual space at time t to generate a video frame from the mixed reality space at time t. The video display unit 106 displays the video frame from the mixed reality space at time t on screens in front of the right and left eyes of the HMD 100.

[0048] At time t+ts+tr, the imaging unit 101 captures a frame of the real-world image at time t+3, and the CG rendering unit 104 generates a frame of the virtual-world image at time t. At time t+ts+tr+1, the image synthesis unit 105 synthesizes the frame of the real-world image at time t+3 and the frame of the virtual-world image at time t to generate a frame of the mixed-reality image at time t. Here, in order to reduce the delay of the real-world image, the image synthesis unit 105 synthesizes the latest frame of the real-world image at time t+3 and the frame of the virtual-world image at time t.

[0049] In step S201 of Figure 4, the communication unit 201 receives real-world images for the right and left eyes, and position and orientation for the right and left eyes, from the HMD 100 for each frame. At time t+ts+tn, the communication unit 201 receives the frame of the real-world image at time t and the position and orientation at time t.

[0050] In step S202, the video delay unit 202 buffers the real-world video for the right eye and left eye received by the communication unit 201 for each frame, and holds the frames of the real-world video for the right eye and left eye up to the present.

[0051] In step S203, the CG data acquisition unit 203 acquires CG data from the CG data management server 300.

[0052] In step S204, the CG rendering unit 204 renders the CG data acquired by the CG data acquisition unit 203, using the position and orientation for the right and left eyes received by the communication unit 201 as the viewpoint for each frame, and generates virtual space images for the right and left eyes. For example, the CG rendering unit 204 renders the CG data using the position and orientation at time t as the viewpoint, and generates a virtual space image at time t. Since rendering takes tr time, the frames of the virtual space image at time t are generated at time t+ts+tr+tn.

[0053] In step S205, the video synthesis unit 205 synthesizes, frame by frame, the real-world images for the right and left eyes held by the video delay unit 202 with the virtual-world images for the right and left eyes generated by the CG rendering unit 204. The video synthesis unit 205 generates mixed-reality images for the right and left eyes through frame by frame synthesis. The video display unit 206 displays the mixed-reality images for the right and left eyes generated by the video synthesis unit 205 on the screens in front of the right and left eyes of the display device 200, frame by frame.

[0054] At time t+ts+tr+tn, the image synthesis unit 205 synthesizes the image frame from the real space at time t with the image frame from the virtual space at time t to generate the image frame from the mixed reality space at time t. The image display unit 206 displays the image frame from the mixed reality space at time t on the screens in front of the right and left eyes of the display device 200.

[0055] The movement of the image displayed by the image display unit 206 does not reflect the actions of the viewer of the display device 200. Therefore, for the viewer of the display device 200, the sense of unease due to the delay before the actions of the HMD 100 are reflected is small, and the sense of unease due to the difference in the amount of delay between the image in the real space and the image in the virtual space is felt more strongly. Therefore, the image synthesis unit 205 synthesizes the frame of the image in the real space at time t with the frame of the image in the virtual space at time t. As a result, the image display unit 206 can display an image in which the amount of delay between the image in the real space and the image in the virtual space matches.

[0056] As described above, the image synthesis unit 105 of the HMD100 synthesizes the image frame of the real space at time t+3 and the image frame of the virtual space at time t to generate the image frame of the mixed reality space at time t. As a result, the mixed reality image on the HMD100 reduces the amount of delay in the real space image caused by the movement of the HMD100 wearer, thereby reducing discomfort for the HMD100 wearer.

[0057] Furthermore, the image synthesis unit 205 of the display device 200 synthesizes the frames of the real-world image at time t with the frames of the virtual-world image at time t to generate the frames of the mixed-reality image at time t. As a result, the mixed-reality image on the display device 200 has the same delay amount as the frames of the real-world image and the frames of the virtual-world image, which reduces the sense of discomfort for the viewer of the display device 200.

[0058] In step S102, the position and orientation calculation unit 102 may predict and calculate the position and orientation at time t+3, taking into account the time until display. In that case, in step S105, the CG rendering unit 104 renders the CG data using the position and orientation at time t+3 as the viewpoint, and generates an image of the virtual space at time t+3. Even in this case, in step S102, the position and orientation calculation unit 102 also calculates the position and orientation at time t. Then, in step S103, the communication unit 107 transmits the position and orientation at time t to the display device 200. In this way, if the above prediction is correct, the HMD 100 can generate an image of a mixed reality space that makes the delay of the image of the virtual space appear small. The display device 200 can generate an image of a mixed reality space based on the position and orientation of the HMD 100, which is not based on future predictions.

[0059] Furthermore, at time t+3 in step S105, the CG rendering unit 104 generates a frame of the virtual space image at time t. Subsequently, the position and orientation calculation unit 102 may recalculate the current position and orientation at time t+3. Then, based on the difference between the position and orientation at time t and the position and orientation at time t+3, the CG rendering unit 104 deforms the above-mentioned frame of the virtual space image at time t and generates a frame of the virtual space image at time t+3. In step S106, the image synthesis unit 105 synthesizes the frame of the real space image at time t+3 with the frame of the virtual space image at time t+3. In this way, the HMD 100 can generate a mixed reality image that appears to minimize the amount of delay as much as possible by deforming the virtual space image after its generation.

[0060] The video synthesis unit 205 may also select a frame from the real-world video frames that has a predetermined delay amount. Alternatively, the HMD 100 may assign a timestamp to the real-world video and position / orientation for each frame and transmit it to the display device 200. In this case, the display device 200 can select a real-world video frame using the real-world video frame and position / orientation with matching timestamps, generate a virtual-world video frame, and synthesize the real-world and virtual-world video frames.

[0061] Furthermore, the display device 200 may be equipped with a position and orientation calculation unit that calculates the position and orientation of the HMD 100 based on the frames of real-world video received by the communication unit 201. In that case, the CG rendering unit 204 can generate frames of virtual-world video based on the calculated position and orientation.

[0062] As described above, the display system 500 includes an HMD 100 and a display device 200. The HMD 100 is an imaging device and includes an imaging unit 101, a position and orientation calculation unit 102, a CG data acquisition unit 103, a CG rendering unit 104, an image synthesis unit 105, an image display unit 106, and a communication unit 107.

[0063] The imaging unit 101 captures images of the real world. The position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 based on the images of the real world. The CG rendering unit 104 is a generation unit and generates images of the first virtual world based on the position and orientation of the HMD 100. The image synthesis unit 105 generates images of the first mixed reality space by combining the images of the real world captured by the imaging unit 101 with the images of the first virtual world, and displays the images of the first mixed reality space on the image display unit 106. The communication unit 107 transmits at least the images of the real world captured by the imaging unit 101 to the display device 200.

[0064] The position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 based on the first frame (for example, the frame at time t) of the above-mentioned real-world video. The CG rendering unit 104 generates the first frame of the first virtual space video based on the position and orientation of the HMD 100 based on the first frame of the above-mentioned real-world video. The video synthesis unit 105 synthesizes a frame different from the first frame of the above-mentioned real-world video (for example, the frame at time t+3) with the first frame of the above-mentioned first virtual space video.

[0065] The display device 200 includes a communication unit 201, a video delay unit 202, a CG data acquisition unit 203, a CG rendering unit 204, a video synthesis unit 205, and a video display unit 206. The communication unit 201 receives video of the real world from the HMD 100. The CG rendering unit 204 is a generation unit that generates video of a second virtual world based on the position and orientation of the HMD 100. The video synthesis unit 205 generates video of a second mixed reality world by combining the video of the real world received by the communication unit 201 with the video of the second virtual world, and displays the video of the second mixed reality world on the video display unit 206.

[0066] The CG rendering unit 204 generates a first frame of the second virtual space image based on the position and orientation of the HMD 100, which is based on the first frame of the above-mentioned real-world space image. The image synthesis unit 205 synthesizes the first frame of the above-mentioned real-world space image and the first frame of the above-mentioned second virtual space image.

[0067] For example, the communication unit 107 transmits to the display device 200 a first frame of the real-world image captured by the imaging unit 101 and the position and orientation of the HMD 100 based on the first frame of the real-world image. The communication unit 201 receives the first frame of the real-world image and the position and orientation of the HMD 100 based on the first frame of the real-world image from the HMD 100. The CG rendering unit 204 generates a first frame of the second virtual space image based on the position and orientation of the HMD 100 based on the first frame of the real-world image received by the communication unit 201.

[0068] The display device 200 may also have a second position and orientation calculation unit that calculates the position and orientation of the HMD 100 based on the first frame of the real-world video received by the communication unit 201. In that case, the CG rendering unit 204 generates a first frame of the second virtual space video based on the position and orientation of the HMD 100 based on the first frame of the real-world video calculated by the second position and orientation calculation unit.

[0069] For example, the video synthesis unit 105 synthesizes a second frame from the video of the real space (for example, the frame at time t+3) that is later than the first frame, with the first frame from the video of the first virtual space. For example, the second frame from the video of the real space is the most recent frame from the video of the real space.

[0070] For example, the position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 at the time of imaging of the first frame in the real-world space, based on the first frame in the real-world space image. The CG rendering unit 104 generates the first frame in the first virtual space image based on the position and orientation of the HMD 100 at the time of imaging of the first frame in the real-world space image. The CG rendering unit 204 generates the first frame in the second virtual space image based on the position and orientation of the HMD 100 at the time of imaging of the first frame in the real-world space image.

[0071] The communication unit 107 transmits to the display device 200 the first frame of the real-world image captured by the imaging unit 101 and the position and orientation of the HMD 100 at the time the first frame of the real-world image was captured. The communication unit 201 receives the first frame of the real-world image and the position and orientation of the HMD 100 at the time the first frame of the real-world image was captured from the HMD 100. The CG rendering unit 204 generates the first frame of the second virtual space image based on the position and orientation of the HMD 100 at the time the first frame of the real-world image was captured, as received by the communication unit 201.

[0072] As described above, the display device 200 may be equipped with a second position and orientation calculation unit. In that case, the second position and orientation calculation unit calculates the position and orientation of the HMD 100 at the time of imaging of the first frame of the real-world image, based on the first frame of the real-world image received by the communication unit 201. The CG rendering unit 204 generates a first frame of the second virtual space image based on the position and orientation of the HMD 100 at the time of imaging of the first frame of the real-world image, calculated by the second position and orientation calculation unit.

[0073] As described above, the video synthesis unit 105 synthesizes a second frame from the video of the real space that is later than the first frame (for example, the frame at time t+3) with the first frame from the video of the first virtual space. The position and orientation calculation unit 102 predicts the position and orientation of the HMD 100 at the time the second frame of the video of the real space is captured, based on the first frame of the video of the real space. The CG rendering unit 104 generates the first frame from the video of the first virtual space based on the position and orientation of the HMD 100 at the time the second frame of the video of the real space is captured. The CG rendering unit 204 generates the first frame from the video of the second virtual space based on the position and orientation of the HMD 100 at the time the first frame of the video of the real space is captured.

[0074] Specifically, the position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 at the time of imaging of the first frame of the real-world image, based on the first frame of the real-world image. Then, the position and orientation calculation unit 102 predicts the position and orientation of the HMD 100 at the time of imaging of the second frame of the real-world image, based on the first frame of the real-world image. The CG rendering unit 104 generates the first frame of the first virtual space image based on the position and orientation of the HMD 100 at the time of imaging of the second frame of the real-world image. The communication unit 107 transmits the first frame of the real-world image captured by the imaging unit 101 and the position and orientation of the HMD 100 at the time of imaging of the first frame of the real-world image to the display device 200.

[0075] The communication unit 201 receives from the HMD 100 a first frame of the real-world image and the position and orientation of the HMD 100 at the time the first frame of the real-world image was captured. The CG rendering unit 204 generates a first frame of the second virtual space image based on the position and orientation of the HMD 100 at the time the first frame of the real-world image was captured, as received by the communication unit 201.

[0076] As described above, the display device 200 may be equipped with a second position and orientation calculation unit. In that case, the second position and orientation calculation unit calculates the position and orientation of the HMD 100 at the time of imaging of the first frame of the real-world image, based on the first frame of the real-world image received by the communication unit 201. The CG rendering unit 204 generates a first frame of the second virtual space image based on the position and orientation of the HMD 100 at the time of imaging of the first frame of the real-world image, calculated by the second position and orientation calculation unit.

[0077] Furthermore, the image synthesis unit 105 synthesizes a second frame from the real-world image (for example, the frame at time t+3) that is later than the first frame, with the first frame from the first virtual-world image. In this case, the position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 at the time the first frame of the real-world image was captured, based on the first frame of the real-world image. The CG rendering unit 104 generates a temporary frame of the first virtual-world image at the time the first frame of the real-world image was captured, based on the position and orientation of the HMD 100 at the time the first frame of the real-world image was captured. The position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 at the time the second frame of the real-world image was captured, based on the second frame of the real-world image. The CG rendering unit 104 generates the first frame of the first virtual space image at the time of the second frame of the real space image, based on the difference between the position and orientation of the HMD 100 at the time of the first frame of the real space image and the position and orientation of the HMD 100 at the time of the second frame of the real space image, and a temporary frame of the first virtual space image.

[0078] Furthermore, the communication unit 107 transmits to the display device 200 the first frame of the real-world image captured by the imaging unit 101 and the position and orientation of the HMD 100 at the time of capturing the first frame of the real-world image. The communication unit 201 receives the first frame of the real-world image and the position and orientation of the HMD 100 at the time of capturing the first frame of the real-world image from the HMD 100. The CG rendering unit 204 generates the first frame of the second virtual space image based on the position and orientation of the HMD 100 at the time of capturing the first frame of the real-world image received by the communication unit 201.

[0079] As described above, the display device 200 may be equipped with a second position and orientation calculation unit. In that case, the second position and orientation calculation unit calculates the position and orientation of the HMD 100 at the time of imaging of the first frame of the real-world image, based on the first frame of the real-world image received by the communication unit 201. The CG rendering unit 204 generates a first frame of the second virtual space image based on the position and orientation of the HMD 100 at the time of imaging of the first frame of the real-world image, calculated by the second position and orientation calculation unit.

[0080] As described above, according to this embodiment, the HMD 100 can reduce the delay in the real-world image of the mixed reality space that the wearer of the HMD 100 sees, and can display a mixed reality space image with a low delay in the real-world image. The display device 200 can display a mixed reality space image in which the delays of the real-world image and the virtual-world image are matched.

[0081] (Second embodiment) Next, a second embodiment will be described. In the second embodiment, an example will be described in which the HMD 100 generates mixed reality images with different delay amounts for the mixed reality image displayed on its own video display unit 106 and transmits them to the display device 200. In this way, the HMD 100 generates the mixed reality image to be displayed on the display device 200, so the display device 200 only needs to be a device that has the function of playing and displaying general streaming video.

[0082] The following description of the second embodiment will only explain the differences from the first embodiment. In the second embodiment, the same components or steps as in the first embodiment will be denoted by the same reference numerals as those described in the first embodiment.

[0083] Figure 6 is a block diagram showing an example of the functional configuration of the HMD100 according to the second embodiment. The HMD100 in Figure 6 is the same as the HMD100 in Figure 2, but with the addition of a video delay unit 401. The HMD100 includes an imaging unit 101, a position and orientation calculation unit 102, a CG data acquisition unit 103, a CG rendering unit 104, a video synthesis unit 105, a video display unit 106, a communication unit 107, and a video delay unit 401.

[0084] Figure 7 is a flowchart showing the processing method of HMD100 in Figure 6, illustrating an example of the processing of HMD100 every frame. Figure 7 is obtained by removing step S103 and adding steps S401 and S402 to Figure 3. HMD100 starts processing the flowchart in Figure 7 at a predetermined timing in each frame. HMD100 performs the processing of steps S101, S102, and S104-S106, similar to Figure 3.

[0085] In step S101, the video delay unit 202 buffers the real-world images for the right and left eyes captured by the imaging unit 101 for each frame, and holds the frames of the real-world images for the right and left eyes up to the present.

[0086] Furthermore, in step S106, the HMD100 combines the latest frame of the real-world image at time t+3 with the frame of the virtual-world image at time t to generate a mixed-reality image for display on the HMD100.

[0087] Subsequently, in step S401, the video synthesis unit 105 synthesizes, frame by frame, the real-world images for the right and left eyes held by the video delay unit 401 and the virtual-world images for the right and left eyes generated by the CG rendering unit 104. The video synthesis unit 105 generates mixed-reality images for the right and left eyes through frame by frame synthesis. Specifically, the video synthesis unit 105 synthesizes the real-world image frame at time t and the virtual-world image frame at time t to generate the mixed-reality image frame at time t.

[0088] In step S402, the communication unit 107 transmits the mixed reality space images for the right eye and left eye, generated in step S401, to the display device 200 in a general streaming format, frame by frame.

[0089] In the display device 200, the communication unit 201 receives images of the mixed reality space for the right eye and the left eye from the HMD 100. The image display unit 206 displays the images of the mixed reality space for the right eye and the left eye received by the communication unit 201 on the screens in front of the right and left eyes of the display device 200.

[0090] As described above, the HMD100 generates images of the mixed reality space for display on the HMD100 and images of the mixed reality space for display on the display device 200.

[0091] The HMD100 displays a mixed reality image frame, which is a composite of a frame of real-world video at time t+3 and a frame of virtual-world video at time t. As a result, the mixed reality image displayed on the HMD100 reduces the amount of delay in the real-world image caused by the wearer's movements, thereby reducing discomfort for the wearer.

[0092] Furthermore, the display device 200 displays a mixed reality image frame, which is a composite of a real-world image frame at time t and a virtual-world image frame at time t. As a result, the mixed reality image displayed on the display device 200 has the same delay amount between the real-world image frame and the virtual-world image frame, which reduces the sense of unnaturalness for the viewer of the display device 200.

[0093] As described above, the HMD100 is an imaging device and includes an imaging unit 101, a position and orientation calculation unit 102, a CG data acquisition unit 103, a CG rendering unit 104, an image synthesis unit 105, an image display unit 106, a communication unit 107, and an image delay unit 401.

[0094] The imaging unit 101 captures images of the real world. The position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 based on the images of the real world. The CG rendering unit 104 generates images of the virtual world based on the position and orientation of the HMD 100. The image synthesis unit 105 generates images of a mixed reality space by combining the images of the real world captured by the imaging unit 101 and the images of the virtual world generated by the CG rendering unit 104. The communication unit 107 transmits the images of the mixed reality space to the display device 200.

[0095] The position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 based on a first frame of the real-world video (for example, the frame at time t). The CG rendering unit 104 generates a first frame of the virtual-world video based on the position and orientation of the HMD 100 based on the first frame of the real-world video. The video synthesis unit 105 generates a first frame of the first mixed-reality video by combining a frame different from the first frame of the real-world video (for example, the frame at time t+3) with the first frame of the virtual-world video. The video synthesis unit 105 then displays the first frame of the first mixed-reality video on the video display unit 106.

[0096] Furthermore, the image synthesis unit 105 generates a first frame of the second mixed reality image by combining a first frame of the image from the real space with a first frame of the image from the virtual space. The communication unit 107 transmits the first frame of the second mixed reality image to the display device 200.

[0097] Specifically, the image synthesis unit 105 generates the first frame of the first mixed reality space image by combining a second frame from the real-world image that is later than the first frame (for example, the frame at time t+3) with the first frame from the first virtual space image. The second frame from the real-world image mentioned above is, for example, the most recent frame from the real-world image.

[0098] The position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 at the time of imaging of the first frame in the real-world space, based on the first frame in the real-world space image. The CG rendering unit 104 generates the first frame in the virtual space image based on the position and orientation of the HMD 100 at the time of imaging of the first frame in the real-world space image.

[0099] The position and orientation calculation unit 102 can predict the position and orientation of the HMD 100 at the time of imaging of the second frame of the real-world image, based on the first frame of the real-world image. In this case, the CG rendering unit 104 generates the first frame of the first virtual space image based on the position and orientation of the HMD 100 at the time of imaging of the second frame of the real-world image. The image synthesis unit 105 generates the first frame of the first mixed reality image by combining the second frame of the real-world image and the first frame of the first virtual space image. The CG rendering unit 104 generates the first frame of the second virtual space image based on the position and orientation of the HMD 100 at the time of imaging of the first frame of the real-world image. The image synthesis unit 105 generates the first frame of the second mixed reality image by combining the first frame of the real-world image and the first frame of the second virtual space image.

[0100] Furthermore, the position and orientation calculation unit 102 can calculate the position and orientation of the HMD 100 at the time of imaging of the first frame in the real-world image, based on the first frame in the real-world image. The CG rendering unit 104 generates a temporary frame of the first virtual space image at the time of imaging of the first frame in the real-world image, based on the position and orientation of the HMD 100 at the time of imaging of the first frame in the real-world image. The position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 at the time of imaging of the second frame in the real-world image, based on the second frame in the real-world image. The CG rendering unit 104 generates the first frame of the first virtual space image at the time of imaging of the second frame in the real-world image, based on the difference between the position and orientation of the HMD 100 at the time of imaging of the first frame in the real-world image and the position and orientation of the HMD 100 at the time of imaging of the second frame in the real-world image, and the temporary frame of the first virtual space image mentioned above.

[0101] As described above, according to this embodiment, the HMD 100 can reduce the delay in the real-world image of the mixed reality space that the wearer of the HMD 100 sees, and can display a mixed reality space image with a low delay in the real-world image. The display device 200 can display a mixed reality space image in which the delays of the real-world image and the virtual-world image are matched.

[0102] (Other embodiments) This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0103] Furthermore, the embodiments described above are merely examples illustrating how to implement this disclosure, and they should not be interpreted as limiting the technical scope of this disclosure. In other words, this disclosure can be implemented in various ways without departing from its technical concept or its main features.

[0104] This embodiment includes the following configurations and methods. (Composition 1) Imaging device and A display system having a display device, The imaging device is An imaging means for capturing images of real space, A first position and orientation calculation means calculates the position and orientation of the imaging device based on the image of the real space, A first generation means that generates an image of a first virtual space based on the position and orientation of the imaging device, A first image synthesis means generates an image of a first mixed reality space by combining an image of the real space captured by the imaging means with an image of the first virtual space, and displays the image of the first mixed reality space. It includes at least a first communication means for transmitting images of the real space captured by the imaging means to the display device, The first position and orientation calculation means calculates the position and orientation of the imaging device based on a first frame of the image of the real space, The first generation means generates a first frame of the first virtual space image based on the position and orientation of the imaging device based on a first frame of the image of the real space, The first image synthesis means synthesizes a frame different from the first frame of the image of the real space with the first frame of the image of the first virtual space. The aforementioned display device is A second communication means for receiving images of the real space from the imaging device, A second generation means for generating an image of a second virtual space based on the position and orientation of the imaging device, The system includes a second image synthesis means that generates an image of a second mixed reality space by combining an image of the real space received by the second communication means with an image of the second virtual space, and displays the image of the second mixed reality space. The second generation means generates a first frame of the second virtual space image based on the position and orientation of the imaging device based on a first frame of the image of the real space, The display system is characterized in that the second image synthesis means synthesizes a first frame from the image of the real space with a first frame from the image of the second virtual space. (Configuration 2) The first communication means transmits to the display device a first frame of the image of the real space captured by the imaging means and the position and orientation of the imaging device based on the first frame of the image of the real space. The second communication means receives from the imaging device a first frame of the image of the real space and the position and orientation of the imaging device based on the first frame of the image of the real space. The display system according to Configuration 1, characterized in that the second generation means generates a first frame of the second virtual space image based on the position and orientation of the imaging device, which is based on a first frame of the real space image received by the second communication means. (Composition 3) The display device has a second position and orientation calculation means that calculates the position and orientation of the imaging device based on the first frame of the image of the real space received by the second communication means, The display system according to Configuration 1, characterized in that the second generation means generates a first frame of the second virtual space image based on the position and orientation of the imaging device, which is based on a first frame of the real space image calculated by the second position and orientation calculation means. (Composition 4) The display system according to any one of configurations 1 to 3, characterized in that the first image synthesis means synthesizes a second frame from the image of the real space that is later than the first frame with the first frame from the image of the first virtual space. (Composition 5) The display system according to configuration 4, characterized in that the second frame of the image of the real space is the most recent frame of the image of the real space. (Composition 6) The first position and orientation calculation means calculates the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space, based on the first frame of the image of the real space. The first generation means generates a first frame of the first virtual space image based on the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space, The display system according to any one of configurations 1 to 5, characterized in that the second generation means generates the first frame of the second virtual space image based on the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space. (Composition 7) The first position and orientation calculation means calculates the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space, based on the first frame of the image of the real space. The first generation means generates a first frame of the first virtual space image based on the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space, The first communication means transmits to the display device a first frame of the image of the real space captured by the imaging means and the position and orientation of the imaging device at the time of capturing the first frame of the image of the real space. The second communication means receives from the imaging device a first frame of the image of the real space and the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space. The display system according to configuration 2, characterized in that the second generation means generates a first frame of the second virtual space image based on the position and orientation of the imaging device at the time of imaging of the first frame of the real space image received by the second communication means. (Composition 8) The first position and orientation calculation means calculates the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space, based on the first frame of the image of the real space. The first generation means generates a first frame of the first virtual space image based on the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space, The second position and orientation calculation means calculates the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space, based on the first frame of the image of the real space received by the second communication means. The display system according to configuration 3, characterized in that the second generation means generates the first frame of the second virtual space image based on the position and orientation of the imaging device at the time of imaging of the first frame of the real space image calculated by the second position and orientation calculation means. (Composition 9) The first image synthesis means synthesizes a second frame from the image of the real space that is after the first frame with the first frame from the image of the first virtual space. The first position and orientation calculation means predicts the position and orientation of the imaging device at the time of imaging of the second frame of the image of the real space, based on the first frame of the image of the real space. The first generation means generates a first frame of the first virtual space image based on the position and orientation of the imaging device at the time of imaging of the second frame of the image of the real space, The display system according to Configuration 1, characterized in that the second generation means generates the first frame of the second virtual space image based on the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space. (Composition 10) The first position and orientation calculation means calculates the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space, based on the first frame of the image of the real space, and predicts the position and orientation of the imaging device at the time of imaging of the second frame of the image of the real space. The first generation means generates a first frame of the first virtual space image based on the position and orientation of the imaging device at the time of imaging of the second frame of the image of the real space, The first communication means transmits to the display device a first frame of the image of the real space captured by the imaging means and the position and orientation of the imaging device at the time of capturing the first frame of the image of the real space. The second communication means receives from the imaging device a first frame of the image of the real space and the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space. The display system according to configuration 9, characterized in that the second generation means generates a first frame of the second virtual space image based on the position and orientation of the imaging device at the time of imaging of the first frame of the real space image received by the second communication means. (Composition 11) The display device has a second position and orientation calculation means that calculates the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space, based on the first frame of the image of the real space received by the second communication means. The display system according to configuration 9, characterized in that the second generation means generates the first frame of the second virtual space image based on the position and orientation of the imaging device at the time of imaging of the first frame of the real space image calculated by the second position and orientation calculation means. (Composition 12) The first image synthesis means synthesizes a second frame from the image of the real space that is after the first frame with the first frame from the image of the first virtual space. The first position and orientation calculation means calculates the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space, based on the first frame of the image of the real space. The first generation means generates a temporary frame of the first virtual space image at the time of imaging of the first frame of the real space image, based on the position and orientation of the imaging device at the time of imaging of the first frame of the real space image. The first position and orientation calculation means calculates the position and orientation of the imaging device at the time of imaging of the second frame of the image of the real space, based on the second frame of the image of the real space. The display system according to Configuration 1, characterized in that the first generation means generates the first frame of the first virtual space image at the time of imaging the second frame of the real space image, based on the difference between the position and orientation of the imaging device at the time of imaging the second frame of the real space image and a temporary frame of the first virtual space image. (Composition 13) The first communication means transmits to the display device a first frame of the image of the real space captured by the imaging means and the position and orientation of the imaging device at the time of capturing the first frame of the image of the real space. The second communication means receives from the imaging device a first frame of the image of the real space and the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space. The display system according to configuration 12, characterized in that the second generation means generates a first frame of the second virtual space image based on the position and orientation of the imaging device at the time of imaging of the first frame of the real space image received by the second communication means. (Composition 14) The display device has a second position and orientation calculation means that calculates the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space, based on the first frame of the image of the real space received by the second communication means. The display system according to configuration 12, characterized in that the second generation means generates the first frame of the second virtual space image based on the position and orientation of the imaging device at the time of imaging of the first frame of the real space image calculated by the second position and orientation calculation means. (Composition 15) An imaging device, An imaging means for capturing images of real space, A position and orientation calculation means that calculates the position and orientation of the imaging device based on the image of the real space, A generation means that generates images in a virtual space based on the position and orientation of the aforementioned imaging device, A video synthesis means generates an image of a mixed reality space by combining an image of the real space captured by the imaging means and an image of the virtual space generated by the generation means. It has communication means for transmitting images of the augmented reality space to a display device, The position and orientation calculation means calculates the position and orientation of the imaging device based on a first frame of the image of the real space, The generation means generates a first frame of the virtual space image based on the position and orientation of the imaging device, which is based on a first frame of the image of the real space. The image synthesis means generates a first frame of the first mixed reality image by combining a frame different from the first frame of the image of the real space with the first frame of the image of the virtual space, and displays the first frame of the image of the first mixed reality image. The image synthesis means generates a first frame of the image of a second mixed reality space by combining a first frame of the image of the real space and a first frame of the image of the virtual space. The imaging device is characterized in that the communication means transmits a first frame of the image of the second mixed reality space to the display device. (Composition 16) The imaging apparatus according to configuration 15, characterized in that the image synthesis means generates a first frame of the image of the first mixed reality space by synthesizing a second frame from the image of the real space that is later than the first frame and a first frame from the image of the first virtual space. (Composition 17) The imaging apparatus according to configuration 16, characterized in that the second frame of the image of the real space is the most recent frame of the image of the real space. (Composition 18) The position and orientation calculation means calculates the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space, based on the first frame of the image of the real space. The imaging apparatus according to any one of configurations 15 to 17, characterized in that the generation means generates a first frame of the image of the virtual space based on the position and orientation of the imaging apparatus at the time of imaging of a first frame of the image of the real space. (Composition 19) The position and orientation calculation means predicts the position and orientation of the imaging device at the time of imaging of the second frame of the image of the real space, based on the first frame of the image of the real space. The generation means generates a first frame of the first virtual space image based on the position and orientation of the imaging device at the time of imaging of the second frame of the image of the real space, The image synthesis means generates a first frame of the image of the first mixed reality space by combining a second frame of the image of the real space and a first frame of the image of the first virtual space. The generation means generates a first frame of the second virtual space image based on the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space, The imaging apparatus according to configuration 16 or 17, characterized in that the image synthesis means generates a first frame of an image of a second mixed reality space by combining a first frame of an image of the real space with a first frame of an image of the second virtual space. (Composition 20) The position and orientation calculation means calculates the position and orientation of the imaging device at the time of imaging of the first frame of the image of the real space, based on the first frame of the image of the real space. The generation means generates a temporary frame of the first virtual space image at the time of imaging of the first frame of the real space image, based on the position and orientation of the imaging device at the time of imaging of the first frame of the real space image. The position and orientation calculation means calculates the position and orientation of the imaging device at the time of imaging of the second frame of the image of the real space, based on the second frame of the image of the real space. The imaging apparatus according to configuration 16 or 17, characterized in that the generation means generates a first frame of the first virtual space image at the time of imaging of the second frame of the real space image, based on the difference between the position and orientation of the imaging apparatus at the time of imaging of the second frame of the real space image and the position and orientation of the imaging apparatus at the time of imaging of the second frame of the real space image, and a provisional frame of the first virtual space image. (Method 1) Imaging device and A processing method for a display system having a display device, The imaging device is An imaging means for capturing images of real space, A first position and orientation calculation means calculates the position and orientation of the imaging device based on the image of the real space, A first generation means that generates an image of a first virtual space based on the position and orientation of the imaging device, A first image synthesis means generates an image of a first mixed reality space by combining an image of the real space captured by the imaging means with an image of the first virtual space, and displays the image of the first mixed reality space. It includes at least a first communication means for transmitting images of the real space captured by the imaging means to the display device, The aforementioned display device is A second communication means for receiving images of the real space from the imaging device, A second generation means for generating an image of a second virtual space based on the position and orientation of the imaging device, The system includes a second image synthesis means that generates an image of a second mixed reality space by combining an image of the real space received by the second communication means with an image of the second virtual space, and displays the image of the second mixed reality space. The processing method of the aforementioned display system is: The first position and orientation calculation means includes the step of calculating the position and orientation of the imaging device based on a first frame of the image of the real space, The first generation means includes the step of generating a first frame of the first virtual space image based on the position and orientation of the imaging device based on a first frame of the image of the real space, The first image synthesis means performs the steps of synthesizing a frame different from the first frame of the image of the real space with the first frame of the image of the first virtual space, The second generation means includes the step of generating a first frame of the second virtual space image based on the position and orientation of the imaging device based on a first frame of the image of the real space, The second image synthesis means performs the steps of synthesizing a first frame from the image of the real space and a first frame from the image of the second virtual space. A method for processing a display system, characterized by having the following features. (Method 2) A method for processing an imaging device, The imaging device is An imaging means for capturing images of real space, A position and orientation calculation means that calculates the position and orientation of the imaging device based on the image of the real space, A generation means that generates images in a virtual space based on the position and orientation of the aforementioned imaging device, A video synthesis means generates an image of a mixed reality space by combining an image of the real space captured by the imaging means and an image of the virtual space generated by the generation means. It has communication means for transmitting images of the augmented reality space to a display device, The processing method of the imaging device is as follows: The position and orientation calculation means includes the step of calculating the position and orientation of the imaging device based on a first frame of the image of the real space, The generation means includes the step of generating a first frame of the image in the virtual space based on the position and orientation of the imaging device based on a first frame of the image in the real space, The image synthesis means generates a first frame of the first mixed reality image by synthesizing a frame different from the first frame of the image of the real space with the first frame of the image of the virtual space, and displays the first frame of the image of the first mixed reality image. The image synthesis means generates a first frame of a second mixed reality image by combining a first frame of an image of the real space and a first frame of an image of the virtual space, The communication means transmits a first frame of the image of the second mixed reality space to the display device. A processing method for an imaging device, characterized by having the following features. [Explanation of Symbols]

[0105] 101 Imaging unit, 102 Position and orientation calculation unit, 103 CG data acquisition unit, 104 CG rendering unit, 105 Image synthesis unit, 106 Image display unit, 107 Communication unit, 201 Communication unit, 202 Image delay unit, 203 CG data acquisition unit, 204 CG rendering unit, 205 Image synthesis unit, 206 Image display unit

Claims

A display system comprising an imaging device having imaging means for imaging a video of the real space and display means, and a display device, wherein: position and orientation acquisition means for acquiring the position and orientation of the imaging device based on the video of the real space; generation means for generating a video of a virtual space based on the position and orientation; first video synthesis means for synthesizing the video of the real space and the video of the virtual space to generate a first composite real space video for display on the display means; second video synthesis means for synthesizing the video of the real space and the video of the virtual space to generate a second composite real space video for display on the display device; the position and orientation acquisition means acquires the position and orientation of the imaging device based on a first frame of the video of the real space; the generation means generates a first frame of the video of the virtual space based on the position and orientation of the imaging device based on the first frame of the video of the real space; the first video synthesis means synthesizes a frame different from the first frame of the video of the real space and the first frame of the video of the virtual space; the second video synthesis means synthesizes the first frame of the video of the real space and the first frame of the video of the virtual space A display system characterized by the above. According to claim 2, the imaging device: first position and orientation acquisition means for acquiring the position and orientation of the imaging device based on the video of the real space; first generation means for generating a video of a first virtual space based on the position and orientation of the imaging device; first video synthesis means for generating a first composite real space video by synthesizing the video of the real space and the video of the first virtual space, and displaying the first composite real space video; and first communication means for transmitting at least the video of the real space to the display device. The first position and orientation acquisition means acquires the position and orientation of the imaging device based on the first frame among the images of the real space. The first generation means generates a first frame among the images of the first virtual space based on the position and orientation of the imaging device based on the first frame among the images of the real space. The first video synthesis means synthesizes a frame different from the first frame among the images of the real space and the first frame among the images of the first virtual space. The display system according to claim 1, characterized in that.

3. The display device A second communication means for receiving an image of the real space from the imaging device, A second generation means for generating an image of a second virtual space based on the position and orientation of the imaging device, By synthesizing the image of the real space received by the second communication means and the image of the second virtual space, a second composite real space image is generated, and the second video synthesis means for displaying the second composite real space image, The second generation means generates a first frame among the images of the second virtual space based on the position and orientation of the imaging device based on the first frame among the images of the real space. The second video synthesis means synthesizes the first frame among the images of the real space and the first frame among the images of the second virtual space. The display system according to claim 2, characterized in that.

4. The first communication means transmits a first frame among the images of the real space captured by the imaging means and the position and orientation of the imaging device based on the first frame among the images of the real space to the display device. The second communication means receives a first frame among the images of the real space from the imaging device and the position and orientation of the imaging device based on the first frame among the images of the real space. The second generation means generates a first frame of the video in the second virtual space based on the position and orientation of the imaging device based on the first frame of the video in the real space received by the second communication means. The display system according to claim 3, characterized in that.

5. The display device has second position and orientation acquisition means for acquiring the position and orientation of the imaging device based on the first frame of the video in the real space, based on the first frame of the video in the real space received by the second communication means. The second generation means generates a first frame of the video in the second virtual space based on the position and orientation of the imaging device based on the first frame of the video in the real space acquired by the second position and orientation acquisition means. The display system according to claim 3, characterized in that.

6. The first video synthesis means synthesizes a second frame after the first frame of the video in the real space and the first frame of the video in the virtual space. The display system according to claim 1, characterized in that.

7. The second frame of the video in the real space is the latest frame of the video in the real space. The display system according to claim 6, characterized in that.

8. The first position and orientation acquisition means acquires the position and orientation of the imaging device at the time of imaging the first frame of the video in the real space, based on the first frame of the video in the real space. The first generation means generates a first frame of the video in the first virtual space based on the position and orientation of the imaging device at the time of imaging the first frame of the video in the real space. The second generation means generates a first frame of the video in the second virtual space based on the position and orientation of the imaging device at the time of imaging the first frame of the video in the real space. The display system according to claim 3, characterized in that...

9. The first position and orientation acquisition means acquires the position and orientation of the imaging device at the time of imaging the first frame among the images of the real space, based on the first frame among the images of the real space. The first generation means generates the first frame among the images of the first virtual space, based on the position and orientation of the imaging device at the time of imaging the first frame among the images of the real space. The first communication means transmits the first frame among the images of the real space captured by the imaging means and the position and orientation of the imaging device at the time of imaging the first frame among the images of the real space to the display device. The second communication means receives the first frame among the images of the real space and the position and orientation of the imaging device at the time of imaging the first frame among the images of the real space from the imaging device. The second generation means generates the first frame among the images of the second virtual space, based on the position and orientation of the imaging device at the time of imaging the first frame among the images of the real space received by the second communication means. The display system according to claim 4, characterized in that...

10. The first position and orientation acquisition means acquires the position and orientation of the imaging device at the time of imaging the first frame among the images of the real space, based on the first frame among the images of the real space. The first generation means generates the first frame among the images of the first virtual space, based on the position and orientation of the imaging device at the time of imaging the first frame among the images of the real space. The second position and orientation acquisition means acquires the position and orientation of the imaging device at the time of imaging the first frame among the images of the real space, based on the first frame among the images of the real space received by the second communication means. The second generation means generates the first frame of the video of the second virtual space based on the position and orientation of the imaging device at the time of imaging the first frame among the videos of the real space acquired by the second position and orientation acquisition means. The display system according to claim 5, characterized in that.

11. The first video synthesis means synthesizes the second frame after the first frame of the video of the real space and the first frame of the video of the first virtual space. The first position and orientation acquisition means predicts the position and orientation of the imaging device at the time of imaging the second frame of the video of the real space based on the first frame of the video of the real space. The first generation means generates the first frame of the video of the first virtual space based on the position and orientation of the imaging device at the time of imaging the second frame of the video of the real space. The second generation means generates the first frame of the video of the second virtual space based on the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space. The display system according to claim 3, characterized in that.

12. The first position and orientation acquisition means acquires the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space based on the first frame of the video of the real space, and predicts the position and orientation of the imaging device at the time of imaging the second frame of the video of the real space. The first generation means generates the first frame of the video of the first virtual space based on the position and orientation of the imaging device at the time of imaging the second frame of the video of the real space. The first communication means transmits the first frame of the video of the real space captured by the imaging means and the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space to the display device. The second communication means receives the first frame of the video of the real space from the imaging device and the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space, Based on the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space received by the second communication means, the second generation means generates the first frame of the video of the second virtual space. The display system according to claim 11, characterized in that.

13. The display device has second position and orientation acquisition means for acquiring the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space based on the first frame of the video of the real space received by the second communication means. Based on the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space acquired by the second position and orientation acquisition means, the second generation means generates the first frame of the video of the second virtual space. The display system according to claim 11, characterized in that.

14. The first video composition means composes the second frame after the first frame of the video of the real space and the first frame of the video of the first virtual space. The first position and orientation acquisition means acquires the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space based on the first frame of the video of the real space. Based on the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space, the first generation means generates a virtual frame of the video of the first virtual space at the time of imaging the first frame of the video of the real space. The first position and orientation acquisition means acquires the position and orientation of the imaging device at the time of imaging the second frame of the video of the real space based on the second frame of the video of the real space. The first generation means generates the first frame of the video of the first virtual space at the time of imaging the second frame of the video of the real space based on the difference between the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space and the position and orientation of the imaging device at the time of imaging the second frame of the video of the real space, and a provisional frame of the video of the first virtual space. The display system according to claim 3, characterized in that.

15. The first communication means transmits to the display device the first frame of the video of the real space imaged by the imaging means and the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space. The second communication means receives from the imaging device the first frame of the video of the real space and the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space. The second generation means generates the first frame of the video of the second virtual space based on the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space received by the second communication means. The display system according to claim 14, characterized in that.

16. The display device has second position and orientation acquisition means for acquiring the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space based on the first frame of the video of the real space received by the second communication means. The second generation means generates the first frame of the video of the second virtual space based on the position and orientation of the imaging device at the time of imaging the first frame of the video of the real space acquired by the second position and orientation acquisition means. The display system according to claim 14, characterized in that.

17. The imaging device is a head-mounted display. The display system according to claim 1, characterized in that. An imaging device having imaging means for imaging a video of the real space and display means, position and orientation acquisition means for acquiring the position and orientation of the imaging device based on the video of the real space, generation means for generating a video of a virtual space based on the position and orientation, first video synthesis means for synthesizing the video of the real space and the video of the virtual space to generate a first composite reality space video for display on the display means, second video synthesis means for synthesizing the video of the real space and the video of the virtual space to generate a second composite reality space video for display on an external display device, the position and orientation acquisition means acquires the position and orientation of the imaging device based on a first frame of the video of the real space, the generation means generates a first frame of the video of the virtual space based on the position and orientation of the imaging device based on the first frame of the video of the real space, the first video synthesis means synthesizes a frame different from the first frame of the video of the real space and the first frame of the video of the virtual space, the second video synthesis means synthesizes the first frame of the video of the real space and the first frame of the video of the virtual space An imaging device characterized by the above. A processing method for a display system having an imaging device having imaging means for imaging a video of the real space and display means, and a display device, a position and orientation acquisition step of acquiring the position and orientation of the imaging device based on the video of the real space, a generation step of generating a video of a virtual space based on the position and orientation, a first video synthesis step of synthesizing the video of the real space and the video of the virtual space to generate a first composite reality space video for display on the display means, A second video synthesis step of synthesizing the video of the real space and the video of the virtual space to generate a video of a second composite reality space for display on the display device. In the position and orientation acquisition step, the position and orientation of the imaging device based on the first frame of the video of the real space are acquired. In the generation step, based on the position and orientation of the imaging device based on the first frame of the video of the real space, the first frame of the video of the virtual space is generated. In the first video synthesis step, a frame different from the first frame of the video of the real space and the first frame of the video of the virtual space are synthesized. In the second video synthesis step, the first frame of the video of the real space and the first frame of the video of the virtual space are synthesized. A processing method of a display system, characterized by the above.

Images