Display system, imaging device, and processing method for display system

Abstract

To provide an imaging apparatus which displays the video of a composite real space where the delay amount of the video of a real space is small and a display device which can display the video of the composite real space where the delay amounts of the video of the real space and the video of a virtual space are matched.SOLUTION: The imaging apparatus generates the first frame of the video of a first virtual space on the basis of the position attitude of the imaging apparatus based on the first frame of the video of the real space and composites a frame different from the first frame of the video of the real space and the first frame of the video of the first virtual space. The display device generates the first frame of the video of a second virtual space on the basis of the position attitude of the imaging apparatus based on the first frame of the video of the real space and composites the first frame of the video of the real space and the first frame of the video of the second virtual space.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a display system, an imaging device, and The present invention relates to a processing method for a display system. [Background technology]

[0002] In recent years, mixed reality (MR) systems have emerged as a technology that seamlessly blends the real world and the virtual world in real time. Mixed reality systems allow objects to be superimposed on the real world as computer graphics (CG) images, and can be viewed from any viewpoint.

[0003] In mixed reality systems, a head-mounted display (hereinafter referred to as HMD) that integrates an imaging device such as a video camera with a display is used as one type of image display device. One type of HMD is the video see-through method. This is a method of superimposing and displaying an image of a virtual space generated according to the position and orientation of the HMD on an image of the real space captured by an imaging device mounted on the HMD. Examples of images of the virtual space include virtual objects and text information drawn using CG (Computer Graphics).

[0004] During the design process in the manufacturing industry, reviews involving a large number of people are often held. In reviews using a mixed reality system, a system is configured that combines an HMD with a handheld information terminal such as a tablet so that multiple people can experience a mixed reality space (a space that combines virtual and real space). This system simultaneously distributes and displays the image of the mixed reality space seen by the HMD wearer on other HMDs and multiple tablet screens. This allows multiple people to simultaneously share the field of view of the HMD wearer and participate in the design and review.

[0005] When generating images in a virtual space, there is a delay before the image reflecting the current position and orientation of the HMD is displayed due to the calculation of the position and orientation of the HMD and the rendering of CG according to that position and orientation. If the delay is large in MR, the user will feel uncomfortable due to a mismatch between their own movements and the display, which can cause sickness. For this reason, images in real space are sometimes displayed with as little delay as possible, even if this differs from the amount of delay in images in virtual space.

[0006] Patent document 1 discloses a technology that reduces the appearance of delay in virtual space images by reacquiring the position and orientation of the HMD after generating the virtual space image and shifting and displaying the generated virtual space image according to the difference from the position and orientation used when generating the virtual space image. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-299669 Summary of the Invention [Problem to be solved by the invention]

[0008] However, when the image of the mixed reality space seen by the HMD wearer is distributed and displayed, the movement of the shared image does not reflect the viewer's own movements at the shared destination. Therefore, the discomfort caused by the delay until the HMD wearer's movements are reflected is small, and the discomfort caused by the difference in delay between the image in real space and the image in virtual space is felt more.

[0009] The object of the present disclosure is to enable an imaging device to display images in a mixed reality space with a small amount of delay in the images in real space, and to enable a display device to display images in a mixed reality space with the same amount of delay between the images in real space and the images in virtual space. [Means for solving the problem]

[0010] The display system is The system has an imaging means for capturing an image of real space and a display means. An imaging device and a display device 、 a display system having the above-mentioned structure, acquisition Position and posture acquisition a generating means for generating an image of a virtual space based on the position and orientation; and a synthesizing means for synthesizing the image of the real space and the image of the virtual space. Then, for display on the display means The first mixed reality space image Generate a first image synthesis means for synthesizing the image; a second image synthesis means for synthesizing the image of the real space and the image of the virtual space to generate an image of a second mixed reality space to be displayed on the display device; and the position and orientation acquisition means teeth , the position and orientation of the image capturing device based on a first frame of the image of the real space is calculated. acquisition The generating means generates a real-space image of the real space. The aforementioned a first frame of the image of the virtual space is generated based on the position and orientation of the image capturing device based on the first frame, and the first image synthesis means synthesizes the first frame of the image of the real space The aforementioned a frame different from the first frame and a frame in the image of the virtual space The aforementioned and the second image synthesis means synthesizes the first frame with the second frame, ... with the The aforementioned a first frame and an image of the virtual space; The aforementioned The first frame is composited with the second frame. [Effects of the Invention]

[0011] According to the present disclosure, an imaging device can display an image in a mixed reality space with a small amount of delay in the image in real space, and a display device can display an image in a mixed reality space with the amount of delay between the image in real space and the image in virtual space matching each other. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a display system. [Figure 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of the display system. [Figure 3] 10 is a flowchart illustrating an example of processing performed by an HMD. [Figure 4]10 is a flowchart illustrating an example of processing by the display device. [Figure 5] FIG. 10 is a diagram showing video delay. [Figure 6] FIG. 2 is a block diagram illustrating an example of the functional configuration of an HMD. [Figure 7] 10 is a flowchart illustrating an example of processing performed by an HMD. DETAILED DESCRIPTION OF THE INVENTION

[0013] (First embodiment) Embodiments will be described in detail below with reference to the drawings. FIG. 1 is a diagram showing an example of the configuration of a display system 500 according to a first embodiment. The display system 500 includes an HMD (head-mounted display) 100, a display device 200, and a CG (computer graphics) data management server 300. The HMD 100 is an example of a head-mounted display device, and is a device that presents an image of a mixed reality space, which is a combination of an image of a real space and an image of a virtual space, in front of the eyes of a user wearing the HMD 100 on their head. The CG data management server 300 is a device that stores CG data necessary to construct a virtual space. The HMD 100 and the display device 200 access the CG data management server 300 via a network, acquire CG data of the virtual reality space, generate an image of the virtual reality space, and display the image of the virtual reality space combined with the image of the real space.

[0014] The HMD 100 and the display device 200 communicate with each other via a network, and can transfer images of real space captured by the HMD 100 and information for generating a virtual space from the HMD 100 to the display device 200. The display device 200 uses the transferred information to generate images of a mixed reality space, thereby enabling the field of view of the wearer of the HMD 100 to be shared and displayed on the display device 200.

[0015] The HMD 100 may be a system separated into a head-mounted part and an image processing device such as a personal computer (PC).The HMD 100 may also be a device such as a smartphone that is held in the hand and displays an image of real space captured on a display.

[0016] The display device 200 may be an HMD, or may be a terminal such as a PC (personal computer) or tablet that can combine and display images of real space and images of virtual space. Also, there may be a plurality of display devices 200, and the image from the HMD 100 may be shared with all of them.

[0017] The HMD 100 and the display device 200 may be located in the same space where the respective users can see each other directly, or may be located far away from each other where they cannot see each other directly. Also, the CG data management server 300 may be integrated with the HMD 100.

[0018] In this embodiment, the HMD 100 captures an image of the real space and calculates the position and orientation of the HMD 100. Then, the HMD 100 uses CG data acquired from the CG data management server 300 to generate an image of the virtual space so that the image is viewed from the calculated position and orientation, and generates an image of the mixed reality space by combining the captured image of the real space and the generated image of the virtual space. The HMD 100 then displays the generated image of the mixed reality space on the display unit of the HMD 100, and transmits the captured image of the real space and the position and orientation of the HMD 100 to the display device 200.

[0019] The display device 200 uses the CG data acquired from the CG data management server 300 to generate an image of the virtual space as seen from the position and orientation of the received HMD 100, and then synthesizes and displays the image of the virtual space with the image of the received real space. In this way, the HMD 100 and the display device 200 can share the image of the mixed reality space.

[0020] 2 is a block diagram showing an example of the functional configuration of the display system 500. The display system 500 includes an HMD 100, a display device 200, and a CG data management server 300.

[0021] The HMD 100 includes an imaging unit 101 , a position and orientation calculation unit 102 , a CG data acquisition unit 103 , a CG rendering unit 104 , an image synthesis unit 105 , an image display unit 106 , and a communication unit 107 .

[0022] The imaging unit 101 captures an image of real space for each frame using a camera attached to the HMD 100. The camera can capture an image corresponding to the eye position of the person wearing the HMD 100, and can capture an image for the right eye and an image for the left eye. Note that the images and position and orientation generated below are generated for each of the right eye and the left eye. The imaging unit 101 captures an image of real space for the right eye and the left eye.

[0023] The position and orientation calculation unit 102 calculates the positions and orientations of the right and left eyes of the HMD 100 for each frame. Here, the position and orientation refers to a three-dimensional position and a three-dimensional orientation. There are various methods for calculating the position and orientation, and any method may be adopted. In this embodiment, the position and orientation calculation unit 102 calculates the positions and orientations of the right and left eyes for each frame using a technique called VISUAL SLAM from real-space images for the right and left eyes captured by the imaging unit 101. VISUAL SLAM is a technique for detecting characteristic points from real-space images and tracking changes in those points over time to calculate the position and orientation of the camera.

[0024] The CG data acquisition unit 103 acquires CG data for constructing a virtual space from an external CG data management server 300 .

[0025] The CG rendering unit 104 receives the CG data acquired by the CG data acquisition unit 103. Then, for each frame, the CG rendering unit 104 renders the CG data using the positions and orientations for the right and left eyes calculated by the position and orientation calculation unit 102 as viewpoints, and generates images of the virtual space for the right and left eyes. The viewing angle during rendering is set to match the viewing angle of the image captured by the imaging unit 101.

[0026] The image synthesis unit 105 synthesizes, for each frame, images of real space for the right eye and left eye captured by the imaging unit 101 and images of virtual space for the right eye and left eye generated by the CG rendering unit 104, to generate images of mixed reality space for the right eye and left eye.

[0027] The video display unit 106 displays the images of the mixed reality space for the right and left eyes generated by the video synthesis unit 105 for each frame on the screens in front of the right and left eyes of the HMD 100.

[0028] The communication unit 107 transmits, for each frame, images of the real space for the right eye and the left eye captured by the imaging unit 101 and the positions and orientations for the right eye and the left eye calculated by the position and orientation calculation unit 102 to the display device 200. The HMD 100 is the sharing source, and the display device 200 is the sharing destination.

[0029] The display device 200 includes a communication unit 201 , a video delay unit 202 , a CG data acquisition unit 203 , a CG rendering unit 204 , a video synthesis unit 205 , and a video display unit 206 .

[0030] The communication unit 201 receives the real space images for the right and left eyes and the positions and orientations for the right and left eyes transmitted from the HMD 100 for each frame.

[0031] The video delay unit 202 buffers the real space video for the right eye and the left eye received by the communication unit 201 for each frame, and holds the real space video frames up to the present. The video delay unit 202 can adjust the delay amount of the real space video by selecting the real space video frames to be used for synthesis according to the number of frames that the video synthesis unit 205 (described later) wants to delay.

[0032] The CG data acquisition unit 203 acquires CG data for constructing a virtual space from an external CG data management server 300 .

[0033] The CG rendering unit 204 renders the CG data acquired by the CG data acquisition unit 203 for each frame, using the positions and orientations for the right eye and left eye received by the communication unit 201 as viewpoints, and generates images of the virtual space for the right eye and left eye. Note that the viewing angle during rendering is made to match the viewing angle of the image captured by the imaging unit 101 of the HMD 100.

[0034] The image synthesis unit 205 synthesizes, for each frame, the images of real space for the right eye and the left eye held by the image delay unit 202 with the images of virtual space for the right eye and the left eye generated by the CG rendering unit 204, to generate images of mixed reality space for the right eye and the left eye.

[0035] The image display unit 206 displays the images of the mixed reality space for the right eye and the left eye generated by the image synthesis unit 205 on the screens in front of the right and left eyes of the display device 200.

[0036] FIG. 3 is a flowchart showing a processing method of the HMD 100, showing an example of processing for each frame by the HMD 100. FIG. 4 is a flowchart showing a processing method of the display device 200, showing an example of processing for each frame by the display device 200. FIG. 5 is a diagram showing the state of delay at each processing stage in the HMD 100 and the display device 200. Below, the processing flow of the HMD 100 and the display device 200 and the delay at each processing stage will be explained according to the flowcharts of FIGS. 3 and 4. The HMD 100 starts the processing of the flowchart of FIG. 3 at a predetermined timing for each frame. The display device 200 starts the processing of the flowchart of FIG. 4 at a predetermined timing for each frame.

[0037] Here, the delay refers to the time it takes for a change in the subject in real space, such as when the camera moves, to be reflected in the image. In the case of an image in real space, the delay is the time it takes for a captured image frame to be displayed. In the case of an image in virtual space, the delay is the time it takes from the time of the calculated position and orientation to the time the image in virtual space reflecting that position and orientation is displayed.

[0038] The horizontal axis of Fig. 5 represents time (frames). The upper part of Fig. 5 shows an image of the real space captured by the HMD 100, the position and orientation calculated by the HMD 100, an image of the virtual space generated by the HMD 100, and an image of the mixed reality space generated by the HMD 100. The lower part of Fig. 5 shows an image of the real space received by the display device 200, the position and orientation received by the display device 200, an image of the virtual space generated by the display device 200, and an image of the mixed reality space generated by the display device 200.

[0039] The times written in the real space images, positions and orientations, virtual space images, and mixed reality space images of the HMD 100 and the display device 200 indicate the times at which the captured frames are generated.

[0040] Here, the time required for the HMD 100 to calculate the position and orientation from the image in real space is defined as ts. The time required for rendering by each of the CG rendering units 104 and 204 is defined as tr. The time required for transmitting the image and position and orientation in real space from the HMD 100 to the display device 200 is defined as tn. Fig. 5 shows an example where ts = 1 frame, tr = 2 frames, and tn = 1 frame.

[0041] 3, the imaging unit 101 captures images of real space for the right eye and the left eye for each frame. At time t in FIG. 5, a frame of the image of real space at time t is captured.

[0042] In step S102, the position and orientation calculation unit 102 calculates the positions and orientations of the right and left eyes of the HMD 100 for each frame based on the real space images for the right and left eyes captured by the imaging unit 101. For example, the position and orientation calculation unit 102 calculates the position and orientation at time t based on the frame of the real space image at time t. Since it takes time ts to calculate the position and orientation, the position and orientation at time t is calculated at time t+ts, where ts=1 frame.

[0043] In step S103, the communication unit 107 transmits the frame of the image in real space at time t and the position and orientation at time t to the display device 200. The transmission takes time tn. At time t+ts in FIG. 5, the communication unit 107 transmits the frame of the image in real space at time t and the position and orientation at time t to the display device 200. At time t+ts+tn, the communication unit 201 of the display device 200 receives the frame of the image in real space at time t and the position and orientation at time t. Here, tn=1 frame.

[0044] In step S 104 , the CG data acquisition unit 103 acquires the CG data from the CG data management server 300 .

[0045] In step S105, the CG rendering unit 104 renders the CG data acquired by the CG data acquisition unit 103 for each frame, using the positions and orientations for the right and left eyes calculated by the position and orientation calculation unit 102 as viewpoints, to generate images of the virtual space for the right and left eyes. For example, the CG rendering unit 104 renders the CG data using the position and orientation at time t as a viewpoint, to generate an image of the virtual space at time t. Since rendering takes time tr, a frame of the image of the virtual space at time t is generated at time t + ts + tr, where tr = 2 frames.

[0046] In step S106, the image synthesis unit 105 synthesizes, for each frame, the images of the real space for the right eye and the left eye captured by the imaging unit 101 and the images of the virtual space for the right eye and the left eye generated by the CG rendering unit 104. The image synthesis unit 105 generates images of the mixed reality space for the right eye and the left eye by synthesizing for each frame. The image display unit 106 displays, for each frame, the images of the mixed reality space for the right eye and the left eye generated by the image synthesis unit 105 on the screens in front of the right and left eyes of the HMD 100.

[0047] For example, the image synthesis unit 105 synthesizes an image frame of the real space at time t+3 with an image frame of the virtual space at time t to generate an image frame of the mixed reality space at time t. The image display unit 106 displays the image frame of the mixed reality space at time t on screens in front of the right and left eyes of the HMD 100.

[0048] At time t+ts+tr, the imaging unit 101 captures a frame of real space video at time t+3, and the CG rendering unit 104 generates a frame of virtual space video at time t. At time t+ts+tr+1, the video composition unit 105 combines the frame of real space video at time t+3 with the frame of virtual space video at time t to generate a frame of mixed reality space video at time t. Here, in order to reduce delay in the real space video, the video composition unit 105 combines the latest frame of real space video at time t+3 with the frame of virtual space video at time t.

[0049] 4, the communication unit 201 receives, for each frame, images in real space for the right and left eyes and the positions and orientations for the right and left eyes from the HMD 100. At time t+ts+tn, the communication unit 201 receives the frame of the image in real space at time t and the position and orientation at time t.

[0050] In step S202, the video delay unit 202 buffers the real space video for the right eye and the left eye received by the communication unit 201 for each frame, and holds the frames of the real space video for the right eye and the left eye up to the present.

[0051] In step S 203 , the CG data acquisition unit 203 acquires CG data from the CG data management server 300 .

[0052] In step S204, the CG rendering unit 204 renders the CG data acquired by the CG data acquisition unit 203 for each frame, using the positions and orientations for the right and left eyes received by the communication unit 201 as viewpoints, to generate images of the virtual space for the right and left eyes. For example, the CG rendering unit 204 renders the CG data using the position and orientation at time t as a viewpoint, to generate an image of the virtual space at time t. Since rendering takes time tr, a frame of the image of the virtual space at time t is generated at time t+ts+tr+tn.

[0053] In step S205, the image synthesis unit 205 synthesizes, for each frame, the images of real space for the right eye and the left eye held by the image delay unit 202 and the images of virtual space for the right eye and the left eye generated by the CG rendering unit 204. The image synthesis unit 205 generates images of mixed reality space for the right eye and the left eye by synthesizing for each frame. The image display unit 206 displays, for each frame, the images of mixed reality space for the right eye and the left eye generated by the image synthesis unit 205 on the screens in front of the right and left eyes of the display device 200.

[0054] At time t+ts+tr+tn, the image synthesis unit 205 synthesizes the image frame of the real space at time t with the image frame of the virtual space at time t to generate an image frame of the mixed reality space at time t. The image display unit 206 displays the image frame of the mixed reality space at time t on screens in front of the right and left eyes of the display device 200.

[0055] The movement of the image displayed by the image display unit 206 does not reflect the movement of the viewer of the display device 200. Therefore, the viewer of the display device 200 feels little discomfort due to the delay until the movement of the person wearing the HMD 100 is reflected, and feels greater discomfort due to the difference in the amount of delay between the image in real space and the image in virtual space. Therefore, the image synthesis unit 205 synthesizes a frame of the image in real space at time t with a frame of the image in virtual space at time t. This allows the image display unit 206 to display an image in which the amount of delay between the image in real space and the image in virtual space matches.

[0056] As described above, the image synthesis unit 105 of the HMD 100 synthesizes a frame of image in real space at time t+3 with a frame of image in virtual space at time t to generate a frame of image in mixed reality space at time t. As a result, the image in mixed reality space in the HMD 100 reduces the amount of delay in the image in real space caused by the movement of the wearer of the HMD 100, and can reduce the sense of discomfort felt by the wearer of the HMD 100.

[0057] Furthermore, the image synthesis unit 205 of the display device 200 synthesizes a frame of an image in the real space at time t with a frame of an image in the virtual space at time t to generate a frame of an image in the mixed reality space at time t. As a result, the image in the mixed reality space on the display device 200 becomes an image in which the delay amounts of the frames of the image in the real space and the frames of the image in the virtual space are aligned, thereby reducing the sense of discomfort felt by a viewer of the display device 200.

[0058] Note that in step S102, the position and orientation calculation unit 102 may predict and calculate the position and orientation at time t+3, taking into account the time until display. In this case, in step S105, the CG rendering unit 104 renders the CG data using the position and orientation at time t+3 as the viewpoint, and generates an image of the virtual space at time t+3. Even in this case, in step S102, the position and orientation calculation unit 102 also calculates the position and orientation at time t. Then, in step S103, the communication unit 107 transmits the position and orientation at time t to the display device 200. In this way, if the prediction is correct, the HMD 100 can generate an image of the mixed reality space that makes it appear as if the amount of delay in the image of the virtual space is small. The display device 200 can generate an image of the mixed reality space based on the position and orientation of the HMD 100, which is not based on a future prediction.

[0059] Furthermore, at time t+3 in step S105, the CG rendering unit 104 generates a frame of video in the virtual space at time t. Thereafter, the position and orientation calculation unit 102 may recalculate the position and orientation at the current time t+3. Thereafter, the CG rendering unit 104 deforms the frame of video in the virtual space at time t based on the difference between the position and orientation at time t and the position and orientation at time t+3, and generates a frame of video in the virtual space at time t+3. In step S106, the image synthesis unit 105 synthesizes the frame of video in the real space at time t+3 with the frame of video in the virtual space at time t+3. In this way, the HMD 100 can generate a mixed reality space image that minimizes the amount of delay as much as possible by performing deformation after generating the virtual space image.

[0060] Furthermore, the image synthesis unit 205 may select frames from the frames of the image in real space that result in a predetermined delay amount. Furthermore, the HMD 100 may assign a timestamp to the image and position and orientation of real space for each frame, and transmit them to the display device 200. In this case, the display device 200 can select a frame of the image in real space using the frame and position and orientation of the image in real space that matches the timestamp, generate a frame of the image in virtual space, and synthesize the frames of the image in real space and the virtual space.

[0061] The display device 200 may also include a position and orientation calculation unit that calculates the position and orientation of the HMD 100 based on frames of images in real space received by the communication unit 201. In this case, the CG rendering unit 204 can generate frames of images in virtual space based on the calculated position and orientation.

[0062] As described above, the display system 500 includes the HMD 100 and the display device 200. The HMD 100 is an imaging device, and includes an imaging unit 101, a position and orientation calculation unit 102, a CG data acquisition unit 103, a CG rendering unit 104, an image synthesis unit 105, an image display unit 106, and a communication unit 107.

[0063] The imaging unit 101 captures an image of real space. The position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 based on the image of the real space. The CG rendering unit 104 is a generation unit that generates an image of a first virtual space based on the position and orientation of the HMD 100. The image synthesis unit 105 synthesizes the image of the real space captured by the imaging unit 101 with the image of the first virtual space to generate an image of a first mixed reality space, and displays the image of the first mixed reality space on the image display unit 106. The communication unit 107 transmits at least the image of the real space captured by the imaging unit 101 to the display device 200.

[0064] The position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 based on a first frame (e.g., a frame at time t) of the image of the real space. The CG rendering unit 104 generates a first frame of an image of a first virtual space based on the position and orientation of the HMD 100 based on the first frame of the image of the real space. The image synthesis unit 105 synthesizes a frame (e.g., a frame at time t+3) different from the first frame of the image of the real space with the first frame of the image of the first virtual space.

[0065] The display device 200 has a communication unit 201, an image delay unit 202, a CG data acquisition unit 203, a CG rendering unit 204, an image synthesis unit 205, and an image display unit 206. The communication unit 201 receives an image of the real space from the HMD 100. The CG rendering unit 204 is a generation unit that generates an image of a second virtual space based on the position and orientation of the HMD 100. The image synthesis unit 205 synthesizes the image of the real space received by the communication unit 201 with the image of the second virtual space to generate an image of a second mixed reality space, and displays the image of the second mixed reality space on the image display unit 206.

[0066] The CG rendering unit 204 generates a first frame of the image of the second virtual space based on the position and orientation of the HMD 100 that is based on the first frame of the image of the real space. The image synthesis unit 205 synthesizes the first frame of the image of the real space and the first frame of the image of the second virtual space.

[0067] For example, the communication unit 107 transmits a first frame of an image of real space captured by the imaging unit 101 and the position and orientation of the HMD 100 based on the first frame of the image of real space to the display device 200. The communication unit 201 receives the first frame of the image of real space and the position and orientation of the HMD 100 based on the first frame of the image of real space from the HMD 100. The CG rendering unit 204 generates a first frame of an image of a second virtual space based on the position and orientation of the HMD 100 based on the first frame of the image of real space received by the communication unit 201.

[0068] The display device 200 may include a second position and orientation calculation unit that calculates the position and orientation of the HMD 100 based on the first frame of the image of real space, based on the first frame of the image of real space received by the communication unit 201. In this case, the CG rendering unit 204 generates the first frame of the image of the second virtual space, based on the position and orientation of the HMD 100 based on the first frame of the image of real space calculated by the second position and orientation calculation unit.

[0069] For example, the image composition unit 105 combines a second frame (e.g., a frame at time t+3) that is later than the first frame in the image of the real space with the first frame in the image of the first virtual space. For example, the second frame in the image of the real space is the latest frame in the image of the real space.

[0070] For example, the position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 at the time of capturing the first frame of the image in real space based on the first frame of the image in real space. The CG rendering unit 104 generates the first frame of the image in the first virtual space based on the position and orientation of the HMD 100 at the time of capturing the first frame of the image in real space. The CG rendering unit 204 generates the first frame of the image in the second virtual space based on the position and orientation of the HMD 100 at the time of capturing the first frame of the image in real space.

[0071] The communication unit 107 transmits a first frame of the image of real space captured by the imaging unit 101 and the position and orientation of the HMD 100 at the time of capturing the first frame of the image of real space to the display device 200. The communication unit 201 receives the first frame of the image of real space and the position and orientation of the HMD 100 at the time of capturing the first frame of the image of real space from the HMD 100. The CG rendering unit 204 generates a first frame of the image of the second virtual space based on the position and orientation of the HMD 100 at the time of capturing the first frame of the image of real space received by the communication unit 201.

[0072] As described above, the display device 200 can be provided with a second position and orientation calculation unit. In this case, the second position and orientation calculation unit calculates the position and orientation of the HMD 100 at the time of capturing the first frame of the image in real space, based on the first frame of the image in real space received by the communication unit 201. The CG rendering unit 204 generates the first frame of the image in the second virtual space, based on the position and orientation of the HMD 100 at the time of capturing the first frame of the image in real space calculated by the second position and orientation calculation unit.

[0073] As described above, the image composition unit 105 combines a second frame (e.g., a frame at time t+3) that is later than the first frame of the image of real space with the first frame of the image of the first virtual space. The position and orientation calculation unit 102 predicts the position and orientation of the HMD 100 at the time of capturing the second frame of the image of real space based on the first frame of the image of real space. The CG rendering unit 104 generates the first frame of the image of the first virtual space based on the position and orientation of the HMD 100 at the time of capturing the second frame of the image of real space. The CG rendering unit 204 generates the first frame of the image of the second virtual space based on the position and orientation of the HMD 100 at the time of capturing the first frame of the image of real space.

[0074] Specifically, the position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 at the time of capturing the first frame of the image of real space, based on the first frame of the image of real space. Then, the position and orientation calculation unit 102 predicts the position and orientation of the HMD 100 at the time of capturing the second frame of the image of real space, based on the first frame of the image of real space. The CG rendering unit 104 generates a first frame of the image of the first virtual space, based on the position and orientation of the HMD 100 at the time of capturing the second frame of the image of real space. The communication unit 107 transmits the first frame of the image of real space captured by the imaging unit 101 and the position and orientation of the HMD 100 at the time of capturing the first frame of the image of real space to the display device 200.

[0075] The communication unit 201 receives a first frame of an image of real space and the position and orientation of the HMD 100 at the time of capturing the first frame of the image of real space from the HMD 100. The CG rendering unit 204 generates a first frame of an image of a second virtual space based on the position and orientation of the HMD 100 at the time of capturing the first frame of the image of real space received by the communication unit 201.

[0076] As described above, the display device 200 can be provided with a second position and orientation calculation unit. In this case, the second position and orientation calculation unit calculates the position and orientation of the HMD 100 at the time of capturing the first frame of the image in real space, based on the first frame of the image in real space received by the communication unit 201. The CG rendering unit 204 generates the first frame of the image in the second virtual space, based on the position and orientation of the HMD 100 at the time of capturing the first frame of the image in real space calculated by the second position and orientation calculation unit.

[0077] Furthermore, the image composition unit 105 combines a second frame (e.g., a frame at time t+3) that is later than the first frame of the image of real space with the first frame of the image of the first virtual space. In this case, the position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 at the time of capturing the first frame of the image of real space, based on the first frame of the image of real space. The CG rendering unit 104 generates a temporary frame of the image of the first virtual space at the time of capturing the first frame of the image of real space, based on the position and orientation of the HMD 100 at the time of capturing the first frame of the image of real space. The position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 at the time of capturing the second frame of the image of real space, based on the second frame of the image of real space. The CG rendering unit 104 generates a first frame of the first virtual space image at the time of capturing the second frame of the real space image based on the difference between the position and orientation of the HMD 100 at the time of capturing the first frame of the real space image and the position and orientation of the HMD 100 at the time of capturing the second frame of the real space image, and a virtual frame of the first virtual space image.

[0078] Furthermore, the communication unit 107 transmits a first frame of the image of real space captured by the imaging unit 101 and the position and orientation of the HMD 100 at the time of capturing the first frame of the image of real space to the display device 200. The communication unit 201 receives the first frame of the image of real space and the position and orientation of the HMD 100 at the time of capturing the first frame of the image of real space from the HMD 100. The CG rendering unit 204 generates a first frame of the image of the second virtual space based on the position and orientation of the HMD 100 at the time of capturing the first frame of the image of real space received by the communication unit 201.

[0079] As described above, the display device 200 can be provided with a second position and orientation calculation unit. In this case, the second position and orientation calculation unit calculates the position and orientation of the HMD 100 at the time of capturing the first frame of the image in real space, based on the first frame of the image in real space received by the communication unit 201. The CG rendering unit 204 generates the first frame of the image in the second virtual space, based on the position and orientation of the HMD 100 at the time of capturing the first frame of the image in real space calculated by the second position and orientation calculation unit.

[0080] As described above, according to this embodiment, the HMD 100 can display a mixed reality space image with a small delay for the real space image by reducing the delay amount for the real space image among the mixed reality space images seen by the wearer of the HMD 100. The display device 200 can display a mixed reality space image with the same delay amount for the real space image and the virtual space image.

[0081] (Second embodiment) Next, a second embodiment will be described. In the second embodiment, an example will be described in which the HMD 100 generates mixed reality space images with different delay amounts for the mixed reality space images displayed on its own image display unit 106 and transmits the images to the display device 200. In this way, the HMD 100 generates mixed reality space images to be displayed on the display device 200, so that the display device 200 can be any device that has the function of playing and displaying general streaming format images.

[0082] The following description of the second embodiment will focus on the differences from the first embodiment. In the second embodiment, the same configurations or steps as those in the first embodiment will be denoted by the same reference numerals as those in the first embodiment.

[0083] Fig. 6 is a block diagram showing an example of the functional configuration of an HMD 100 according to the second embodiment. The HMD 100 in Fig. 6 is obtained by adding a video delay unit 401 to the HMD 100 in Fig. 2. The HMD 100 includes an imaging unit 101, a position and orientation calculation unit 102, a CG data acquisition unit 103, a CG rendering unit 104, a video synthesis unit 105, a video display unit 106, a communication unit 107, and the video delay unit 401.

[0084] Fig. 7 is a flowchart showing a processing method of the HMD 100 in Fig. 6, and shows an example of processing for each frame by the HMD 100. Fig. 7 is obtained by deleting step S103 from Fig. 3 and adding steps S401 and S402. The HMD 100 starts the processing of the flowchart in Fig. 7 at a predetermined timing for each frame. The HMD 100 performs the processing of steps S101, S102, and S104 to S106, similar to Fig. 3.

[0085] In step S101, the video delay unit 202 buffers the real space video for the right eye and the left eye captured by the imaging unit 101 for each frame, and holds the frames of real space video for the right eye and the left eye up to the present.

[0086] In step S106, the HMD 100 combines the latest frame of the image in the real space at time t+3 with the frame of the image in the virtual space at time t to generate an image in the mixed reality space to be displayed on the HMD 100.

[0087] Then, in step S401, the image synthesis unit 105 synthesizes, for each frame, the real space images for the right eye and the left eye held by the image delay unit 401 and the virtual space images for the right eye and the left eye generated by the CG rendering unit 104. The image synthesis unit 105 generates mixed reality space images for the right eye and the left eye by synthesizing each frame. Specifically, the image synthesis unit 105 synthesizes a frame of real space image at time t with a frame of virtual space image at time t to generate a frame of mixed reality space image at time t.

[0088] In step S402, the communication unit 107 transmits the images of the mixed reality space for the right eye and the left eye generated in step S401 to the display device 200 in a general streaming format for each frame.

[0089] In the display device 200, the communication unit 201 receives images of the mixed reality space for the right eye and the left eye from the HMD 100. The image display unit 206 displays the images of the mixed reality space for the right eye and the left eye received by the communication unit 201 on screens in front of the right and left eyes of the display device 200.

[0090] As described above, the HMD 100 generates an image of the mixed reality space to be displayed on the HMD 100 and an image of the mixed reality space to be displayed on the display device 200.

[0091] The HMD 100 displays a frame of video in mixed reality space that is a combination of a frame of video in real space at time t+3 and a frame of video in virtual space at time t. As a result, the video in mixed reality space displayed on the HMD 100 reduces the amount of delay in the image in real space caused by the movement of the wearer of the HMD 100, thereby reducing the sense of discomfort felt by the wearer of the HMD 100.

[0092] Furthermore, display device 200 displays a frame of an image in a mixed reality space obtained by combining a frame of an image in a real space at time t and a frame of an image in a virtual space at time t. As a result, the image in the mixed reality space displayed on display device 200 has the same amount of delay between the frame of the image in the real space and the frame of the image in the virtual space, thereby reducing the sense of discomfort felt by a viewer of display device 200.

[0093] As described above, the HMD 100 is an imaging device and includes an imaging unit 101, a position and orientation calculation unit 102, a CG data acquisition unit 103, a CG rendering unit 104, an image synthesis unit 105, an image display unit 106, a communication unit 107, and an image delay unit 401.

[0094] The imaging unit 101 captures an image of real space. The position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 based on the image of real space. The CG rendering unit 104 generates an image of virtual space based on the position and orientation of the HMD 100. The image synthesis unit 105 generates an image of mixed reality space by synthesizing the image of real space captured by the imaging unit 101 and the image of virtual space generated by the CG rendering unit 104. The communication unit 107 transmits the image of mixed reality space to the display device 200.

[0095] The position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 based on a first frame (e.g., a frame at time t) of the image in real space. The CG rendering unit 104 generates a first frame of the image in virtual space based on the position and orientation of the HMD 100 based on the first frame of the image in real space. The image synthesis unit 105 generates the first frame of the image in the first mixed reality space by synthesizing a frame (e.g., a frame at time t+3) different from the first frame of the image in real space with the first frame of the image in virtual space. The image synthesis unit 105 then displays the first frame of the image in the first mixed reality space on the image display unit 106.

[0096] Furthermore, the image composition unit 105 generates the first frame of the image of the second mixed reality space by combining the first frame of the image of the real space with the first frame of the image of the virtual space. The communication unit 107 transmits the first frame of the image of the second mixed reality space to the display device 200.

[0097] Specifically, the image composition unit 105 generates the first frame of the first mixed reality space image by combining a second frame (e.g., a frame at time t+3) that is later than the first frame of the real space image with the first frame of the first virtual space image. The second frame of the real space image is, for example, the latest frame of the real space image.

[0098] The position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 at the time of capturing the first frame of the image in real space based on the first frame of the image in real space. The CG rendering unit 104 generates the first frame of the image in virtual space based on the position and orientation of the HMD 100 at the time of capturing the first frame of the image in real space.

[0099] The position and orientation calculation unit 102 can predict the position and orientation of the HMD 100 at the time of capturing a second frame of the real space image, based on the first frame of the real space image. In this case, the CG rendering unit 104 generates a first frame of the first virtual space image, based on the position and orientation of the HMD 100 at the time of capturing the second frame of the real space image. The image composition unit 105 generates a first frame of the first mixed reality space image by combining the second frame of the real space image with the first frame of the first virtual space image. The CG rendering unit 104 generates a first frame of the second virtual space image based on the position and orientation of the HMD 100 at the time of capturing the first frame of the real space image. The image composition unit 105 generates a first frame of the second mixed reality space image by combining the first frame of the real space image with the first frame of the second virtual space image.

[0100] Furthermore, the position and orientation calculation unit 102 can calculate the position and orientation of the HMD 100 at the time of capturing the first frame of the real space image, based on the first frame of the real space image. The CG rendering unit 104 generates a temporary frame of the first virtual space image at the time of capturing the first frame of the real space image, based on the position and orientation of the HMD 100 at the time of capturing the first frame of the real space image. The position and orientation calculation unit 102 calculates the position and orientation of the HMD 100 at the time of capturing the second frame of the real space image, based on the second frame of the real space image. The CG rendering unit 104 generates the first frame of the first virtual space image at the time of capturing the second frame of the real space image, based on the difference between the position and orientation of the HMD 100 at the time of capturing the first frame of the real space image and the position and orientation of the HMD 100 at the time of capturing the second frame of the real space image, and the temporary frame of the first virtual space image.

[0101] As described above, according to this embodiment, the HMD 100 can display a mixed reality space image with a small delay for the real space image by reducing the delay amount for the real space image among the mixed reality space images seen by the wearer of the HMD 100. The display device 200 can display a mixed reality space image with the same delay amount for the real space image and the virtual space image.

[0102] (Other embodiments) The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0103] It should be noted that the above-described embodiments merely illustrate specific examples of implementing the present disclosure, and the technical scope of the present disclosure should not be construed as being limited by these embodiments. In other words, the present disclosure can be implemented in various forms without departing from its technical concept or main features.

[0104] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) An imaging device; a display device, The imaging device is an imaging means for capturing an image of real space; a first position and orientation calculation means for calculating a position and orientation of the image capturing device based on the image in the real space; a first generation means for generating an image of a first virtual space based on the position and orientation of the imaging device; a first image composition means for generating an image of a first mixed reality space by combining an image of the real space captured by the imaging means with an image of the first virtual space, and displaying the image of the first mixed reality space; a first communication means for transmitting at least an image of the real space captured by the imaging means to the display device; the first position and orientation calculation means calculates a position and orientation of the image capture device based on a first frame of the image in the real space; the first generation means generates a first frame of the image of the first virtual space based on a position and orientation of the imaging device that is based on a first frame of the image of the real space; the first image synthesis means synthesizes a frame different from the first frame in the image of the real space with the first frame in the image of the first virtual space; The display device includes: a second communication means for receiving an image of the real space from the imaging device; a second generation means for generating an image of a second virtual space based on the position and orientation of the imaging device; a second image composition means for generating an image of a second mixed reality space by combining an image of the real space received by the second communication means with an image of the second virtual space, and displaying the image of the second mixed reality space; the second generation means generates a first frame of the image of the second virtual space based on a position and orientation of the imaging device that is based on a first frame of the image of the real space; A display system characterized in that the second image synthesis means synthesizes a first frame of the image of the real space with a first frame of the image of the second virtual space. (Configuration 2) the first communication means transmits to the display device a first frame of an image of real space captured by the imaging means and a position and orientation of the imaging device based on the first frame of the image of real space; the second communication means receives from the imaging device a first frame of the image of the real space and a position and orientation of the imaging device based on the first frame of the image of the real space; The display system described in configuration 1, wherein the second generation means generates a first frame of the image of the second virtual space based on the position and orientation of the imaging device that is based on a first frame of the image of the real space received by the second communication means. (Configuration 3) the display device includes a second position and orientation calculation means that calculates a position and orientation of the imaging device based on a first frame of the image of the real space, based on a first frame of the image of the real space received by the second communication means; The display system described in configuration 1, wherein the second generation means generates a first frame of the image of the second virtual space based on the position and orientation of the imaging device that is based on the first frame of the image of the real space calculated by the second position and orientation calculation means. (Configuration 4) The display system according to any one of configurations 1 to 3, wherein the first image synthesis means synthesizes a second frame that comes after the first frame of the image of the real space with the first frame of the image of the first virtual space. (Configuration 5) 5. The display system of claim 4, wherein the second frame of the image of real space is the most recent frame of the image of real space. (Configuration 6) the first position and orientation calculation means calculates, based on a first frame of the image in real space, a position and orientation of the image capture device at the time of capturing the first frame of the image in real space; the first generation means generates a first frame of the image of the first virtual space based on a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; The display system according to any one of configurations 1 to 5, wherein the second generation means generates a first frame of the image of the second virtual space based on the position and orientation of the imaging device at the time of capturing the first frame of the image of the real space. (Configuration 7) the first position and orientation calculation means calculates, based on a first frame of the image in real space, a position and orientation of the image capture device at the time of capturing the first frame of the image in real space; the first generation means generates a first frame of the image of the first virtual space based on a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; the first communication means transmits to the display device a first frame of the image of real space captured by the imaging means and a position and orientation of the imaging device at the time of capturing the first frame of the image of real space; the second communication means receives from the imaging device a first frame of the image of the real space and a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; The display system according to configuration 2, wherein the second generation means generates a first frame of the image of the second virtual space based on the position and orientation of the imaging device at the time of capturing the first frame of the image of the real space received by the second communication means. (Configuration 8) the first position and orientation calculation means calculates, based on a first frame of the image in real space, a position and orientation of the image capture device at the time of capturing the first frame of the image in real space; the first generation means generates a first frame of the image of the first virtual space based on a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; the second position and orientation calculation means calculates a position and orientation of the imaging device at the time of capturing a first frame of the image of real space based on the first frame of the image of real space received by the second communication means; The display system described in configuration 3, wherein the second generation means generates a first frame of the image of the second virtual space based on the position and orientation of the imaging device at the time of capturing the first frame of the image of the real space calculated by the second position and orientation calculation means. (Configuration 9) the first image synthesis means synthesizes a second frame that is later than the first frame in the image of the real space with the first frame in the image of the first virtual space; the first position and orientation calculation means predicts a position and orientation of the image capture device at the time of capturing a second frame of the image in real space based on a first frame of the image in real space; the first generation means generates a first frame of the image of the first virtual space based on a position and orientation of the imaging device at the time of capturing a second frame of the image of the real space; The display system according to configuration 1, wherein the second generation means generates a first frame of the image of the second virtual space based on the position and orientation of the imaging device at the time of capturing the first frame of the image of the real space. (Configuration 10) the first position and orientation calculation means calculates, based on a first frame of the image in real space, a position and orientation of the image capture device at the time of capturing a first frame of the image in real space, and predicts a position and orientation of the image capture device at the time of capturing a second frame of the image in real space; the first generation means generates a first frame of the image of the first virtual space based on a position and orientation of the imaging device at the time of capturing a second frame of the image of the real space; the first communication means transmits to the display device a first frame of the image of real space captured by the imaging means and a position and orientation of the imaging device at the time of capturing the first frame of the image of real space; the second communication means receives from the imaging device a first frame of the image of the real space and a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; The display system of configuration 9, wherein the second generation means generates a first frame of the image of the second virtual space based on the position and orientation of the imaging device at the time of capturing the first frame of the image of the real space received by the second communication means. (Configuration 11) the display device includes second position and orientation calculation means for calculating, based on the first frame of the image of real space received by the second communication means, a position and orientation of the image capture device at the time of capturing the first frame of the image of real space; The display system described in configuration 9, wherein the second generation means generates a first frame of the image of the second virtual space based on the position and orientation of the imaging device at the time of capturing the first frame of the image of the real space calculated by the second position and orientation calculation means. (Configuration 12) the first image synthesis means synthesizes a second frame that is later than the first frame in the image of the real space with the first frame in the image of the first virtual space; the first position and orientation calculation means calculates, based on a first frame of the image in real space, a position and orientation of the image capture device at the time of capturing the first frame of the image in real space; the first generation means generates a provisional frame of the image of the first virtual space at the time of capturing the first frame of the image of the real space, based on a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; the first position and orientation calculation means calculates, based on a second frame of the image in real space, a position and orientation of the image capture device at the time of capturing the second frame of the image in real space; The display system described in configuration 1, characterized in that the first generation means generates a first frame of the first virtual space image at the time of capturing a second frame of the real space image based on a difference between a position and orientation of the imaging device at the time of capturing a first frame of the real space image and a position and orientation of the imaging device at the time of capturing a second frame of the real space image, and a provisional frame of the first virtual space image. (Configuration 13) the first communication means transmits to the display device a first frame of the image of real space captured by the imaging means and a position and orientation of the imaging device at the time of capturing the first frame of the image of real space; the second communication means receives from the imaging device a first frame of the image of the real space and a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; The display system described in configuration 12, wherein the second generation means generates a first frame of the image of the second virtual space based on the position and orientation of the imaging device at the time of capturing the first frame of the image of the real space received by the second communication means. (Configuration 14) the display device includes second position and orientation calculation means for calculating, based on the first frame of the image of real space received by the second communication means, a position and orientation of the image capture device at the time of capturing the first frame of the image of real space; The display system described in configuration 12, wherein the second generation means generates a first frame of the image of the second virtual space based on the position and orientation of the imaging device at the time of capturing the first frame of the image of the real space calculated by the second position and orientation calculation means. (Configuration 15) An imaging device, an imaging means for capturing an image of real space; a position and orientation calculation means for calculating a position and orientation of the image capturing device based on the image in the real space; a generating means for generating an image of a virtual space based on the position and orientation of the imaging device; an image synthesis means for synthesizing an image of the real space captured by the imaging means and an image of the virtual space generated by the generation means to generate an image of the mixed reality space; a communication means for transmitting an image of the mixed reality space to a display device; the position and orientation calculation means calculates a position and orientation of the image capture device based on a first frame of the image in real space; the generating means generates a first frame of the image of the virtual space based on a position and orientation of the imaging device that is based on a first frame of the image of the real space; the image synthesis means generates a first frame of an image of a first mixed reality space by synthesizing a frame different from the first frame of the image of the real space with the first frame of the image of the virtual space, and displays the first frame of the image of the first mixed reality space; the image synthesis means synthesizes a first frame of the image of the real space and a first frame of the image of the virtual space to generate a first frame of the image of the second mixed reality space; The imaging device, wherein the communication means transmits a first frame of the image of the second mixed reality space to the display device. (Configuration 16) The imaging device described in configuration 15, wherein the image synthesis means generates the first frame of the image of the first mixed reality space by synthesizing a second frame that comes after the first frame of the image of the real space with the first frame of the image of the first virtual space. (Configuration 17) 17. The imaging device according to claim 16, wherein the second frame of the image of the real space is the most recent frame of the image of the real space. (Configuration 18) the position and orientation calculation means calculates, based on a first frame of the image in real space, a position and orientation of the image capture device at the time of capturing the first frame of the image in real space; The imaging device described in any one of configurations 15 to 17, characterized in that the generation means generates the first frame of the image of the virtual space based on the position and orientation of the imaging device at the time of capturing the first frame of the image of the real space. (Configuration 19) the position and orientation calculation means predicts a position and orientation of the imaging device at the time of capturing a second frame of the image in real space based on a first frame of the image in real space; the generating means generates a first frame of the image of the first virtual space based on a position and orientation of the imaging device at the time of capturing a second frame of the image of the real space; the image synthesis means generates a first frame of the image of the first mixed reality space by synthesizing a second frame of the image of the real space and a first frame of the image of the first virtual space; the generating means generates a first frame of the image of the second virtual space based on a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; The imaging device according to configuration 16 or 17, wherein the image synthesis means generates the first frame of the image of the second mixed reality space by synthesizing a first frame of the image of the real space with a first frame of the image of the second virtual space. (Configuration 20) the position and orientation calculation means calculates, based on a first frame of the image in real space, a position and orientation of the image capture device at the time of capturing the first frame of the image in real space; the generation means generates a provisional frame of the first virtual space image at the time of capturing the first frame of the real space image, based on a position and orientation of the imaging device at the time of capturing the first frame of the real space image; the position and orientation calculation means calculates, based on a second frame of the image in real space, a position and orientation of the image capture device at the time of capturing the second frame of the image in real space; The imaging device described in configuration 16 or 17, characterized in that the generation means generates a first frame of the first virtual space image at the time of capturing a second frame of the real space image based on a difference between a position and orientation of the imaging device at the time of capturing a first frame of the real space image and a position and orientation of the imaging device at the time of capturing a second frame of the real space image, and a provisional frame of the first virtual space image. (Method 1) An imaging device; a display device; and a processing method for a display system having the display device, comprising: The imaging device is an imaging means for capturing an image of real space; a first position and orientation calculation means for calculating a position and orientation of the image capturing device based on the image in the real space; a first generation means for generating an image of a first virtual space based on the position and orientation of the imaging device; a first image composition means for generating an image of a first mixed reality space by combining an image of the real space captured by the imaging means with an image of the first virtual space, and displaying the image of the first mixed reality space; a first communication means for transmitting at least an image of the real space captured by the imaging means to the display device; The display device includes: a second communication means for receiving an image of the real space from the imaging device; a second generation means for generating an image of a second virtual space based on the position and orientation of the imaging device; a second image composition means for generating an image of a second mixed reality space by combining an image of the real space received by the second communication means with an image of the second virtual space, and displaying the image of the second mixed reality space; The processing method of the display system includes: a step in which the first position and orientation calculation means calculates the position and orientation of the image capture device based on a first frame of the image in the real space; a step in which the first generation means generates a first frame of the image of the first virtual space based on a position and orientation of the imaging device that is based on a first frame of the image of the real space; a step in which the first image synthesis means synthesizes a frame different from the first frame in the image of the real space with the first frame in the image of the first virtual space; a step in which the second generation means generates a first frame of the image of the second virtual space based on a position and orientation of the imaging device that is based on a first frame of the image of the real space; a step in which the second image synthesis means synthesizes a first frame of the image of the real space and a first frame of the image of the second virtual space; 10. A processing method for a display system, comprising: (Method 2) A processing method for an imaging device, comprising: The imaging device is an imaging means for capturing an image of real space; a position and orientation calculation means for calculating a position and orientation of the image capturing device based on the image in the real space; a generating means for generating an image of a virtual space based on the position and orientation of the imaging device; an image synthesis means for synthesizing an image of the real space captured by the imaging means and an image of the virtual space generated by the generation means to generate an image of the mixed reality space; a communication means for transmitting an image of the mixed reality space to a display device; The processing method of the imaging device includes: a step in which the position and orientation calculation means calculates the position and orientation of the image capture device based on a first frame of the image in the real space; generating a first frame of the image of the virtual space based on a position and orientation of the imaging device that is based on a first frame of the image of the real space; a step in which the image synthesis means synthesizes a frame different from the first frame of the image of the real space with the first frame of the image of the virtual space to generate a first frame of the image of a first mixed reality space, and displays the first frame of the image of the first mixed reality space; a step in which the image synthesis means synthesizes a first frame of the image of the real space and a first frame of the image of the virtual space to generate a first frame of the image of the second mixed reality space; transmitting a first frame of the image of the second mixed reality space to the display device by the communication means; 10. A processing method for an imaging device, comprising: [Explanation of symbols]

[0105] 101 imaging unit, 102 position and orientation calculation unit, 103 CG data acquisition unit, 104 CG rendering unit, 105 video synthesis unit, 106 video display unit, 107 communication unit, 201 communication unit, 202 video delay unit, 203 CG data acquisition unit, 204 CG rendering unit, 205 video synthesis unit, 206 video display unit

Claims

1. A display system comprising an imaging device having an imaging means for capturing an image of real space and a display means, and a display device, a position and orientation acquisition means for acquiring a position and orientation of the image capturing device based on the image in the real space; a generating means for generating an image of a virtual space based on the position and orientation; a first image synthesis means for synthesizing the image of the real space and the image of the virtual space to generate an image of a first mixed reality space to be displayed on the display means; a second image composition means for combining the image of the real space and the image of the virtual space to generate an image of a second mixed reality space to be displayed on the display device; the position and orientation acquisition means acquires a position and orientation of the imaging device based on a first frame of the image of the real space; the generating means generates a first frame of the image of the virtual space based on a position and orientation of the imaging device that is based on the first frame of the image of the real space; the first image synthesis means synthesizes a frame different from the first frame in the image of the real space with the first frame in the image of the virtual space; The second image synthesizing means synthesizes the first frame of the image of the real space with the first frame of the image of the virtual space. A display system characterized by:

2. The imaging device comprises: a first position and orientation acquisition means for acquiring a position and orientation of the image capturing device based on the image of the real space; a first generation means for generating an image of a first virtual space based on the position and orientation of the imaging device; the first image composition means for generating an image of a first mixed reality space by combining an image of the real space with an image of the first virtual space, and displaying the image of the first mixed reality space; a first communication means for transmitting at least the image of the real space to the display device; the first position and orientation acquisition means acquires a position and orientation of the imaging device based on the first frame of the image in the real space; the first generation means generates a first frame of the image of the first virtual space based on a position and orientation of the imaging device that is based on the first frame of the image of the real space; The first image synthesizing means synthesizes the frame different from the first frame in the image of the real space with the first frame in the image of the first virtual space.

2. The display system of claim 1.

3. The display device a second communication means for receiving an image of the real space from the imaging device; a second generating means for generating an image of a second virtual space based on the position and orientation of the imaging device; a second image composition means for generating an image of a second mixed reality space by combining the image of the real space received by the second communication means with the image of the second virtual space, and displaying the image of the second mixed reality space; the second generation means generates a first frame of the image of the second virtual space based on a position and orientation of the imaging device that is based on the first frame of the image of the real space; The second image synthesizing means synthesizes the first frame of the image of the real space with the first frame of the image of the second virtual space.

3. The display system according to claim 2.

4. the first communication means transmits, to the display device, a first frame of an image of real space captured by the imaging means and a position and orientation of the imaging device based on the first frame of the image of real space; the second communication means receives from the imaging device a first frame of the image of the real space and a position and orientation of the imaging device based on the first frame of the image of the real space; The second generating means generates a first frame of the image of the second virtual space based on a position and orientation of the imaging device that is based on a first frame of the image of the real space received by the second communication means.

4. The display system according to claim 3.

5. the display device includes a second position and orientation acquisition means that acquires a position and orientation of the imaging device based on a first frame of the image of the real space, based on a first frame of the image of the real space received by the second communication means; The second generating means generates a first frame of the image of the second virtual space based on a position and orientation of the imaging device that is based on a first frame of the image of the real space acquired by the second position and orientation acquiring means.

4. The display system according to claim 3.

6. The first image synthesis means synthesizes a second frame, which is a frame subsequent to the first frame, of the image of the real space with the first frame of the image of the virtual space.

2. The display system of claim 1.

7. The second frame of the image of the real space is the most recent frame of the image of the real space.

7. The display system of claim 6.

8. the first position and orientation acquisition means acquires, based on a first frame of the image in the real space, a position and orientation of the image capture device at the time of capturing the first frame of the image in the real space; the first generation means generates a first frame of the image of the first virtual space based on a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; The second generating means generates a first frame of the image of the second virtual space based on a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space.

4. The display system according to claim 3.

9. the first position and orientation acquisition means acquires, based on a first frame of the image in the real space, a position and orientation of the image capture device at the time of capturing the first frame of the image in the real space; the first generation means generates a first frame of the image of the first virtual space based on a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; the first communication means transmits to the display device a first frame of the image of real space captured by the imaging means and a position and orientation of the imaging device at the time of capturing the first frame of the image of real space; the second communication means receives from the imaging device a first frame of the image of the real space and a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; The second generating means generates a first frame of the image of the second virtual space based on a position and orientation of the imaging device at the time of capturing a first frame of the image of the real space received by the second communication means.

5. The display system according to claim 4.

10. the first position and orientation acquisition means acquires, based on a first frame of the image in the real space, a position and orientation of the image capture device at the time of capturing the first frame of the image in the real space; the first generation means generates a first frame of the image of the first virtual space based on a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; the second position and orientation acquisition means acquires a position and orientation of the imaging device at the time of capturing a first frame of the image of the real space based on the first frame of the image of the real space received by the second communication means; The second generating means generates a first frame of the image of the second virtual space based on the position and orientation of the imaging device at the time of capturing a first frame of the image of the real space acquired by the second position and orientation acquiring means.

6. The display system according to claim 5.

11. the first image synthesis means synthesizes a second frame of the image of the real space that is subsequent to the first frame and the first frame of the image of the first virtual space; the first position and orientation acquisition means predicts a position and orientation of the image capture device at the time of capturing a second frame of the image in real space based on a first frame of the image in real space; the first generation means generates a first frame of the image of the first virtual space based on a position and orientation of the imaging device at the time of capturing a second frame of the image of the real space; The second generating means generates a first frame of the image of the second virtual space based on a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space.

4. The display system according to claim 3.

12. the first position and orientation acquisition means acquires, based on a first frame of the image in real space, a position and orientation of the image capture device at the time of capturing a first frame of the image in real space, and predicts a position and orientation of the image capture device at the time of capturing a second frame of the image in real space; the first generation means generates a first frame of the image of the first virtual space based on a position and orientation of the imaging device at the time of capturing a second frame of the image of the real space; the first communication means transmits to the display device a first frame of the image of real space captured by the imaging means and a position and orientation of the imaging device at the time of capturing the first frame of the image of real space; the second communication means receives from the imaging device a first frame of the image of the real space and a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; The second generating means generates a first frame of the image of the second virtual space based on a position and orientation of the imaging device at the time of capturing a first frame of the image of the real space received by the second communication means.

12. The display system of claim 11.

13. the display device includes a second position and orientation acquisition means for acquiring a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space based on the first frame of the image of the real space received by the second communication means; The second generating means generates a first frame of the image of the second virtual space based on the position and orientation of the imaging device at the time of capturing a first frame of the image of the real space acquired by the second position and orientation acquiring means.

12. The display system of claim 11.

14. the first image synthesis means synthesizes a second frame of the image of the real space that is subsequent to the first frame and the first frame of the image of the first virtual space; the first position and orientation acquisition means acquires, based on a first frame of the image in the real space, a position and orientation of the image capture device at the time of capturing the first frame of the image in the real space; the first generation means generates a provisional frame of the image of the first virtual space at the time of capturing the first frame of the image of the real space, based on a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; the first position and orientation acquisition means acquires, based on a second frame of the image in real space, a position and orientation of the image capture device at the time of capturing the second frame of the image in real space; The first generation means generates a first frame of the first virtual space image at the time of capturing a second frame of the real space image, based on a difference between a position and orientation of the image capturing device at the time of capturing a first frame of the real space image and a position and orientation of the image capturing device at the time of capturing a second frame of the real space image, and a provisional frame of the first virtual space image.

4. The display system according to claim 3.

15. the first communication means transmits to the display device a first frame of the image of real space captured by the imaging means and a position and orientation of the imaging device at the time of capturing the first frame of the image of real space; the second communication means receives from the imaging device a first frame of the image of the real space and a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space; The second generating means generates a first frame of the image of the second virtual space based on a position and orientation of the imaging device at the time of capturing a first frame of the image of the real space received by the second communication means.

15. The display system of claim 14.

16. the display device includes a second position and orientation acquisition means for acquiring a position and orientation of the imaging device at the time of capturing the first frame of the image of the real space based on the first frame of the image of the real space received by the second communication means; The second generating means generates a first frame of the image of the second virtual space based on the position and orientation of the imaging device at the time of capturing a first frame of the image of the real space acquired by the second position and orientation acquiring means.

15. The display system of claim 14.

17. The imaging device is a head-mounted display.

2. The display system of claim 1.

18. An imaging device having an imaging means for capturing an image of real space and a display means, a position and orientation acquisition means for acquiring a position and orientation of the image capturing device based on the image in the real space; a generating means for generating an image of a virtual space based on the position and orientation; a first image synthesis means for synthesizing the image of the real space and the image of the virtual space to generate an image of a first mixed reality space to be displayed on the display means; a second image composition means for combining the image of the real space and the image of the virtual space to generate an image of a second mixed reality space to be displayed on an external display device; the position and orientation acquisition means acquires a position and orientation of the imaging device based on a first frame of the image of the real space; the generating means generates a first frame of the image of the virtual space based on a position and orientation of the imaging device that is based on the first frame of the image of the real space; the first image synthesis means synthesizes a frame different from the first frame in the image of the real space with the first frame in the image of the virtual space; The second image synthesizing means synthesizes the first frame of the image of the real space with the first frame of the image of the virtual space. An imaging device characterized by:

19. A processing method for a display system having an imaging device having an imaging means for capturing an image of real space and a display means, and a display device, comprising: a position and orientation acquisition step of acquiring a position and orientation of the imaging device based on the image in the real space; a generating step of generating an image of a virtual space based on the position and orientation; a first image synthesis step of synthesizing the image of the real space and the image of the virtual space to generate an image of a first mixed reality space to be displayed on the display means; a second image synthesis step of synthesizing the image of the real space and the image of the virtual space to generate an image of a second mixed reality space to be displayed on the display device, the position and orientation acquiring step acquires a position and orientation of the image capturing device based on a first frame of the image in the real space; In the generating step, a first frame of the image of the virtual space is generated based on a position and orientation of the imaging device that is based on the first frame of the image of the real space; In the first image synthesis step, a frame different from the first frame in the image of the real space is synthesized with the first frame in the image of the virtual space; In the second image synthesis step, the first frame of the image of the real space and the first frame of the image of the virtual space are synthesized. A processing method for a display system.

Images