Image display apparatus, method for controlling image display apparatus, and storage medium

Abstract

To shorten a delay time required from the acquisition of a picked-up image to the start of display than before.SOLUTION: Acquiring, by an image display device, a first virtual image based on a first captured image and first posture information corresponding to the first captured image, the first virtual image representing a virtual space viewed from a viewpoint corresponding to a position and posture of an imaging unit when the first captured image is acquired, and acquiring a first corrected virtual image by correcting the first virtual image based on second posture information acquired after the first posture information; A second captured image acquired after the first captured image and the first corrected virtual image are combined to acquire a composite image, and the composite image is displayed on a display unit.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to mixed reality presentation technology. [Background technology]

[0002] In recent years, mixed reality (MR) technology has become known as a technology that seamlessly blends the real world and the virtual world in real time. One MR technology is an MR system that uses a video see-through head-mounted display (HMD; hereinafter, referred to as "HMD" as necessary). In an MR system, an image of a subject to be observed from the pupil position of the HMD wearer is captured by an imaging unit built into the HMD, and an image in which CG (Computer Graphics) is superimposed on the captured image is presented to the HMD wearer, allowing the user to experience an MR space.

[0003] In an MR system, many processes are performed from capture to display, such as exposure and image processing of the captured image, calculations to determine the position and orientation of the HMD, CG rendering and compositing with the captured image, image processing of the displayed image, and data transmission between various components. The time required for these processes can cause a delay in the displayed image following the head movement of the HMD wearer, which can cause discomfort to the HMD wearer. For example, Patent Document 1 discloses a technology that corrects a captured image based on line-of-sight information, generates a virtual image to be composited with it, and displays the composite image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-231106 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above-mentioned conventional technology has the following problem: The configuration of Patent Document 1 corrects a captured image based on the viewer's line of sight and generates a virtual image to be combined with it, thereby reducing the delay time up to that point. However, no consideration is given to the delay caused by the processing time required to draw the virtual image itself or the delay caused by the processing time required for the virtual image to be subsequently displayed on the HMD.

[0006] The present invention has been made in consideration of the above-described circumstances, and has an object to reduce the delay time required from the acquisition of a captured image to the start of its display compared to the conventional art. [Means for solving the problem]

[0007] The present disclosure includes an image display device having an imaging means for imaging a real space and acquiring an imaged image, an attitude detection means for detecting the attitude of the imaging means and acquiring attitude information, a drawing means for drawing a first virtual image representing a virtual space viewed from a viewpoint corresponding to a position and attitude of the imaging means when the first imaged image was acquired, the position and attitude being determined from a first imaged image and first attitude information corresponding to the first imaged image, a correction means for correcting the first virtual image based on second attitude information acquired after the first attitude information to acquire a first corrected virtual image, a synthesis means for synthesizing a second imaged image acquired after the first imaged image with the first corrected virtual image to acquire a composite image, and a display means for displaying the composite image. [Effects of the Invention]

[0008] According to the configuration of the present invention, the delay time from the acquisition of a captured image to the start of display is reduced compared to the conventional method. can also be shortened. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an MR system. [Figure 2] 5A and 5B are diagrams illustrating a process of generating a composite image from a captured image and an image in a virtual space. [Figure 3] FIG. 2 is a diagram showing an example of the functional arrangement of an HMD and an image processing device according to the first embodiment. [Figure 4] FIG. 10 is a diagram illustrating a delay time in the prior art. [Figure 5] FIG. 4 is a diagram illustrating a delay time according to an embodiment of the present invention. [Figure 6] 5A to 5C are views for explaining correction processing according to the first embodiment. [Figure 7] 3 is a flowchart of processing of the MR system according to the first embodiment. [Figure 8] 10A to 10C are diagrams illustrating correction processing according to the second embodiment. [Figure 9] 5A to 5C are diagrams illustrating a correction process for a virtual image. [Figure 10] FIG. 10 is a block diagram showing an example of the functional arrangement of an HMD and an image processing device according to a third embodiment. [Figure 11] FIG. 1 is a diagram showing an example of the hardware configuration of an HMD and a computer device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In the following embodiment, an example in which an image display device according to the present invention is applied to an MR system will be described. As will be described later, the image display device (MR system) may be configured as a single unit, or may be configured as multiple units connected to each other so that they can communicate via wired or wireless communication. In the latter configuration, for example, a first unit equipped with an imaging unit, a posture sensor, a display unit, etc. is worn on the user's head, and a second unit that handles operations with a high processing load (for example, drawing a virtual image) is configured as a separate image processing device.

[0012] [First embodiment] First, an example of the configuration of an MR system according to this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the MR system according to this embodiment includes an HMD 101 (first unit), which is an example of a head-mounted display device, and an image processing device 104 (second unit). The image processing device 104 according to this embodiment includes a computer device 103 that generates an image of a mixed reality space (a space in which a real space and a virtual space are combined) to be displayed on the HMD 101, and a controller 102 that acts as an intermediary between the HMD 101 and the computer device 103.

[0013] First, the HMD 101 will be described. The HMD 101 has an imaging unit that captures images of real space, an orientation sensor that measures (measurement process) the orientation of the HMD 101 (imaging unit), and a display unit that displays an image of mixed reality space transmitted from an image processing device 104. The HMD 101 also functions as a synchronization control device for these multiple devices. The HMD 101 transmits to the controller 102 an image captured by the imaging unit and orientation information that indicates the orientation of the HMD 101 (imaging unit) measured by the sensor. The HMD 101 also receives from the controller 102 an image of mixed reality space generated by the computer device 103 based on the captured image and the orientation information, and displays the image on the display unit. As a result, an image of mixed reality space is presented in front of the eyes of a user wearing the HMD 101 on their head.

[0014] The HMD 101 may operate using power supplied from the image processing device 104 (controller 102) or from its own battery. In other words, the method of supplying power to the HMD 101 is not limited to a specific method.

[0015] 1, the HMD 101 and the image processing device 104 (controller 102) are connected by a wire. However, the connection between the HMD 101 and the image processing device 104 (controller 102) is not limited to a wired connection, and may be wireless, or a combination of wireless and wired. In other words, the connection between the HMD 101 and the image processing device 104 (controller 102) is not limited to a specific connection.

[0016] Next, the controller 102 will be described. The controller 102 performs various image processing (resolution conversion, color space conversion, distortion correction of the optical system of the imaging unit of the HMD 101, encoding, etc.) on the captured image received from the HMD 101. The controller 102 then transmits the processed captured image and the posture information received from the HMD 101 to the computer device 103. The controller 102 also performs similar image processing on the image of the mixed reality space received from the computer device 103 and transmits the image to the HMD 101.

[0017] Next, the computer device 103 will be described. The computer device 103 obtains the position and orientation of the HMD 101 (the position and orientation of the imaging unit of the HMD 101) based on the captured image and orientation information received from the controller 102, and generates an image of the virtual space seen from a viewpoint having the obtained position and orientation. The computer device 103 then generates a composite image (image of mixed reality space) of the image of the virtual space and the captured image received from the HMD 101 via the controller 102, and transmits the generated composite image to the controller 102.

[0018] Here, a process for generating a composite image from a captured image and an image of a virtual space will be described with reference to FIG. 2. A captured image 201 includes a marker 202 artificially placed in real space (for simplicity, the number of markers is one in FIG. 2, but in reality, multiple markers are included). The computer device 103 extracts the marker 202 from the captured image 201 and calculates the position and orientation of the HMD 101 based on the extracted marker 202 and orientation information received from the controller 102. The computer device 103 then generates an image 203 representing the virtual space as viewed from a viewpoint (corresponding to the viewpoint of the HMD wearer) having the calculated position and orientation. The image 203 includes a virtual object 204. The computer device 103 then generates an image 205 of a mixed reality space, which is a composite image obtained by combining the captured image 201 and the image 203 of the virtual space, and transmits the generated image 205 to the HMD 101 via the controller 102. When combining the captured image 201 and the image 203 in the virtual space, information about the depth in the three-dimensional space and information about the transparency of the virtual object 204 may be used. This makes it possible to generate a composite image 205 that takes into account the front-to-back relationship between the real object and the virtual object 204, or a composite image 205 in which the virtual object 204 is combined in a semi-transparent state.

[0019] 1, the computer device 103 and the controller 102 are shown as separate devices, but the computer device 103 and the controller 102 may be integrated. In this embodiment, a configuration in which the computer device 103 and the controller 102 are integrated will be described. In the following, the device in which the computer device 103 and the controller 102 are integrated will be referred to as an image processing device 104.

[0020] Next, an example of the functional configuration of each of the HMD 101 and the image processing device 104 will be described using the block diagram in Fig. 3. First, the HMD 101 will be described. The HMD 101 has an imaging unit 301, an orientation sensor 302, a display unit 303, a first processing unit 304, a correction unit 305, a synthesis unit 306, a second processing unit 307, and an I / F 308.

[0021] The image capturing unit 301 is an image capturing means configured to capture an image of the real space and acquire the captured image. The image capturing unit 301 of this embodiment is used to both acquire a background image to be combined with an image of the virtual space and acquire a registration image to be used to generate position and orientation information. The image capturing unit 301 is The HMD 101 has an imaging unit for one eye and an imaging unit for the right eye. The imaging unit for the left eye captures moving images in real space corresponding to the left eye of the person wearing the HMD 101, and the left eye imaging unit outputs an image of each frame of the moving images (captured image). The imaging unit for the right eye captures moving images in real space corresponding to the right eye of the person wearing the HMD 101, and the right eye imaging unit outputs an image of each frame of the moving images (captured image). In other words, the imaging unit 301 acquires captured images as stereo images having a parallax that approximately matches the parallax between the left and right eyes of the person wearing the HMD 101. Note that in an HMD for an MR system, it is preferable to arrange the imaging unit so that the central optical axis of the imaging range of the imaging unit approximately matches the line of sight of the person wearing the HMD.

[0022] The left-eye and right-eye imaging sections each include an optical system and an imaging device. Light entering from the outside world enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the incoming light as a captured image. The imaging device may be, for example, an imaging element such as a CMOS sensor or a CCD sensor.

[0023] The orientation sensor 302 is an orientation detection unit configured to detect the orientation of the image capture unit 301 and acquire orientation information. The orientation sensor 302 of this embodiment measures various data required to determine the position and orientation of the image capture unit 301 (HMD 101) and outputs the measured orientation information. The orientation sensor 302 is implemented using a magnetic sensor, an ultrasonic sensor, an acceleration sensor, an angular velocity sensor, or the like.

[0024] The display unit 303 has a display unit for the right eye and a display unit for the left eye. An image of the mixed reality space for the left eye is displayed on the display unit for the left eye, and an image of the mixed reality space for the right eye is displayed on the display unit for the right eye. Both the display unit for the left eye and the display unit for the right eye have a display optical system and a display element. The display optical system may be a decentered optical system such as a free-form prism, or may be a normal coaxial optical system or an optical system with a zoom mechanism. The display element may be, for example, a small liquid crystal display, an organic EL display, or a retinal scanning type device using MEMS. Light from the image displayed on the display element enters the eyes of the wearer of the HMD 101 via the display optical system.

[0025] The first processing unit 304 performs various types of image processing on the captured image acquired by the imaging unit 301. Here, different processing may be performed for the image processing for generating a background image used to synthesize the display image displayed on the display unit 303 and the image processing for generating a positioned image used to generate position and orientation information by the generation unit 311.

[0026] The correction unit 305 performs correction processing on the virtual image received from the image processing device 104 via the I / F 308 based on changes in the orientation information of the orientation sensor 302. The correction unit 305 detects changes in the viewpoint position and orientation of the HMD wearer based on the orientation information associated with the captured image used to generate position and orientation information for rendering the virtual image and the orientation information associated with a more recent captured image used to synthesize the display image. If the amount of change is equal to or greater than a specified value (predetermined threshold), the correction unit 305 performs processing to correct the shape, size, etc. of the virtual image so that it appears as if it were observed from the viewpoint of the HMD wearer after the change. This processing may involve shifting in the horizontal or vertical direction, changing the size by enlarging or reducing, or performing geometric transformation such as homography transformation. In other words, the correction unit 305 estimates the movement of the HMD (viewpoint) based on the orientation information of different frames and performs corrections according to the movement of the HMD on the virtual image generated from the captured image of a previous frame, thereby generating a virtual image corresponding to the current position and orientation of the HMD. The correction processing by the correction unit 305 can be performed at a frame rate higher than the frame rate of the virtual image generated by the rendering unit 313. For example, if a virtual image is generated at 60 fps and the HMD 101 supports image capture and display at 120 fps, the correction unit 305 detects changes in the viewpoint position and orientation of the HMD wearer at each arrival timing of the captured image used for synthesis, and corrects the virtual image. Even when a sufficient frame rate cannot be achieved with high-load processing such as calculation processing for image capture or image rendering processing in virtual space, it is possible to achieve a faster frame rate for the entire system, from image capture to display.

[0027] The composition unit 306 generates a display image by combining the virtual image corrected by the correction unit 305 with the captured image output from the image capture unit 301. The composition unit 306 performs processing such as chromakey composition and alpha blending, and may also perform more advanced composition processing that takes into account the anteroposterior relationship between the captured image and the virtual image by using depth information.

[0028] The second processing unit 307 performs various types of image processing on the display image generated by the combining unit 306. Examples of the image processing performed here include offset and gain adjustment processing, pixel defect correction, and distortion correction processing of the display optical system. These processes correct individual variations in the display devices and display optical systems that make up the display unit 303.

[0029] The captured image output from the imaging unit 301 and the orientation information output from the orientation sensor 302 are both transmitted to the image processing device 104 via the I / F 308 .

[0030] Next, a description will be given of the image processing device 104. The image processing device 104 includes an I / F 309, a preprocessing unit 310, a generating unit 311, a content DB 312, and a drawing unit 313.

[0031] The image processing device 104 receives the captured image and orientation information transmitted from the HMD 101 via the I / F 309. The preprocessing unit 310 performs image processing on the captured image received from the HMD 101 via the I / F 309 as preprocessing for generating position and orientation information in a generation unit 311.

[0032] The generation unit 311 extracts (recognizes) feature information from the captured image for the left eye and the captured image for the right eye, which have been subjected to image processing by the first processing unit 304 and the pre-processing unit 310. The feature information is information that can provide clues for understanding the three-dimensional structure (geometric structure) of the subject or background in the captured image, such as the position, shape, and orientation. Natural feature points or predetermined markers may be used. Alternatively, the captured image may be captured by irradiating it with visible or invisible pattern light, and the pattern in the captured image may be extracted as feature information. The generation unit 311 then determines the respective positions and orientations of the left-eye and right-eye imaging units based on the extracted feature information and the orientation information received from the HMD 101 via the I / F 309. The process for determining the positions and orientations of the imaging units based on the markers in the images and the positions and orientations measured by sensors provided in the HMD together with the imaging units that captured the images is well known, and therefore a description of this technology will be omitted.

[0033] A content DB (database) 312 stores various data (virtual space data) necessary for rendering images of a virtual space. The virtual space data includes, for example, data defining each virtual object constituting the virtual space (for example, data defining the geometric shape, color, texture, placement position and orientation, etc., of the virtual object). The virtual space data also includes, for example, data defining a light source placed in the virtual space (for example, data defining the type, position and orientation, etc., of the light source).

[0034] The rendering unit 313 constructs a virtual image using the virtual space data stored in the content DB 312. Then, the rendering unit 313 generates an image (left) of the virtual space seen from a viewpoint having the position and orientation of the imaging unit for the left eye determined by the generation unit 311. The rendering unit 313 also generates an image (right) of the virtual space seen from a viewpoint having the position and orientation of the imaging unit for the right eye determined by the generation unit 311. Then, the rendering unit 313 transmits the image (left) of the virtual space and the image (right) of the virtual space to the HMD 101 via the I / F 309.

[0035] Next, the reduction in delay time according to this embodiment will be described with reference to Fig. 4 and Fig. 5. In Fig. 4 and Fig. 5, the horizontal axis represents time. Fig. 4 is a diagram illustrating the delay time of an MR system that does not have a correction unit for correcting an image in virtual space using posture information (i.e., equivalent to the delay time in the prior art).

[0036] The imaging unit 301 sequentially acquires captured images of frame (N), frame (N+1), and frame (N+2).

[0037] The first processing unit 304 and the preprocessing unit 310 perform various image processing including preprocessing required for the generation unit 311 to calculate the position and orientation, and a delay time including the transmission delay at the I / F 308 and the I / F 309 is added to each frame.

[0038] The generation unit 311 performs processing to determine the position and orientation of the HMD 101 using the captured image and orientation information, and the drawing unit 313 draws an image in the virtual space from the determined position and orientation, and these processing times are added together as delay time.

[0039] The composition unit 306 composes the captured image and the image of the virtual space. Here, in order to match the temporal consistency between the captured image that serves as the background and the image of the virtual space, the captured image used to generate the position and orientation for rendering the image of the virtual space and the captured image used as the background of the composite image are captured images of the same frame (N).

[0040] In the second processing unit 307, various image processes are performed on the composite image generated in the composition unit 306, and the processing time is added as a delay time. Thereafter, the composite image is displayed on the display unit 303.

[0041] At this time, the time taken from the start of acquisition of the captured image of frame (N) to the start of display of the composite image of frame (N) is represented by delay time (N).

[0042] FIG. 5 is a diagram illustrating the delay time of an MR system having a correction unit for correcting an image in a virtual space using posture information.

[0043] The imaging unit 301 sequentially acquires captured images of frame (N), frame (N+1), and frame (N+2). The first processing unit 304 and pre-processing unit 310 perform various image processing including pre-processing required for the generation unit 311 to determine the position and orientation. The generation unit 311 performs processing to determine the position and orientation of the HMD 101 using the captured images and orientation information, and the drawing unit 313 draws an image in the virtual space from the determined position and orientation. The processing up to this point is the same as that shown in FIG. 4.

[0044] The correction unit 305 estimates a change in the position and orientation of the HMD 101 based on a change between past orientation information used for generating the position and orientation for drawing an image in the virtual space by the drawing unit 313 and orientation information associated with the current captured image. Then, the correction unit 305 executes a correction process (conversion process) on the image in the virtual space according to the amount of change in the position and orientation.

[0045] The composition unit 306 composes the captured image of the latest frame (N+4) with the image of the virtual space of the corrected frame (N). As described above, in the MR system according to this embodiment, the captured image used to generate the position and orientation for rendering the image of the virtual space and the captured image used as the background of the composite image are not the same image (captured image of the same frame). In other words, the image of the virtual space to be composited with the captured image of the latest frame (N+4) used as the background of the composite image is generated based on the captured image of the previous frame (N). This makes it possible to achieve the same results as in the conventional method. Compared to FIG. 4, the delay time from when the captured image of frame (N+4) is captured until when a composite image is generated using the captured image of frame (N+4) is reduced.

[0046] Furthermore, the second processing unit 307 performs various image processing on the generated composite image, and the processing time is added as a delay time. Thereafter, the composite image is displayed on the display unit 303.

[0047] In this case, the time taken from the start of capturing the captured image of frame (N+4) to the start of displaying the composite image of frame (N+4) is represented by delay time (N+4). Comparing delay time (N) in Figure 4 with delay time (N+4) in Figure 5, it can be seen that applying correction processing based on changes in posture information significantly reduces the delay time from capturing the background image to displaying the composite image.

[0048] Next, the correction process based on changes in posture information according to this embodiment will be described with reference to Fig. 6. The horizontal direction in Fig. 6 represents the passage of time. However, the numbers "1," "2," and "3" added after the image names and information names in Fig. 6 are symbols used to distinguish individual images or information, and do not indicate frame numbers.

[0049] The imaging unit 301 captures captured images 1, 2, ... at a predetermined cycle (for example, 120 fps here). The orientation sensor 302 captures orientation information 1, 2, ... corresponding to the captured images 1, 2, .... The drawing unit 313 draws virtual images 1, 2, ... at a predetermined cycle (for example, 60 fps here) based on the positions and orientations calculated using the captured images and the orientation information.

[0050] Correction unit 305 corrects virtual image 1 based on orientation information 1 associated with captured image 1 to generate corrected virtual image 1, and further corrects virtual image 1 based on orientation information 2 associated with captured image 2 to generate corrected virtual image 1'. Next, correction unit 305 corrects virtual image 2 based on orientation information 3 associated with captured image 3 to generate corrected virtual image 2, and further corrects virtual image 2 based on orientation information 4 associated with captured image 4 (not shown) to generate corrected virtual image 2'.

[0051] The composition unit 306 composes the captured image 1 and the corrected virtual image 1, the captured image 2 and the corrected virtual image 1', and the captured image 3 and the corrected virtual image 2, respectively, to generate composite image 1, composite image 2, and composite image 3, which are displayed on the display unit 303. In this way, the correction unit 305 performs frame interpolation of the virtual images in accordance with the frame rate of the captured images, so that a high-quality composite image can be displayed at a high frame rate and with low delay.

[0052] The processing flow of the MR system according to this embodiment will be described with reference to the flowchart of FIG.

[0053] In step S701, the image capturing unit 301 captures an image of the real space. In step S702, the orientation sensor 302 performs a process of associating the orientation information, which was acquired simultaneously with or closest to the timing of acquisition of the captured image by the imaging unit 301, with the captured image. Note that the process of associating the captured image with the orientation information may be performed by a control unit (not shown) rather than the orientation sensor 302. The captured image and orientation information are transmitted to the image processing device 104.

[0054] In step S703, the generation unit 311 extracts feature information (natural feature points, markers, etc.) from the captured image, and calculates the position and orientation of the image capture unit 301 based on the feature information and orientation information. In step S704, the rendering unit 313 generates a new image based on the position and orientation determined by the generation unit 311. The virtual image is constructed using the virtual space data stored in the content DB 312.

[0055] In step S705, the correction unit 305 acquires, from the orientation sensor 302, orientation information associated with the latest captured image used by the composition unit 306 to compose the display image. In step S706, it is determined whether the change between the orientation information associated with the captured image used to generate the position and orientation information for rendering the virtual image and the orientation information associated with the latest captured image used to synthesize the display image is equal to or greater than a specified value. If the result of this determination is that the change in the orientation information is equal to or greater than the specified value, the process proceeds to step S707. On the other hand, if the change in the orientation information is not equal to or greater than the specified value, the process proceeds to step S708. In step S707, the correction unit 305 performs processing to correct the shape, size, etc. of the virtual image so that it appears as if it were observed from the viewpoint of the HMD wearer after the change in posture information. In step S708, the correction unit 305 does not correct the virtual image and proceeds to step S709.

[0056] Here, the processes in steps S706, S707, and S708 determine whether or not to correct the virtual image depending on whether the change in posture information is equal to or greater than a specified value. This is because if the HMD wearer's viewpoint movement is small, the amount of deformation of the virtual image is also small, and the effect of correction is hardly noticeable. By not performing correction when the HMD wearer's viewpoint movement is small in this way, it is possible to reduce the processing load and achieve power saving effects.

[0057] However, the determination of whether or not to correct the virtual image by the correction unit 305 according to this embodiment is not limited to this. For example, a configuration is conceivable in which the virtual image is corrected when the change in posture information is equal to or less than the second specified value, and the virtual image is not corrected when the change in posture information is greater than the second specified value. This is because, when the HMD wearer moves their viewpoint extremely large, such as when they quickly turn their head, the displayed image itself cannot be correctly recognized during the movement, and therefore the effect of correction is not obtained. Furthermore, the determination of whether or not to correct the virtual image by the correction unit 305 according to this embodiment may further be based on whether or not the change in posture information is within a specified range. In this way, by not performing correction depending on the magnitude of the change in the HMD wearer's viewpoint movement, it is possible to reduce the processing load and achieve power saving effects.

[0058] In step S709, the composition unit 306 generates a composite image by combining the latest captured image with the corrected virtual image corrected by the correction unit 305 based on the change in posture information. In step S710, the second processing unit 307 performs various image processes on the composite image generated by the composition unit 306. In step S711, the display unit 303 displays the composite image that has been subjected to image processing by the second processing unit 307.

[0059] In this way, by correcting the virtual image based on the posture information associated with the captured image in the correction unit 305, the delay time from the start of capturing the captured image to the start of displaying the display image can be significantly reduced.

[0060] Furthermore, in this embodiment, the frame rate (display cycle) of the composite image displayed by the display unit 303 and the frame rate (imaging cycle) of the captured image acquired by the imaging unit 301 are higher than the frame rate (drawing cycle) of the virtual image drawn by the drawing unit 313. However, by generating a corrected virtual image corresponding to the imaging cycle or display cycle (for example, 120 fps in this case) in the correction unit 305, it is possible to generate and display a mixed reality space image at a frame rate higher than the drawing performance (drawing cycle) of the drawing unit 313.

[0061] In addition, in MR systems, the drawing cycle is shortened to further improve the image quality of virtual images. It is also possible to further reduce the frame rate and increase the time required for rendering processing per frame. In the MR system of this embodiment, even in such a case, correction processing is performed on the high-quality virtual image based on the difference with the posture information associated with the captured image, thereby making it possible to interpolate the frame rate to a period corresponding to the capture period of the captured image. That is, in the example of the correction processing described above, the frame rate is doubled by generating two corrected virtual images from one virtual image, but the frame rate can be tripled or more by generating three or more corrected virtual images from one virtual image.

[0062] In the MR system of this embodiment, periodic fluctuations that cause temporary missing frames of the virtual image may occur due to fluctuations in the processing load of the generation process of position and orientation information and the rendering process on the image processing device 104. However, even in such cases, the missing virtual image can be complemented by correcting the virtual image based on the difference from the orientation information associated with the captured image.

[0063] As described above, the correction unit 305 corrects the virtual image based on the posture information associated with the captured image, allowing the HMD wearer to have a more realistic MR experience.

[0064] [Second embodiment] In each of the following embodiments, including this embodiment, differences from the first embodiment will be described, and unless otherwise specified below, they will be considered to be the same as the first embodiment. In the first embodiment, a configuration was described in which correction processing is performed using virtual image 1 as the original image in order to generate corrected virtual image 1 and corrected virtual image 1' in correction unit 305. In this embodiment, a configuration will be described in which corrected virtual image 1 is used as the original image in order to generate corrected virtual image 1'.

[0065] The correction process based on changes in posture information according to this embodiment will be described with reference to Fig. 8. The horizontal direction in Fig. 8 represents the passage of time. However, the numbers "1," "2," and "3" added after the image names and information names in Fig. 8 are symbols used to distinguish individual images or information, and do not indicate frame numbers.

[0066] The imaging unit 301 captures captured images 1, 2, ... at a predetermined cycle (for example, 120 fps here). The orientation sensor 302 captures orientation information 1, 2, ... corresponding to the captured images 1, 2, .... The drawing unit 313 draws virtual images 1, 2, ... at a predetermined cycle (for example, 60 fps here) based on the positions and orientations calculated using the captured images and the orientation information.

[0067] The correction unit 305 corrects virtual image 1 based on orientation information 1 associated with captured image 1 to generate corrected virtual image 1. The correction unit 305 also corrects corrected virtual image 1 based on orientation information 2 associated with captured image 2 to generate corrected virtual image 1'. Next, the correction unit 305 corrects virtual image 2 based on orientation information 3 associated with captured image 3 to generate corrected virtual image 2, and further corrects corrected virtual image 2 based on orientation information 4 associated with captured image 4 (not shown) to generate corrected virtual image 2'.

[0068] The synthesis unit 306 synthesizes the captured image 1 and the corrected virtual image 1, the captured image 2 and the corrected virtual image 1', and the captured image 3 and the corrected virtual image 2, respectively, to generate composite image 1, composite image 2, and composite image 3, which are then displayed on the display unit 303.

[0069] Here, the correction process for the virtual image will be explained using FIG. 9. The upper part of FIG. 9 shows the original image before correction. The top row shows a virtual image, and the bottom row shows a corrected virtual image. In this embodiment, when correcting an image, a plane is projected onto another plane using a projective transformation, for example, a homography transformation, can be used to correct the image.

[0070] The grid points in the original virtual image and the corrected virtual image represent their respective pixel coordinates. Considering the coordinates of the corrected virtual image as the reference, pixel P0' in the corrected virtual image corresponds to pixel P0 in the original virtual image, but pixel data for the coordinates corresponding to pixel P0' does not exist in the original virtual image. Therefore, the pixels surrounding pixel P0, for example, pixels P1, P2, P3, and P4, are weighted according to the relative distance between pixel P1 to pixel P4 and pixel P0, added together, and the resulting value is divided by four to obtain the pixel data for pixel P0'. In this way, pixel data for each pixel in the corrected virtual image is obtained by calculating pixel data for all pixels using the surrounding pixel data in the original virtual image. Here, the pixel data is sent in raster scan order, i.e., from coordinates (n, m) to coordinates (n, m+5), followed by pixel data for row n+1, row n+2, and so on. Therefore, correction unit 305 cannot start processing to obtain pixel data for pixel P0' at coordinates (n+1, m+1) in the corrected virtual image until pixel data for row n+5, which includes pixels P3 and P4 in the original virtual image, is input. Memory capacity is required to hold this pixel data for the original virtual image, and the time until processing to obtain pixel data for pixel P0' in the corrected virtual image begins is a delay time. The greater the amount of deformation of the corrected virtual image, the greater the distance in the row direction between pixel P0 in the original virtual image and pixel P0' in the corrected virtual image. This requires more memory capacity and increases the processing delay time.

[0071] In this embodiment, the correction unit 305 uses the corrected virtual image 1 as the original image when correcting the virtual image based on the orientation information 2 associated with the captured image 2. Therefore, the amount of deformation between the original image and the corrected virtual image is smaller than when correction is performed using the virtual image 1 as the original image as in the first embodiment. Therefore, compared to the method of the first embodiment, the required memory capacity is reduced and the delay time related to processing is also further reduced.

[0072] In this way, by using the corrected virtual image 1 as the source image to generate the corrected virtual image 1' in the correction unit 305, it is possible to reduce the memory capacity required for the correction process and the delay time from the start of image capture to the start of display. Also, as in the first embodiment, it is possible to generate a corrected virtual image corresponding to the image capture cycle (for example, 120 fps in this case) of the captured images, and it is possible to improve the frame rate of the entire system beyond the rendering performance. Therefore, the HMD wearer can experience a more realistic MR experience.

[0073] [Third embodiment] In the first and second embodiments, the captured image acquired by the imaging unit 301 is used as a background image used to synthesize a display image displayed on the display unit 303, and also as a registration image used to generate position and orientation information by the generation unit 311. In this embodiment, a configuration will be described in which, in addition to the imaging unit 301 for background images, another imaging unit is provided for registration images.

[0074] An example of the functional configuration of each of the HMD 101 and the image processing device 104 will be described using the block diagram of Fig. 10. First, the HMD 101 will be described. The HMD 101 of this embodiment has an imaging unit 301, an orientation sensor 302, a display unit 303, a first processing unit 304, a correction unit 305, a synthesis unit 306, a second processing unit 307, an I / F 308, a second imaging unit 701, and a third processing unit 702. The same reference numerals are used to designate components corresponding to those in the first embodiment.

[0075] The imaging unit 301 is for capturing an image of real space to be combined with an image of virtual space. The imaging unit 301 has an imaging unit for the left eye and an imaging unit for the right eye, and acquires captured images as stereo images having a parallax that approximately matches the parallax between the left eye and the right eye of the wearer of the HMD 101.

[0076] The second imaging unit 701 has multiple imaging units for capturing aligned images used to generate position and orientation information, and acquires captured images as stereo images with parallax. Each imaging unit captures a moving image of the real space and outputs an image (captured image) of each frame in the moving image. Each imaging unit of the second imaging unit 701 has an optical system and an imaging device. Light entering from the outside enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the incoming light as a captured image. For example, an imaging element such as a CMOS sensor or a CCD sensor is used as the imaging device.

[0077] The imaging unit 301 and the second imaging unit 701 of the HMD 101 may use either a rolling shutter imaging element or a global shutter imaging element, taking into account various factors such as pixel count, image quality, noise, sensor size, power consumption, and cost. Alternatively, a combination of these may be used depending on the application. For example, a rolling shutter imaging element, capable of obtaining higher-quality images, may be used to capture images to be combined with virtual space images, while a global shutter imaging element, which does not cause image streaming, may be used to capture natural features and markers. Image streaming is a phenomenon that occurs due to the operating principle of the rolling shutter method, in which exposure processing is initiated sequentially for each line in the scanning direction. Specifically, this is known as a phenomenon in which a time lag occurs in the exposure timing of each line, causing the image of the subject to appear distorted and flow when the imaging unit or subject moves during the exposure time. With the global shutter method, exposure processing is performed for all lines simultaneously, eliminating time lag in the exposure timing of each line and eliminating image streaming.

[0078] In the MR system according to this embodiment, the imaging unit 301 uses a rolling shutter type imaging element as an imaging device, and the second imaging unit 701 uses a global shutter type imaging element as an imaging device.

[0079] The orientation sensor 302 measures various data necessary to determine the position and orientation of the device itself, and outputs the measured orientation information. The display unit 303 has a display unit for the right eye and a display unit for the left eye. The first processing unit 304 performs various types of image processing on the captured image acquired by the imaging unit 301. Here, image processing is performed to generate a background image used for compositing the display image displayed on the display unit 303.

[0080] The third processing unit 702 performs various types of image processing on the captured image acquired by the second imaging unit 701. Here, image processing is performed to generate a registered image used by the generation unit 311 to generate position and orientation information.

[0081] The correction unit 305 performs correction processing on the virtual image received from the image processing device 104 via the I / F 308 based on changes in the orientation information of the orientation sensor 302. The correction processing can be performed using the same method as in the MR systems of the first and second embodiments. The correction processing in the correction unit 305 can be performed at a frame rate higher than the frame rate of the image of the virtual space generated by the rendering unit 313. For example, if the virtual image is generated at 60 fps and the HMD 101 is capable of capturing and displaying at 120 fps, the correction unit 305 detects changes in the viewpoint position and orientation of the HMD wearer each time a captured image used for synthesis arrives, and corrects the virtual image. This allows for faster frame rates across the entire system, from capturing to display, even when a sufficient frame rate cannot be achieved due to high-load processing such as the calculation processing for generating position and orientation information and the rendering processing of the image of the virtual space. It is possible to achieve a frame rate.

[0082] The synthesis unit 306 synthesizes the virtual image corrected by the correction unit 305 with the captured image output from the imaging unit 301 to generate a display image. The second processing unit 307 performs various image processing on the display image generated by the synthesis unit 306 . The captured image output from the second imaging unit 701 and the orientation information output from the orientation sensor 302 are both transmitted to the image processing device 104 via the I / F 308 .

[0083] Next, a description will be given of the image processing device 104. The image processing device 104 includes an I / F 309, a preprocessing unit 310, a generating unit 311, a content DB 312, and a drawing unit 313.

[0084] The image processing device 104 receives the captured image and orientation information transmitted from the HMD 101 via the I / F 309. The pre-processing unit 310 performs image processing on the captured image obtained by the second imaging unit 701 and received from the HMD 101 via the I / F 309 as pre-processing for generating position and orientation information in the generation unit 311.

[0085] The generation unit 311 extracts (recognizes) feature information (natural feature points, markers, etc.) from the captured image for the left eye and the captured image for the right eye that have been subjected to image processing by the third processing unit 702 and the pre-processing unit 310. Then, the generation unit 311 determines the positions and orientations of the image capturing unit for the left eye and the image capturing unit for the right eye based on the extracted feature information and orientation information received from the HMD 101 via the I / F 309.

[0086] A content DB (database) 312 stores various data (virtual space data) required to render images in a virtual space.

[0087] The rendering unit 313 constructs a virtual image using the virtual space data stored in the content DB 312. Then, the rendering unit 313 generates an image (left) of the virtual space seen from a viewpoint having the position and orientation of the imaging unit for the left eye determined by the generation unit 311. The rendering unit 313 also generates an image (right) of the virtual space seen from a viewpoint having the position and orientation of the imaging unit for the right eye determined by the generation unit 311. Then, the rendering unit 313 transmits the image (left) of the virtual space and the image (right) of the virtual space to the HMD 101 via the I / F 309.

[0088] In this way, by configuring the image capture unit 301 and the second image capture unit 701 of the HMD 101 separately and selecting the optimal device for each depending on the purpose of the background image and the alignment image, it is possible to achieve high alignment accuracy while providing a high-quality background display image. This allows the HMD wearer to experience a more realistic MR experience. Furthermore, since the process of determining position and orientation using the alignment image generally requires a high computational load, a sufficient frame rate may not be achieved if the alignment image has a large number of pixels. Therefore, a device with a small number of pixels may be selected as the second image capture unit 701 for acquiring the alignment image. Furthermore, in the MR system according to this embodiment, where the rendering frame rate of the rendering unit 313 is limited to 60 fps, a frame rate of 60 fps for the alignment image used to generate position and orientation information is sufficient. In other words, by reducing the number of pixels of the second image capture unit 701 for acquiring the alignment image and setting the frame rate to 60 fps, the amount of image data transmitted from the HMD 101 to the image processing device 104 can be reduced. By reducing the amount of data transmitted, it becomes possible to use thinner transmission cables between I / F 308 and I / F 309, and it also becomes possible to support wireless communication.

[0089] [Fourth embodiment] The functional units of the HMD 101 and the image processing device 104 shown in FIGS. 3 and 10 are hardware components. It may be implemented in hardware, or some of the functional units may be implemented in software (computer program).

[0090] In the latter case, the imaging unit 301, second imaging unit 701, orientation sensor 302, display unit 303, and I / F 308 may be implemented as hardware, and the remaining functional units may be implemented as software in the HMD 101. In this case, the software is stored in a memory included in the HMD 101, and the processor included in the HMD 101 executes the software to realize the functions of the corresponding functional units.

[0091] 11A, an example of the hardware configuration of such an HMD 101 will be described. The HMD 101 has a processor 1110, a RAM 1120, a non-volatile memory 1130, an imaging unit 1140, a posture sensor 1150, a display unit 1160, and an I / F 1170 as hardware resources.

[0092] The processor 1110 executes various processes using computer programs and data stored in the RAM 1120. As a result, the processor 1110 controls the overall operation of the HMD 101 and executes or controls each of the processes described above as being performed by the HMD 101.

[0093] The RAM 1120 has an area for storing computer programs and data loaded from the nonvolatile memory 1130, and an area for storing data received from the image processing device 104 via the I / F 1170. The RAM 1120 also has a work area used by the processor 1110 when executing various processes. In this way, the RAM 1120 can provide various areas as needed.

[0094] The non-volatile memory 1130 non-temporarily stores computer programs and data for causing the processor 1110 to execute or control the above-described operations of the HMD 101. The computer programs include computer programs for causing the CPU 1101 to execute the functions of the functional units (excluding the image capture unit 301, second image capture unit 701, orientation sensor 302, display unit 303, and I / F 308) of the HMD 101 shown in Figures 3 and 10. The computer programs and data stored in the non-volatile memory 1130 are loaded into the RAM 1120 as appropriate under the control of the processor 1110, and become targets for processing by the processor 1110.

[0095] The imaging unit 1140 includes the imaging unit 301 and the second imaging unit 701 shown in FIGS. 3 and 10. The orientation sensor 1150 includes the orientation sensor 302 shown in FIGS. 3 and 10. The display unit 1160 includes the display unit 303 shown in FIGS. 3 and 10. The I / F 1170 includes the I / F 308 shown in FIGS. 3 and 10. The processor 1110, RAM 1120, nonvolatile memory 1130, imaging unit 1140, orientation sensor 1150, display unit 1160, and I / F 1170 are all connected to a bus 1180. Note that the configuration shown in FIG. 11A is an example of a configuration applicable to the HMD 101, and can be changed or modified as appropriate.

[0096] Next, a description will be given of an example configuration of the image processing device 104. The image processing device 104 can be configured by a computer device capable of executing software corresponding to the functional units (excluding the I / F 309 and the content DB 312) shown in Figures 3 and 10. An example hardware configuration of a computer device applicable to the image processing device 104 will be described using the block diagram in Figure 11B.

[0097] The CPU 1101 executes various processes using computer programs and data stored in the RAM 1102 and ROM 1103. The image processing device 104 controls the overall operation of the computer device, and also executes or controls the processes described above as being performed by the image processing device 104 to which the computer device is applied.

[0098] The RAM 1102 has an area for storing computer programs and data loaded from the ROM 1103 or the external storage device 1106, and an area for storing data received from the HMD 101 via the I / F 1107. The RAM 1102 also has a work area used by the CPU 1101 when executing various processes. In this way, the RAM 1102 can provide various areas as needed. The ROM 1103 non-temporarily stores setting data, startup programs, and the like for the computer device.

[0099] The operation unit 1104 is a user interface such as a keyboard, a mouse, or a touch panel, and allows the user to input various instructions to the CPU 1101 by operating it.

[0100] The display unit 1105 is configured with a liquid crystal screen, a touch panel screen, or the like, and can display the processing results of the CPU 1101 as images, text, etc. The display unit 1105 may also be a projection device such as a projector that projects images and text.

[0101] The external storage device 1106 is a large-capacity information storage device such as a hard disk drive or a solid-state drive. An operating system (OS) is stored in the external storage device 1106. The external storage device 1106 also non-temporarily stores computer programs and data for causing the CPU 1101 to execute the functions of the functional units (excluding the I / F 309 and content DB 312) of the image processing device 104 shown in FIGS. 3 and 10. The external storage device 1106 also stores the content DB 312.

[0102] Computer programs and data stored in the external storage device 1106 are loaded into the RAM 1102 as appropriate under the control of the CPU 1101 and are then processed by the CPU 1101 .

[0103] The I / F 1107 is a communication interface for performing data communication with the HMD 101, and functions as the I / F 309 in Figures 3 and 10. In other words, this computer device performs data communication with the HMD 101 via the I / F 1107.

[0104] The CPU 1101, RAM 1102, ROM 1103, operation unit 1104, display unit 1105, external storage device 1106, and I / F 1107 are all connected to a bus 1108. Note that the configuration shown in Fig. 11B is an example of a configuration applicable to the image processing device 104, and can be changed / modified as appropriate.

[0105] [Fifth embodiment] 3 and 10 are merely examples, and for example, the processes described above as being performed by the HMD 101 may be shared and executed by a plurality of devices, or the processes described above as being performed by the image processing device 104 may be shared and executed by a plurality of devices. In this case, the processes may be shared only by a device on the local side (edge ​​side), or the processes may be shared between a device on the local side and a device on the network (such as a cloud server).

[0106] Furthermore, instead of a head-mounted display device, a "portable device having an imaging unit, an orientation sensor, and a display unit" such as a smartphone may be used. Furthermore, such a portable device may be added to the MR system in addition to the head-mounted display device. In such a case, the image processing device 104 generates an image of the mixed reality space according to the position and orientation of the head-mounted display device, and displays the image of the head-mounted display device. The image of the mixed reality space is generated according to the position and orientation of the portable device and is distributed to the display device. The method of generating the image of the mixed reality space is the same as in the above embodiment.

[0107] Furthermore, the HMD 101 and the image processing device 104 may be integrated, or instead of a head-mounted display device, the above-mentioned portable device and the image processing device 104 may be integrated.

[0108] In addition, in the above embodiment, the posture sensor 302 has been described as being included in the HMD 101, but this is not limited thereto. For example, the necessary information may be obtained from an image captured by an objective camera installed around the wearer of the HMD 101.

[0109] Furthermore, the numerical values, calculation methods, processing execution timing, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and it is not intended that each embodiment be limited to these examples.

[0110] In addition, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments may be selectively used.

[0111] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0112] The disclosure of this specification includes the following configurations, methods, and programs.

[0113] [Configuration 1] an imaging means for capturing an image of a real space and acquiring a captured image; an attitude detection means for detecting the attitude of the imaging means and acquiring attitude information; a rendering means for rendering a first virtual image representing a virtual space viewed from a viewpoint corresponding to a position and orientation of the imaging means when the first captured image was acquired, the position and orientation being determined from a first captured image and first orientation information corresponding to the first captured image; a correcting means for correcting the first virtual image based on second orientation information acquired after the first orientation information to acquire a first corrected virtual image; a synthesis means for synthesizing a second captured image acquired after the first captured image with the first corrected virtual image to obtain a synthesized image; a display means for displaying the composite image; An image display device having the above configuration.

[0114] [Configuration 2] the correction means corrects the first virtual image based on third orientation information acquired after the second orientation information to acquire a second corrected virtual image; the combining means combines a third captured image, which is acquired after the second captured image, with the second corrected virtual image to obtain a second combined image; the display means displays the second composite image following the composite image. 2. The image display device according to claim 1.

[0115] [Configuration 3] the correction means corrects the first corrected virtual image based on third orientation information acquired after the second orientation information to acquire a second corrected virtual image; the combining means combines a third captured image, which is acquired after the second captured image, with the second corrected virtual image to obtain a second combined image; the display means displays the second composite image following the composite image. 2. The image display device according to claim 1.

[0116] [Configuration 4] a frame rate of the composite image displayed by the display means is higher than a frame rate of the virtual image drawn by the drawing means; 4. The image display device according to any one of configurations 1 to 3.

[0117] [Configuration 5] a frame rate of the captured image acquired by the imaging means is higher than a frame rate of the virtual image drawn by the drawing means; 5. The image display device according to any one of configurations 1 to 4.

[0118] [Configuration 6] the first attitude information is attitude information acquired by the attitude detection means at the same time as or the closest timing to when the imaging means acquires the first captured image, The image display device according to any one of configurations 1 to 5, wherein the second posture information is posture information acquired by the posture detection means at the same time as or closest to the time when the imaging means acquires the second captured image.

[0119] [Configuration 7] 4. The image display device according to configuration 2 or 3, wherein the third orientation information is orientation information acquired by the orientation detection means at the same time as or closest to the time when the imaging means acquires the third captured image.

[0120] [Configuration 8] The image display device includes: a first unit in which the imaging means, the attitude detection means, the correction means, the synthesis means, and the display means are provided; a second unit in which the drawing means is provided; It has The first unit and the second unit are connected to each other so as to be able to communicate with each other via wire or wirelessly. 8. The image display device according to any one of configurations 1 to 7.

[0121] [Configuration 9] The imaging means a first image capturing means for capturing captured images used to synthesize the composite image; a second image capturing means for capturing an image used to determine the position and orientation; Including, 9. The image display device according to any one of configurations 1 to 8.

[0122] [Configuration 10] a frame rate of the captured image acquired by the second imaging means is lower than a frame rate of the captured image acquired by the first imaging means; 10. The image display device according to configuration 9.

[0123] [Configuration 11] the image display device has at least the imaging means, the attitude detection means, and a head-mounted display device provided with the display means; 11. The image display device according to any one of configurations 1 to 10.

[0124] [Method 12] A control method for an image display device having an imaging unit that captures an image of a real space to acquire a captured image, an attitude sensor that detects an attitude of the imaging unit to acquire attitude information, and a display unit, comprising: acquiring, based on a first captured image and first orientation information corresponding to the first captured image, a first virtual image representing a virtual space viewed from a viewpoint corresponding to the position and orientation of the imaging unit when the first captured image was acquired; correcting the first virtual image based on second orientation information acquired after the first orientation information to acquire a first corrected virtual image; acquiring a composite image by combining a second captured image acquired after the first captured image with the first corrected virtual image; displaying the composite image on the display unit; A control method for an image display device having the above configuration.

[0125] [Program 13] An image display device having an imaging unit that captures an image of a real space and acquires a captured image, an attitude sensor that detects an attitude of the imaging unit and acquires attitude information, a display unit, and a processor, the processor comprising: acquiring, based on a first captured image and first orientation information corresponding to the first captured image, a first virtual image representing a virtual space viewed from a viewpoint corresponding to the position and orientation of the imaging unit when the first captured image was acquired; correcting the first virtual image based on second orientation information acquired after the first orientation information to acquire a first corrected virtual image; acquiring a composite image by combining a second captured image acquired after the first captured image with the first corrected virtual image; displaying the composite image on the display unit; A program to execute. [Explanation of symbols]

[0126] 101:HMD 104: Image processing device

Claims

1. an imaging means for capturing an image of a real space and acquiring a captured image; an attitude detection means for detecting the attitude of the imaging means and acquiring attitude information; a drawing means for drawing a first virtual image representing a virtual space viewed from a viewpoint corresponding to a position and orientation of the imaging means when the first captured image was acquired, the position and orientation being determined from a first captured image and first orientation information corresponding to the first captured image; a correcting means for correcting the first virtual image based on second orientation information acquired after the first orientation information to acquire a first corrected virtual image; a combining unit that combines a second captured image, which is captured after the first captured image, with the first corrected virtual image to obtain a combined image; a display means for displaying the composite image; An image display device having the above configuration.

2. the correcting means corrects the first virtual image based on third orientation information acquired after the second orientation information to acquire a second corrected virtual image; the combining means combines a third captured image, which is acquired after the second captured image, with the second corrected virtual image to obtain a second combined image; the display means displays the second composite image following the composite image; The image display device according to claim 1 .

3. the correcting means corrects the first corrected virtual image based on third orientation information acquired after the second orientation information to acquire a second corrected virtual image; the combining means combines a third captured image, which is acquired after the second captured image, with the second corrected virtual image to obtain a second combined image; the display means displays the second composite image following the composite image; The image display device according to claim 1 .

4. a frame rate of the composite image displayed by the display means is higher than a frame rate of the virtual image drawn by the drawing means; The image display device according to claim 1 .

5. a frame rate of the captured image acquired by the imaging means is higher than a frame rate of the virtual image drawn by the drawing means; The image display device according to claim 1 .

6. the first attitude information is attitude information acquired by the attitude detection means at the same time as or closest to the time when the imaging means acquires the first captured image, 2. The image display device according to claim 1, wherein the second orientation information is orientation information acquired by the orientation detection means at the same time as or closest to the time when the imaging means acquires the second captured image.

7. 4. The image display device according to claim 2, wherein the third orientation information is orientation information acquired by the orientation detection means at the same time as or closest to the time when the imaging means acquires the third captured image.

8. The image display device includes: a first unit in which the imaging means, the attitude detection means, the correction means, the synthesis means, and the display means are provided; a second unit in which the drawing means is provided; It has The first unit and the second unit are connected to each other so as to be able to communicate with each other via wire or wirelessly. The image display device according to claim 1 .

9. The imaging means a first image capturing means for capturing captured images used to synthesize the composite image; a second image capturing means for capturing an image used to determine the position and orientation; Including, The image display device according to claim 1 .

10. a frame rate of the captured image acquired by the second imaging means is lower than a frame rate of the captured image acquired by the first imaging means; The image display device according to claim 9 .

11. the image display device has at least the imaging means, the attitude detection means, and a head-mounted display device provided with the display means; The image display device according to claim 1 .

12. A control method for an image display device having an imaging unit that captures an image of a real space to acquire a captured image, an attitude sensor that detects an attitude of the imaging unit to acquire attitude information, and a display unit, comprising: acquiring, based on a first captured image and first orientation information corresponding to the first captured image, a first virtual image representing a virtual space viewed from a viewpoint corresponding to the position and orientation of the imaging unit when the first captured image was acquired; correcting the first virtual image based on second orientation information acquired after the first orientation information to acquire a first corrected virtual image; acquiring a composite image by combining a second captured image acquired after the first captured image with the first corrected virtual image; displaying the composite image on the display unit; A control method for an image display device having the above configuration.

13. An image display device having an imaging unit that captures an image of a real space and acquires a captured image, an attitude sensor that detects an attitude of the imaging unit and acquires attitude information, a display unit, and a processor, the processor comprising: acquiring, based on a first captured image and first orientation information corresponding to the first captured image, a first virtual image representing a virtual space viewed from a viewpoint corresponding to the position and orientation of the imaging unit when the first captured image was acquired; correcting the first virtual image based on second orientation information acquired after the first orientation information to acquire a first corrected virtual image; acquiring a composite image by combining a second captured image acquired after the first captured image with the first corrected virtual image; displaying the composite image on the display unit; A program to execute.

Images