Head-mounted display device, control method and program for head-mounted display device

Abstract

To operate a first imaging portion, a second imaging portion, and a display portion in synchronization with each other.SOLUTION: A head-mounted display device includes a display portion, a first imaging portion, a second imaging portion, and a generation portion that generates a first signal for controlling the start of exposure of the first imaging portion and a second signal for controlling the start of exposure of the second imaging portion on the basis of a signal indicating the timing of image input to the display portion.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present disclosure relates to a head-mounted display device, a system, and a method for controlling a head-mounted display device. [Background technology]

[0002] In recent years, mixed reality (MR) technology has become known as a technology for seamlessly fusing the real world and the virtual world in real time. One MR technology is an MR system that uses a video see-through HMD (Head Mounted Display: hereinafter, referred to as "HMD" as necessary). In the MR system, an image of a subject that approximately matches the subject observed from the pupil position of the HMD wearer is captured by an imaging unit built into the HMD, and an image in which CG (Computer Graphics) is superimposed on the captured image is presented to the HMD wearer. This allows the user to experience an MR space.

[0003] In an MR system, the position and orientation of the HMD are calculated by performing arithmetic processing using captured images and various sensor information, and it is desirable to operate the imaging units and various sensors in a synchronized state as much as possible. For example, the following Patent Document 1 discloses a technique for synchronizing multiple imaging units by supplying common drive signals and synchronization signals to the multiple imaging units. In addition, the following Patent Document 2 discloses a technique for synchronizing multiple imaging units with different exposure times by aligning the centers of gravity of the exposure times. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2000-341719 A [Patent Document 2] JP 2006-005608 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technologies of Patent Documents 1 and 2 have the following problems. The configurations of Patent Documents 1 and 2 are limited to synchronization between imaging units. In a system that handles various sensors in addition to an imaging unit, such as an MR system, if the imaging unit and various sensors operate asynchronously, sufficient calculation accuracy may not be obtained, and in that case, positional deviation may occur between the captured image and CG. In addition, if the display unit and the imaging unit operate asynchronously, the delay time from imaging to display may vary for each frame, or frames may overlap or be missing at long-second intervals, which may cause an HMD wearer to feel uncomfortable.

[0006] An object of the present disclosure is to enable a first imaging unit, a second imaging unit, and a display unit to operate in synchronization with each other. [Means for solving the problem]

[0007] The head-mounted display device has a display unit, a first imaging unit, a second imaging unit, and a generation unit that generates a first signal for controlling the start of exposure of the first imaging unit and a second signal for controlling the start of exposure of the second imaging unit based on a signal indicating the timing of image input to the display unit. Effect of the Invention

[0008] According to the present disclosure, the first imaging section, the second imaging section, and the display section can be operated in synchronization. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an MR system. [Diagram 2] FIG. 11 is a diagram illustrating a process for generating a composite image. [Diagram 3] 5A to 5C are diagrams illustrating operation timings of the imaging unit and the attitude sensor. [Figure 4] FIG. 1 is a diagram illustrating delay fluctuations due to asynchronous operation. [Diagram 5] FIG. 2 is a block diagram showing an example of a functional configuration. [Figure 6] FIG. 2 illustrates the generation of an external synchronization input. [Figure 7] FIG. 2 illustrates the generation of an external synchronization input. [Figure 8] 10A and 10B are diagrams illustrating synchronous operations of an imaging unit and an attitude sensor. [Figure 9] FIG. 2 is a diagram illustrating a generation process for generating a synchronization signal. [Figure 10] FIG. 2 is a diagram illustrating a generation process for generating a synchronization signal. [Figure 11] FIG. 2 is a diagram illustrating a generation process for generating a synchronization signal. [Figure 12] 13 is a flowchart of a synchronous operation control. [Figure 13] FIG. 2 is a diagram illustrating an example of a functional configuration. [Figure 14] FIG. 11 is a diagram illustrating an example of setting a synchronization reference timing. [Figure 15] 13 is a flowchart of a synchronous operation control. [Figure 16] FIG. 2 is a block diagram showing an example of a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the following embodiments do not limit the scope of the claims. Although the embodiments describe a number of features, not all of these features are essential, and the features may be combined in any manner. Furthermore, in the drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] (First embodiment) FIG. 1 is a diagram showing an example of the configuration of an MR (mixed reality) system 100 according to a first embodiment. The MR system 100 includes an HMD 101 and an image processing device 104. The HMD 101 is an example of a head-mounted display device. The image processing device 104 includes a computer device 103 and a controller 102. The computer device 103 generates an image of a mixed reality space (a space in which a real space and a virtual space are combined) to be displayed on the HMD 101. The controller 102 mediates between the HMD 101 and the computer device 103.

[0012] First, the HMD 101 will be described. As shown in FIG. 16(a), the HMD 101 includes an imaging unit 1640 that captures an image of a real space, an orientation sensor 1660 that measures (measures) the position and orientation of the HMD 101, and a display unit 1670 that displays an image of a mixed reality space transmitted from the image processing device 104. The HMD 101 also functions as a synchronization control device for these multiple devices. The HMD 101 transmits to the controller 102 an image captured by the imaging unit 1640 and the position and orientation of the HMD 101 measured by the orientation sensor 1660. The HMD 101 also receives an image of a mixed reality space generated by the computer device 103 based on the captured image and the position and orientation from the controller 102 and displays the image on the display unit 1670. As a result, an image of a mixed reality space is presented in front of the eyes of a user wearing the HMD 101 on his head.

[0013] The HMD 101 may operate on a power supply voltage supplied from the controller 102, or may operate on a power supply voltage supplied from a battery included in the HMD 101. In other words, the method of supplying the power supply voltage to the HMD 101 is not limited to a specific method.

[0014] 1, the HMD 101 and the controller 102 are connected by a wire. However, the connection between the HMD 101 and the controller 102 is not limited to a wired connection, and may be wireless or a combination of wireless and wired. In other words, the connection between the HMD 101 and the controller 102 is not limited to a specific connection.

[0015] Next, the controller 102 will be described. The controller 102 performs various image processing (resolution conversion, color space conversion, distortion correction of the optical system of the imaging unit 1640 of the HMD 101, encoding, etc.) on the captured image transmitted from the HMD 101. Then, the controller 102 transmits the captured image that has undergone image processing and the position and orientation transmitted from the HMD 101 to the computer device 103. The controller 102 also performs similar image processing on the image of the mixed reality space transmitted from the computer device 103 and transmits it to the HMD 101.

[0016] Next, the computer device 103 will be described. The computer device 103 obtains the position and orientation of the HMD 101 (the position and orientation of the imaging unit 1640 of the HMD 101) based on the captured image and the position and orientation received from the controller 102, and generates an image of the virtual space seen from a viewpoint having the obtained position and orientation. The computer device 103 then generates a composite image (image of mixed reality space) of the image of the virtual space and the captured image transmitted from the HMD 101 via the controller 102, and transmits the generated composite image to the controller 102.

[0017] FIG. 2 is a diagram showing a process of generating a composite image 205 from a captured image 201 and an image 203 of a virtual space. The captured image 201 includes a marker 202 artificially arranged in a real space (for the sake of simplicity, the number of markers is set to one in FIG. 2, but in reality, a plurality of markers are included). The computer device 103 extracts the marker 202 from the captured image 201, and obtains the position and orientation of the HMD 101 based on the extracted marker 202 and the position and orientation received from the controller 102. Then, the computer device 103 generates an image 203 of the virtual space seen from a viewpoint having the obtained position and orientation. The image 203 includes a virtual object 204. Then, the computer device 103 generates an image 205 of a mixed reality space, which is a composite image obtained by combining the captured image 201 and the image 203 of the virtual space, and transmits the generated image 205 to the HMD 101. When combining the captured image 201 and the image 203 in the virtual space, information regarding the depth in the three-dimensional space and information regarding the transparency of the virtual object can be used to generate a composite image that takes into account the front-to-back relationship between the real object and the virtual object, or a composite image in which the virtual object is combined in a semi-transparent state.

[0018] 1, the computer device 103 and the controller 102 are separate devices, but the computer device 103 and the controller 102 may be integrated. In this embodiment, a configuration in which the computer device 103 and the controller 102 are integrated will be described. In the following, the device in which the computer device 103 and the controller 102 are integrated will be referred to as an image processing device 104.

[0019] The HMD 101 has an imaging unit 1640. In consideration of various factors such as the number of pixels, image quality, noise, sensor size, power consumption, and cost, the imaging unit 1640 can use a rolling shutter type imaging element and a global shutter type imaging element separately, or can use them in combination depending on the purpose. For example, a configuration can be used in which a rolling shutter type imaging element capable of acquiring a higher quality image is used for capturing a captured image 201 to be combined with an image 203 of a virtual space, and a global shutter type imaging element without image streaming is used for capturing an image of a marker 202. Image streaming is a phenomenon that occurs due to the operating principle of the rolling shutter type in which exposure processing is started sequentially for each line in the scanning direction.

[0020] Specifically, as shown in Fig. 3, this is known as a phenomenon in which a time lag occurs in the exposure timing of each line, and when the imaging unit 1640 or the subject moves during the exposure time, the subject is recorded in a flowing, deformed manner. In Fig. 3, the horizontal axis indicates time, and "exposure time (rolling)" indicates the exposure time of each line captured by an imaging element using a rolling shutter method. "Exposure time (global)" indicates the exposure time of each line captured by an imaging element using a global shutter method. In the case of the global shutter method, exposure processing is performed on all lines simultaneously, so there is no time lag in the exposure timing of each line and no image flow occurs.

[0021] As shown in FIG. 3, when the imaging unit 1640 receives an external synchronization input (external synchronization input signal), a processing time t Exp_Ready After that, exposure of each pixel starts, and the exposure time t Exp Each pixel is exposed for t Img_Out After that, the image capturing unit 1640 starts outputting a synchronization signal (synchronization output, synchronization output signal) and an image signal (image output). In the case of a global shutter type image sensor, the time t Sync_In From exposure start time t Exp_Start Time to exposure center time t Exp_Center , exposure end time tExp_End is uniquely determined as a peculiar value for each image sensor. On the other hand, in the case of a rolling shutter type image sensor, the difference Δt between the exposure start timing of the first line and the last line is determined depending on which line exposure time is used as the reference. Exp_Start The exposure start time, exposure center time, and exposure end time can take different values ​​within the range corresponding to the above.

[0022] Similarly, when the orientation sensor 1660 that measures the position and orientation of the HMD 101 receives an external synchronization input, the processing time until the start of measurement is t Sens_Ready Then, the measurement of the position and orientation (data acquisition) starts. Then, at a time t Sens_Out After that, output of the position and orientation (data output) starts.

[0023] 5 shows an example of the functional configuration of the HMD 101. The HMD 101 has a first imaging unit 501, a second imaging unit 502, an attitude sensor 503, and a display unit 504. The first imaging unit 501 corresponds to the imaging element that captures images by the rolling shutter method in the imaging unit 1640 in FIG. 16(a). The second imaging unit 502 corresponds to the imaging element that captures images by the global shutter method in the imaging unit 1640 in FIG. 16(a). The attitude sensor 503 corresponds to the attitude sensor 1660 in FIG. 16(a). The display unit 504 corresponds to the display unit 1670 in FIG. 16(a).

[0024] 4 is a diagram for explaining synchronous operation of the first imaging unit 501, the second imaging unit 502, and the attitude sensor 503 by an external synchronization input, and delay variation due to asynchronous operation of the display unit 504. In FIG. 4, synchronization between the devices is achieved by providing a common external synchronization input to the first imaging unit 501 of the rolling shutter type, the second imaging unit 502 of the global shutter type, and the attitude sensor 503. As described in FIG. 3, the first imaging unit 501, the second imaging unit 502, and the attitude sensor 503 each have their own unique processing time. Therefore, the time t Sync_In(N)With respect to the synchronization signal (N) given to the image sensor 503, there is a difference between the time when the first image capture unit 501 and the second image capture unit 502 actually perform exposure and the time when the orientation sensor 503 performs measurement. For example, a case will be described where the central exposure time of the first image capture unit 501 and the second image capture unit 502 is considered as the reference. In that case, the measurement time of the position and orientation by the orientation sensor 503 is t Sens_Meas(N) The central exposure time of the first image capture unit 501 is t Exp_Center_CAM1(N) The central exposure time of the second image capture unit 502 is t Exp_Center_CAM2(N) These times do not match.

[0025] Here, a configuration will be described in which the first imaging unit 501 is used to acquire the captured image 201 to be composited with the image 203 of the virtual space, and the second imaging unit 502 is used to capture the marker 202. The image 203 of the virtual space is generated based on the captured image 201 of the marker 202 acquired by the second imaging unit 502 and the position and orientation acquired by the orientation sensor 503, so that an error due to a difference in acquisition timing between the two occurs. Since the acquisition timing of the captured image acquired by the first imaging unit 501 is also different from the acquisition timing of the captured image by the second imaging unit 502 and the acquisition timing of the position and orientation by the orientation sensor 503, the influence on the position accuracy of the image of the virtual space superimposed on the captured image becomes even greater. For this reason, in the MR system 100, it is preferable to perform synchronization operations of the exposure timing of the first imaging unit 501, the exposure timing of the second imaging unit 502, and the acquisition timing of the position and orientation by the orientation sensor 503 so that they coincide with each other with higher accuracy.

[0026] Time t Proc_Start(N) is the time when the first captured image (N) acquired by the first imaging unit 501, the second captured image (N) acquired by the second imaging unit 502, and the orientation data (N) acquired by the orientation sensor 503 are all aligned based on the synchronization signal (N). Proc_Start(N)At this time, processing (N) is started in the processing unit. The processing here refers to captured image processing, position and orientation information generation, image synthesis, and other processing in the MR system 100. The synthetic image (N) generated in processing (N) is displayed on the display unit 504, which operates asynchronously with the first imaging unit 501, the second imaging unit 502, and the orientation sensor 503, at a time t Disp(N) The image is displayed based on the synchronization signal input to the

[0027] The time t of the exposure center when the first captured image (N) is acquired by the first image capturing unit 501 Exp_Center_CAM1(N) The time t at which the composite image (N) is displayed on the display unit 504 Disp(N) The delay time (N) in the MR system 100 is defined as the delay time (N). Similarly, the exposure center time t Exp_Center_CAM1(N-1) From the time t Disp(N-1) The delay time (N-1) is set to the delay time (N). In this case, the delay time (N-1) is longer than the delay time (N). In this way, when the frame rate of the imaging units 501 and 502 and the frame rate of the display unit 504 are asynchronous, the delay time varies for each frame.

[0028] Furthermore, at time t Disp(N+1) Since the processing in the processing unit (N+1) has not been completed at this time, the composite image (N) that has already been prepared is displayed again. The composite image (N+1) is actually displayed at a time t Disp(N+2) In this case, the delay time (N+1) is greater than the delay time (N) and the delay time (N-1).

[0029] In this way, when the frame rate of the display unit 504 is faster than the frame rate of the first imaging unit 501, the generation of the composite image does not keep up with the display update timing, and the same composite image is displayed, which gives the user an uncomfortable feeling in the smoothness of the moving image. On the other hand, when the frame rate of the display unit 504 is slower than the frame rate of the first imaging unit 501, two composite images are generated between the display update timings, and one composite image is skipped without being displayed, which also gives the user an uncomfortable feeling.

[0030] 5 is a block diagram showing an example of the functional configuration of the HMD 101 and the image processing device 104. First, the HMD 101 will be described. The HMD 101 has a first imaging unit 501, a second imaging unit 502, a posture sensor 503, a display unit 504, a detection unit 505, a control unit 506, a generation unit 507, a setting unit 508, and an I / F 509.

[0031] The first imaging unit 501 is for capturing an image of the real space to be synthesized with an image of the virtual space, and has an imaging unit for the left eye and an imaging unit for the right eye. The imaging unit for the left eye captures a moving image of the real space corresponding to the left eye of the person wearing the HMD 101, and the imaging unit for the left eye outputs an image (captured image) of each frame in the moving image. The imaging unit for the right eye captures a moving image of the real space corresponding to the right eye of the person wearing the HMD 101, and the imaging unit for the right eye outputs an image (captured image) of each frame in the moving image. In other words, the first imaging unit 501 obtains a captured image as a stereo image having a parallax that approximately matches the positions of the left eye and right eye of the person wearing the HMD 101. Note that, in the HMD 101 for the MR system 100, it is preferable to arrange the first imaging unit 501 so that the central optical axis of the imaging range of the imaging unit 501 approximately matches the line of sight of the person wearing the HMD 101.

[0032] Each of the left-eye imaging section and the right-eye imaging section has an optical system and an imaging device. Light entering from the outside world enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the entering light as a captured image. A rolling shutter type imaging element is used as the imaging device of the first imaging section 501. The first imaging section 501 periodically outputs a captured image, and also outputs a synchronization signal indicating the output start timing (image output timing) of the captured image.

[0033] The second imaging unit 502 has a plurality of imaging units for capturing images of the real space used for alignment, and obtains captured images as stereo images having parallax. Each imaging unit captures a moving image of the real space, and outputs an image (captured image) of each frame in the moving image. Each imaging unit of the second imaging unit 502 has an optical system and an imaging device. Light entering from the outside world enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the light entering as a captured image. A global shutter type imaging element is used as the imaging device of the imaging unit 502. The plurality of imaging units of the imaging unit 502 start exposure every time they receive a synchronization signal from the generating unit 507, and end exposure when the exposure time for one frame has elapsed.

[0034] The orientation sensor 503 functions as a detection unit, detects (measures) the position and orientation of the HMD 101, and outputs the detected position and orientation every time a synchronization signal is received from the generation unit 507. The orientation sensor 503 is implemented by a magnetic sensor, an ultrasonic sensor, an acceleration sensor, an angular velocity sensor, or the like.

[0035] The display unit 504 has a display unit for the right eye and a display unit for the left eye. The image of the mixed reality space for the left eye received from the image processing device 104 via the I / F 509 is displayed on the display unit for the left eye, and the image of the mixed reality space for the right eye received from the image processing device 104 via the I / F 509 is displayed on the display unit for the right eye. Both the display unit for the left eye and the display unit for the right eye have a display optical system and a display element. The display optical system may be a decentered optical system such as a free-form prism, or may be a normal coaxial optical system or an optical system having a zoom mechanism. For example, a small liquid crystal display, an organic EL display, or a retina scan type device using MEMS is used as the display element. Light from an image displayed on the display element enters the eye of the wearer of the HMD 101 through the display optical system. The display unit 504 displays a composite image of the mixed reality space received from the image processing device 104 for each frame based on a synchronization signal of the composite image received from the image processing device 104. The synchronization signal of the composite image is a signal indicating the timing of image input to the display unit 504.

[0036] The detection unit 505 detects a synchronization signal (a signal indicating the start timing of image display on the display unit 504) of the composite image of the mixed reality space received from the image processing device 104 via the I / F 509, and upon detecting the synchronization signal, notifies the generation unit 507 of the detection.

[0037] The control unit 506 controls the generation of a synchronization signal by the generation unit 507 in accordance with changes in processing times required for various processes received from the image processing device 104 via the I / F 509, the exposure times of the first imaging unit 501 and the second imaging unit 502 set by the setting unit 508, and a synchronization reference timing setting. Specifically, the control unit 506 first determines a synchronization reference line for synchronizing the exposure time of the second imaging unit 502 and the orientation sensor 503 with that of a line from among the lines of the first imaging unit 501 exposed at different timings. Next, the control unit 506 determines a synchronization reference timing that serves as a reference for synchronizing the exposure timing of the first imaging unit 501, the exposure timing of the second imaging unit 502, and the acquisition timing of the attitude sensor 503, depending on the exposure time setting of the first imaging unit 501, the exposure time setting of the second imaging unit 502, and the synchronization reference timing setting of which timing during the exposure time of the synchronization reference line to synchronize (start of exposure, center of exposure, end of exposure, etc.), and sets this in the generation unit 507.

[0038] Upon receiving a notification from the detection unit 505, the generation unit 507 generates a synchronization signal to be supplied to the first imaging unit 501, the second imaging unit 502, and the orientation sensor 503 based on the synchronization signal detected by the detection unit 505 and the synchronization reference timing determined by the control unit 506. Then, the generation unit 507 supplies the generated synchronization signal to the first imaging unit 501, the second imaging unit 502, and the orientation sensor 503. The setting unit 508 sets various parameters used in the operation of the HMD 101.

[0039] The first captured image output from the first imaging unit 501, the second captured image output from the second imaging unit 502, and the position and orientation output from the orientation sensor 503 are all transmitted to the image processing device 104 via the I / F 509.

[0040] Next, the image processing device 104 will be described. The image processing device 104 includes an I / F 510, a processing unit 511, a generating unit 512, a content DB 513, a synthesis unit 514, a processing unit 515, a comparison unit 516, and a setting unit 517. The image processing device 104 receives the captured image and the position and orientation transmitted from the HMD 101 via the I / F 510. The processing unit 511 performs various image processes on the captured image received from the HMD 101 via the I / F 510.

[0041] The generation unit 512 extracts (recognizes) the marker 202 from the captured image for the left eye and the captured image for the right eye that have been subjected to image processing by the processing unit 511. The generation unit 512 then obtains the respective positions and orientations of the imaging unit for the left eye and the imaging unit for the right eye based on the marker 202 and the positions and orientations received from the HMD 101 via the I / F 510, and outputs a synchronization signal for the captured images. The process for obtaining the positions and orientations of the imaging units based on the marker 202 in an image and the positions and orientations measured by the orientation sensor 503 provided in the HMD 101 together with the imaging unit that captured the image is well known, and therefore a description of this technology will be omitted.

[0042] A content DB (database) 513 stores various data (virtual space data) necessary for rendering images of a virtual space. The virtual space data includes, for example, data defining each virtual object constituting the virtual space (for example, data defining the geometric shape, color, texture, placement position and orientation, etc., of the virtual object). The virtual space data also includes, for example, data defining a light source placed in the virtual space (for example, data defining the type, position and orientation, etc., of the light source).

[0043] The synthesis unit 514 constructs a virtual space using the virtual space data stored in the content DB 513. Then, the synthesis unit 514 generates an image L of the virtual space seen from a viewpoint having the position and orientation of the imaging unit for the left eye obtained by the generation unit 512. Also, the synthesis unit 514 generates an image R of the virtual space seen from a viewpoint having the position and orientation of the imaging unit for the right eye obtained by the generation unit 512. Then, the synthesis unit 514 generates a synthesized image L by synthesizing the image L of the virtual space and the image captured by the imaging unit for the left eye as an image L of the mixed reality space for the left eye. Also, the synthesis unit 514 generates a synthesized image R by synthesizing the image R of the virtual space and the image captured by the imaging unit for the right eye as an image R of the mixed reality space for the right eye. Then, the synthesis unit 514 outputs a synchronization signal for the synthesized image.

[0044] The processing unit 515 performs various types of image processing on each of the mixed reality space image L and the mixed reality space image R generated by the composition unit 514. The processing unit 515 functions as a transmission unit and transmits the mixed reality space image L and the mixed reality space image R that have been subjected to the image processing, and a synchronization signal for the composite image of the composition unit 514, to the HMD 101 via the I / F 510.

[0045] The comparison unit 516 compares the synchronization signal of the captured image output from the generation unit 512 with the synchronization signal of the composite image output from the synthesis unit 514, and detects a change in the processing time taken from when the captured image is acquired by the imaging unit 501 to when the position and orientation information is generated by the generation unit 512. Then, the comparison unit 516 transmits the detected change in the processing time to the HMD 101 via the I / F 510.

[0046] The setting unit 517 sets various parameters used in the operation of the image processing device 104 .

[0047] Fig. 6 is a diagram for explaining an example of generation of a synchronization signal supplied by the generation unit 507 to the first imaging unit 501 and the second imaging unit 502. In Fig. 6, the horizontal axis represents time.

[0048] When the detection unit 505 detects a synchronization signal for a composite image of the mixed reality space received from the image processing device 104 via the I / F 509, it notifies the generation unit 507 of the detection. Upon receiving the notification, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of exposure of the first imaging unit 501" after an offset of 601 from the detection timing of the synchronization signal, based on the synchronization reference timing set by the control unit 506. Here, a case will be described in which the synchronization reference timing is set to the center of the exposure period of the first imaging unit 501 and the second imaging unit 502. Here, the time from the detection timing of the synchronization signal to the "synchronization reference timing in the exposure time of the frame next to the detection timing of the first imaging unit 501" is defined as t 61 The "processing time until the start of exposure" specific to the first imaging unit 501 is set as t 62 The exposure time for one frame of the first image capture unit 501 is t 63 The exposure time for one line of the first image capture unit 501 is t 64 In addition, the total number of imaging lines in the first imaging unit 501 is L CAM1_TOTAL , sync reference line L 61 Then, the offset 601 at this time can be calculated according to the following formula:

[0049] Offset 601=t 61 -t 62 -t 64 / 2-(t 63 -t 64 )×L 61 / L CAM1_TOTAL

[0050] Similarly, when the generating unit 507 receives a notification indicating that the detecting unit 505 has detected a synchronization signal for a composite image in a mixed reality space, the generating unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of exposure of the second imaging unit 502" after an offset 602 from the detection timing of the synchronization signal based on the synchronization reference timing set by the control unit 506. Here, the "processing time until the start of exposure" specific to the second imaging unit 502 is defined as t 65 The exposure time for one frame of the second image capture unit 502 is t 66 In this case, the offset 602 can be calculated according to the following formula:

[0051] Offset 602=t 61 -t 65 -t 66 / 2

[0052] The control unit 506 may periodically or irregularly obtain and set the offsets 601 and 602. The generation unit 507 supplies the thus generated “synchronization signal for controlling the start of exposure of the first imaging unit 501” to the first imaging unit 501, and supplies the thus generated “synchronization signal for controlling the start of exposure of the second imaging unit 502” to the second imaging unit 502.

[0053] The first imaging unit 501 starts exposure when it receives the "synchronization signal for controlling the start of exposure of the first imaging unit 501" generated in this way. The second imaging unit 502 starts exposure when it receives the "synchronization signal for controlling the start of exposure of the second imaging unit 502" generated in this way. Since the center time of the exposure time of the first imaging unit 501 and the center time of the exposure time of the second imaging unit 502 match the synchronization reference timing, as a result, the first imaging unit 501 and the second imaging unit 502 are exposed in synchronization with each other. In other words, the generation unit 507 generates a synchronization signal to be supplied to the first imaging unit 501 and the second imaging unit 502 so that the center time of the exposure time of the first imaging unit 501 and the center time of the exposure time of the second imaging unit 502 match each other.

[0054] In the above description, the synchronization reference timing in the synchronous operation is set to the center of the exposure period of the first imaging unit 501 and the second imaging unit 502, but this is not limiting. The setting of the synchronization reference timing in this embodiment is not limited to this, and can be any timing during the exposure time.

[0055] Fig. 7 is a diagram for explaining the synchronous operation of the display unit 504, the imaging unit 501, the imaging unit 502, and the orientation sensor 503 according to this embodiment. In Fig. 7, the horizontal axis represents time. Here, a generation process is described for generating a synchronization signal to be supplied to the second imaging unit 502 and the orientation sensor 503 in the first imaging unit 501, taking into consideration the difference in exposure time for each line of an image captured by an image sensor of a rolling shutter system.

[0056] When the detection unit 505 detects a synchronization signal (synchronization input) of a composite image received from the image processing device 104 via the I / F 509, it notifies the generation unit 507 of that fact. Upon receiving the notification, the generation unit 507 generates a synchronization signal to be supplied to the first imaging unit 501 and the second imaging unit 502 and a synchronization signal to be supplied to the orientation sensor 503 in accordance with the synchronization reference timing set by the control unit 506.

[0057] Here, it is assumed that the exposure time of the first imaging unit 501 and the exposure time of the second imaging unit 502 are equal, and the control unit 506 or the setting unit 508 has set the "synchronization reference timing to be the exposure start time of the first line of the image captured by the first imaging unit 501." In this case, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of exposure of the first imaging unit 501" after an offset 701 from the detection timing of the synchronization signal. The offset 701 is calculated by subtracting the time t 62 It can be obtained by subtracting

[0058] Upon receiving the thus generated "synchronization signal for controlling the start of exposure of the first imaging unit 501," the first imaging unit 501 starts exposure, and outputs data of each line captured by the exposure in accordance with its own synchronization output. The generating unit 507 also generates a "synchronization signal (external synchronization input) for controlling the start of exposure of the second imaging unit 502" after an offset 704 from the detection timing of the synchronization signal. The offset 704 is calculated by multiplying the time from the detection timing of the synchronization signal to the "exposure start time of the first line of the captured image by the second imaging unit 502" by a time t 65 It can be obtained by subtracting

[0059] Upon receiving the thus generated "synchronization signal for controlling the start of exposure of the second imaging unit 502," the second imaging unit 502 starts exposure, and outputs data of each line captured by the exposure in accordance with its own synchronization output. Furthermore, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of measurement (acquisition timing) of the position and orientation by the orientation sensor 503" after an offset 707 from the detection timing of the synchronization signal. The offset 707 is calculated by subtracting the processing time t Sens_Ready Upon receiving the thus generated "synchronization signal for controlling the start of measurement of the position and orientation by the orientation sensor 503," the orientation sensor 503 starts measuring the position and orientation (acquiring data), and outputs the acquired position and orientation.

[0060] Also, assume that the exposure time of the first imaging unit 501 and the exposure time of the second imaging unit 502 are equal, and the control unit 506 or the setting unit 508 has set the "synchronization reference timing to be the exposure center time of the central line of the image captured by the first imaging unit 501." In this case, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of exposure of the first imaging unit 501" after an offset 702 from the detection timing of the synchronization signal. The offset 702 is calculated by subtracting the time t72 The generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of exposure of the second imaging unit 502" after an offset 705 from the detection timing of the synchronization signal. The offset 705 is calculated by subtracting the time t 65 The generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of measurement of the position and orientation by the orientation sensor 503" after an offset 708 from the detection timing of the synchronization signal. The offset 708 is calculated by subtracting the processing time t Sens_Ready It can be obtained by subtracting

[0061] Also, assume that the exposure time of the first imaging unit 501 and the exposure time of the second imaging unit 502 are equal, and the control unit 506 or the setting unit 508 has set the "synchronization reference timing to be the exposure end time of the final line of the image captured by the first imaging unit 501." In this case, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of exposure of the first imaging unit 501" after an offset 703 from the detection timing of the synchronization signal. The offset 703 is calculated by subtracting the time t 62 The generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of exposure of the second imaging unit 502" after an offset 706 from the detection timing of the synchronization signal. The offset 706 is calculated by subtracting the time t 65The generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of measurement of the position and orientation by the orientation sensor 503" after an offset 709 from the detection timing of the synchronization signal. The offset 709 is calculated by subtracting the processing time t Sens_Ready It can be obtained by subtracting

[0062] In this way, the control unit 506 sets the generation unit 507 so that the synchronization signals that the generation unit 507 supplies to the first imaging unit 501, the second imaging unit 502, and the orientation sensor 503 are generated based on the synchronization reference timing set by the setting unit 508.

[0063] In addition, offsets for the first imaging unit 501, the second imaging unit 502, and the attitude sensor 503 can be adjusted based on various settings made by the control unit 506 or the setting unit 508 at any timing, such as the exposure start time of the last line or the exposure end time of the first line. This makes it possible to perform synchronous operations.

[0064] FIG. 8 is a diagram showing a generation process for generating synchronization signals to be supplied to the first imaging unit 501, the second imaging unit 502, and the orientation sensor 503, taking into consideration the processing times for captured image processing, position and orientation information generation, image synthesis, etc., in order to further reduce the delay time from imaging to display.

[0065] The detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the Nth frame (frame (N)) from the image processing device 104 via the I / F 509. Then, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (for capturing frame (N)) of the first imaging unit 501" after an offset 801 and supplies the signal to the first imaging unit 501. Also, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image (frame (N)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (for capturing frame (N)) of the second imaging unit 502" after an offset 803 and supplies the signal to the second imaging unit 502. The offsets 801 and 803 can be calculated by the calculation method described with reference to FIG. 7 (offset calculation method in the case where the "synchronization reference timing is the exposure center time of the center line of the captured image by the first imaging unit 501"). Furthermore, when the detection unit 505 detects a synchronization signal of a composite image corresponding to a captured image (frame (N)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of measurement of position and orientation by the orientation sensor 503 (for frame (N))" after an offset 807, and supplies the signal to the orientation sensor 503. The offset 807 can be calculated by the calculation method described with reference to Fig. 7 (an offset calculation method in the case where the "synchronization reference timing is the exposure center time of the central line of the image captured by the first imaging unit 501").

[0066] Assume that the setting unit 508 performs a setting (setting change) to change the exposure time of the second imaging unit 502 between the time when the detection unit 505 detects a synchronization signal of a composite image corresponding to a captured image (frame (N)) and the time when the detection unit 505 detects a synchronization signal of a composite image corresponding to frame (N+1).

[0067] At this time, the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+1)th frame (frame (N+1)) from the image processing device 104 via the I / F 509. Then, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of exposure of the second imaging unit 502 (for capturing frame (N+1))" after the offset 804 and supplies it to the second imaging unit 502. The offset 804 can be obtained by a method similar to that for the offset 803.

[0068] No switching of the offset occurs when no setting change is made by the setting unit 508. The control unit 506 sets a synchronization reference timing in the generation unit 507 so that the synchronization signals to be supplied to the first imaging unit 501, the second imaging unit 502, and the attitude sensor 503 are generated based on an offset according to a setting change made by the setting unit 508 to the exposure time of the imaging unit.

[0069] At this time, the delay time from when a captured image corresponding to frame (N) is acquired by the first imaging unit 501, after processing (N) is performed by various processing units, until a composite image using frame (N) is displayed on the display unit 504 is represented by delay time (N). Similarly, the delay time from when a captured image corresponding to frame (N+1) is acquired by the first imaging unit 501, after processing (N+1) is performed by various processing units, until a composite image using frame (N+1) is displayed on the display unit 504 is represented by delay time (N+1). Here, the time t 81 , and the time t 82 Although the composite image is ready, it is waiting to be displayed, which causes excessive delay. Therefore, at time t 81 and t 82 By setting the offset time so that the delay time in the MR system 100 is smaller, it is possible to reduce the delay time in the MR system 100.

[0070] Specifically, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the Nth frame (frame (N)) from the image processing device 104 via the I / F 509, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (for capturing frame (N)) of the first imaging unit 501" after an offset 802 and supplies it to the first imaging unit 501. Also, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the Nth frame (frame (N)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (for capturing frame (N)) of the second imaging unit 502" after an offset 805 and supplies it to the second imaging unit 502. Furthermore, when the detection unit 505 detects a synchronization signal for a composite image corresponding to the Nth frame captured image (frame (N)), the generation unit 507 generates a “synchronization signal (external synchronization input) for controlling the start of position and orientation measurement by the orientation sensor 503 (for frame (N))” after an offset 808, and supplies the signal to the orientation sensor 503.

[0071] Here, it is assumed that the setting unit 508 performs a setting (setting change) to change the exposure time of the second imaging unit 502 between the time when the detection unit 505 detects a synchronization signal corresponding to frame (N) and the time when the detection unit 505 detects a synchronization signal corresponding to frame (N+1).

[0072] At this time, the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+1)th frame (frame (N+1)) from the image processing device 104 via the I / F 509. Then, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of exposure of the second imaging unit 502 (for capturing frame (N+1))" after the offset 806, and supplies the signal to the second imaging unit 502.

[0073] No switching of the offset occurs when no setting change is made by the setting unit 508. The control unit 506 sets a synchronization reference timing in the generation unit 507 so that the synchronization signals to be supplied to the first imaging unit 501, the second imaging unit 502, and the attitude sensor 503 are generated based on an offset according to a setting change made by the setting unit 508 to the exposure time of the imaging unit.

[0074] At this time, the delay time from when a captured image corresponding to frame (N) is acquired by the first imaging unit 501, after processing (N)' is performed in various processing units, until a composite image using frame (N) is displayed on the display unit 504 is represented by delay time (N)'. Similarly, the delay time from when a captured image corresponding to frame (N+1) is acquired by the first imaging unit 501, after processing (N+1)' is performed in various processing units, until a composite image using frame (N+1) is displayed on the display unit 504 is represented by delay time (N+1)'. The time t 81 ', and the time t 82 ',t 81 and t 82 , and the delay time of the MR system 100 is reduced.

[0075] Also, t 81 and 82 , t 81 ' and t 82 ', the time from the completion of processing (N) and processing (N+1) to display has changed due to the setting unit 508 making a setting (setting change) to change the exposure time of the second imaging unit 502. Therefore, when adjusting the offset to reduce delay, it is necessary to adjust the offset amount according to the setting of the exposure time of the first imaging unit 501 and the second imaging unit 502, the setting of the synchronization reference timing by the setting unit 508, and a combination of various processing times. Here, the comparison unit 516 of the image processing device 104 compares the synchronization signal of the captured image after various processing with the synchronization signal of the composite image, detects a fluctuation in the processing time from a change in the time difference, and feeds back the result to the control unit 506 of the HMD 101, thereby making it possible to adjust the offset.

[0076] By performing the above processing, it is possible to realize synchronous operation of the first imaging unit 501, the second imaging unit 502, and the orientation sensor 503 with respect to the display unit 504, thereby making it possible to reduce the delay time from imaging to display in the MR system 100.

[0077] FIG. 9 shows a generation process for generating synchronization signals to be supplied to the first imaging unit 501, the second imaging unit 502, and the orientation sensor 503, using the central exposure time of the central line in an image captured by a rolling shutter type image sensor as the synchronization reference timing.

[0078] The detection unit 505 detects a synchronization signal for a composite image corresponding to the captured image of the Nth frame (frame (N)) from the image processing device 104 via the I / F 509. Then, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (frame (for N imaging)) of the first imaging unit 501" after an offset 901 and supplies it to the first imaging unit 501. Also, when the detection unit 505 detects a synchronization signal for a composite image corresponding to the captured image of the Nth frame (frame (N)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (frame (for N imaging)) of the second imaging unit 502" after an offset 903 and supplies it to the second imaging unit 502. The offsets 901 and 903 are calculated using the calculation method described with reference to FIG. 7 (when the "synchronization reference timing is the exposure center time of the central line of the captured image by the first imaging unit 501" 7) and the method for reducing the delay time described with reference to Fig. 8. Also, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the Nth frame (frame (N)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of measurement of the position and orientation by the orientation sensor 503 (for frame N)" after the offset 905 and supplies it to the orientation sensor 503. The offset 905 can be obtained by the calculation method described with reference to Fig. 7 (offset calculation method when the "synchronization reference timing is the exposure center time of the central line of the image captured by the first imaging unit 501") and the method for reducing the delay time described with reference to Fig. 8.

[0079] Here, it is assumed that the setting unit 508 performs a setting (second setting change) to change the exposure time of the second imaging unit 502 between the time when the detection unit 505 detects a synchronization signal for a composite image corresponding to frame (N) and the time when the detection unit 505 detects a synchronization signal for a composite image corresponding to frame (N+1).

[0080] At this time, the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+1)th frame (frame (N+1)) from the image processing device 104 via the I / F 509. Then, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (for capturing frame (N+1)) of the first imaging unit 501" after the offset 901 and supplies it to the first imaging unit 501. Also, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+1)th frame (frame (N+1)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (for capturing frame (N+1)) of the second imaging unit 502" after the offset 904 and supplies it to the second imaging unit 502. The offset 904 can be obtained by a method similar to that of the offset 903. Furthermore, when the detection unit 505 detects a synchronization signal for a composite image corresponding to the (N+1)th frame captured image (frame (N+1)), the generation unit 507 generates a “synchronization signal (external synchronization input) for controlling the start of position and orientation measurement by the orientation sensor 503 (for frame (N+1))” after the offset 905, and supplies the signal to the orientation sensor 503.

[0081] Also, assume that the setting unit 508 performs a setting (first setting change) to change the exposure time of the first imaging unit 501 between the time when the detection unit 505 detects a synchronization signal for a composite image corresponding to frame (N+1) and the time when the detection unit 505 detects a synchronization signal for a frame (N+2).

[0082] At this time, the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+2)th frame (frame (N+2)) from the image processing device 104 via the I / F 509. Then, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (for capturing frame (N+2)) of the first imaging unit 501" after an offset 902 and supplies the signal to the first imaging unit 501. The offset 902 can be obtained by a method similar to that for the offset 901. In addition, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+2)th frame (frame (N+2)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (for capturing frame (N+2)) of the second imaging unit 502" after an offset 904 and supplies the signal to the second imaging unit 502. Furthermore, when the detection unit 505 detects a synchronization signal for a composite image corresponding to the (N+2)th frame captured image (frame (N+2)), the generation unit 507 generates a “synchronization signal (external synchronization input) for controlling the start of position and orientation measurement by the orientation sensor 503 (for frame (N+2))” after the offset 905, and supplies the signal to the orientation sensor 503.

[0083] Thereafter, no switching of the offset occurs because no setting change is made by the setting unit 508. The control unit 506 sets the synchronization reference timing in the generation unit 507 so that the synchronization signals to be supplied to the first imaging unit 501, the second imaging unit 502, and the attitude sensor 503 are generated based on the offset according to the setting change of the exposure time of the imaging unit by the setting unit 508.

[0084] By carrying out the above processing, even when the settings are changed, it is possible to match the center time of the exposure time of the center line of the first imaging unit 501, the center time of the exposure time of the second imaging unit 502, and the timing of acquiring the position and orientation data of the orientation sensor 503. Furthermore, it is possible to realize a synchronous operation with the display unit 504 while further reducing the delay time from imaging to display.

[0085] FIG. 10 shows a generation process for generating a synchronization signal to be supplied to the second imaging unit 502 and the orientation sensor 503, using the exposure start time of the first line in an image captured by a rolling shutter type image sensor as the synchronization reference timing.

[0086] The detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the Nth frame (frame (N)) from the image processing device 104 via the I / F 509. Then, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (for capturing frame N) of the first imaging unit 501" after an offset 1001 and supplies the signal to the first imaging unit 501. Also, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the Nth frame (frame (N)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (for capturing frame (N)) of the second imaging unit 502" after an offset 1002 and supplies the signal to the second imaging unit 502. The offsets 1001 and 1002 are calculated by the calculation method described using FIG. 7 (offset calculation method when the "synchronization reference timing is the exposure start time of the first line of the captured image by the first imaging unit 501") and the method for reducing the delay time described using FIG. 8. Furthermore, when the detection unit 505 detects a synchronization signal for a composite image corresponding to the captured image of the Nth frame (frame (N)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of measurement of the position and orientation by the orientation sensor 503 (for frame N)" after the offset 1003 and supplies it to the orientation sensor 503. The offset 1003 can be calculated by the calculation method described in Fig. 7 (offset calculation method when the "synchronization reference timing is the exposure start time of the first line of the captured image by the first imaging unit 501") and the method for reducing the delay time described in Fig. 8.

[0087] Here, it is assumed that the setting unit 508 performs a setting (second setting change) to change the exposure time of the second imaging unit 502 between the time when the detection unit 505 detects a synchronization signal for a composite image corresponding to frame (N) and the time when the detection unit 505 detects a synchronization signal for a composite image corresponding to frame (N+1).

[0088] At this time, the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+1)th frame (frame (N+1)) from the image processing device 104 via the I / F 509. Then, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (for capturing frame (N+1)) of the first imaging unit 501" after the offset 1001 and supplies it to the first imaging unit 501. Even if the second setting change is made, the relationship between the exposure start time of the first imaging unit 501 and the processing time from the external synchronization input to the exposure start time of the second imaging unit 502 does not change, so there is no need to change the offset. Therefore, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+1)th frame (frame (N+1)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of exposure of the second imaging unit 502 (for capturing frame (N+1))" after an offset of 1002 and supplies it to the second imaging unit 502. Even if the second setting change is performed, the relationship between the exposure start time of the first imaging unit 501 and the processing time from the external synchronization input of the orientation sensor 503 to the start of measurement of the position and orientation does not change, so there is no need to change the offset. Therefore, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+1)th frame (frame (N+1)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of measurement of the position and orientation by the orientation sensor 503 (for frame (N+1))" after an offset of 1003 and supplies it to the orientation sensor 503.

[0089] Also, assume that the setting unit 508 performs a setting (first setting change) to change the exposure time of the first imaging unit 501 between the time when the detection unit 505 detects a synchronization signal for a composite image corresponding to frame (N) and the time when the detection unit 505 detects a synchronization signal for a composite image corresponding to frame (N+2).

[0090] At this time, even if the first setting change is performed, the exposure start time of the first imaging unit 501 does not change, so there is no need to change the offset. Therefore, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+2)th frame (frame (N+2)) from the image processing device 104 via the I / F 509, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start of the first imaging unit 501 (for capturing frame (N+2))" after the offset 1001 and supplies it to the first imaging unit 501. Also, even if the first setting change is performed, the relationship between the exposure start time of the first imaging unit 501 and the processing time from the external synchronization input to the exposure start time of the second imaging unit 502 does not change, so there is no need to change the offset. Therefore, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+2)th frame (frame (N+2)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of exposure of the second imaging unit 502 (for capturing frame (N+2))" after an offset of 1002 and supplies it to the second imaging unit 502. Even if the first setting change is performed, the relationship between the exposure start time of the first imaging unit 501 and the processing time from the external synchronization input of the orientation sensor 503 to the start of measurement of the position and orientation does not change, so there is no need to change the offset. Therefore, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+2)th frame (frame (N+2)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of measurement of the position and orientation by the orientation sensor 503 (for frame (N+2))" after an offset of 1003 and supplies it to the orientation sensor 503.

[0091] Thereafter, no setting change is made by the setting unit 508. In this manner, the control unit 506 sets the synchronization reference timing in the generation unit 507 so that the synchronization signals to be supplied to the first imaging unit 501, the second imaging unit 502, and the attitude sensor 503 are generated based on the offset according to the setting change of the exposure time of the imaging units by the setting unit 508.

[0092] By carrying out the above processing, even when the settings are changed, it is possible to match the exposure start time of the first line of an image captured by the first imaging unit 501, the exposure start time of the first line of an image captured by the second imaging unit 502, and the data acquisition timing of the attitude sensor 503. Furthermore, it is possible to realize a synchronous operation with the display unit 504 with a smaller delay time from imaging to display.

[0093] Also, it should be noted that when the synchronization reference timing is set to the exposure start time, even if the settings of the first imaging unit 501 and the second imaging unit 502 are changed, there is no need to change the offset for generating the external synchronization input to the first imaging unit 501, the second imaging unit 502, and the orientation sensor 503. This is not limited to only the case of the first line, but also applies when the synchronization reference timing is set to the exposure start time of any line. By utilizing this, it is possible to simplify the offset change process when the settings are changed, and reduce the processing load and circuit size of the HMD 101.

[0094] FIG. 11 shows a generation process for generating synchronization signals to be supplied to a first imaging unit 501, a second imaging unit 502, and an orientation sensor 503, using the exposure end time of a center line in an image captured by a rolling shutter type image sensor as the synchronization reference timing.

[0095] The detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the Nth frame (frame (N)) from the image processing device 104 via the I / F 509. Then, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (frame (for N imaging)) of the first imaging unit 501" after an offset 1101 and supplies it to the first imaging unit 501. Also, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the Nth frame (frame (N)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (frame (for N imaging)) of the second imaging unit 502" after an offset 1103 and supplies it to the second imaging unit 502. The offsets 1101 and 1103 are calculated as "the exposure end time of the final line and the exposure end time of the central line" from the offset calculation method when the "synchronization reference timing is the exposure end time of the final line of the captured image by the first imaging unit 501" described with reference to FIG. 8. When the detection unit 505 detects a synchronization signal for a composite image corresponding to the Nth frame captured image (frame (N)), the generation unit 507 generates a synchronization signal (external synchronization input) for controlling the start of position and orientation measurement by the orientation sensor 503 (for frame N) after the offset 1105 and supplies the synchronization signal to the orientation sensor 503. The offset 1105 can be obtained by subtracting the difference between the exposure end time of the final line and the exposure end time of the central line from the offset calculation method in the case where the synchronization reference timing is set to the exposure end time of the final line of the captured image by the first imaging unit 501 as the synchronization reference timing described with reference to FIG. 7, and by the method for reducing the delay time described with reference to FIG. 8.

[0096] Here, it is assumed that the setting unit 508 performs a setting (second setting change) to change the exposure time of the second imaging unit 502 between the time when the detection unit 505 detects a synchronization signal for a composite image corresponding to frame (N) and the time when the detection unit 505 detects a synchronization signal for a composite image corresponding to frame (N+1).

[0097] At this time, the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+1)th frame (frame (N+1)) from the image processing device 104 via the I / F 509. Then, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (for capturing frame (N+1)) of the first imaging unit 501" after an offset 1101 and supplies the signal to the first imaging unit 501. Also, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+1)th frame (frame (N+1)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (for capturing frame (N+1)) of the second imaging unit 502" after an offset 1104 and supplies the signal to the second imaging unit 502. The offset 1104 can be obtained in the same manner as the offset 1103. Furthermore, when the detection unit 505 detects a synchronization signal for a composite image corresponding to the (N+1)th frame captured image (frame (N+1)), the generation unit 507 generates a “synchronization signal (external synchronization input) for controlling the start of position and orientation measurement by the orientation sensor 503 (for frame (N+1))” after the offset 1105, and supplies the signal to the orientation sensor 503.

[0098] Also, assume that the setting unit 508 performs a setting (first setting change) to change the exposure time of the first imaging unit 501 between the time when the detection unit 505 detects a synchronization signal for a composite image corresponding to frame (N+1) and the time when the detection unit 505 detects a synchronization signal for a composite image corresponding to frame (N+2).

[0099] At this time, the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+2)th frame (frame (N+2)) from the image processing device 104 via the I / F 509. Then, the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the exposure start (for capturing frame (N+2)) of the first imaging unit 501" after the offset 1102 and supplies it to the first imaging unit 501. The offset 1102 can be obtained in the same manner as the offset 1101. Even if the first setting change is made, the relationship between the exposure end time of the first imaging unit 501 and the processing time from the external synchronization input to the exposure end time of the second imaging unit 502 does not change, so there is no need to change the offset. Therefore, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+2)th frame (frame (N+2)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of exposure of the second imaging unit 502 (for capturing frame (N+2))" after an offset 1104 and supplies it to the second imaging unit 502. Even if the first setting change is performed, the relationship between the end of exposure of the first imaging unit 501 and the processing time from the external synchronization input of the orientation sensor 503 to the start of measurement of the position and orientation does not change, so there is no need to change the offset. Therefore, when the detection unit 505 detects a synchronization signal of a composite image corresponding to the captured image of the (N+2)th frame (frame (N+2)), the generation unit 507 generates a "synchronization signal (external synchronization input) for controlling the start of measurement of the position and orientation by the orientation sensor 503 (for frame (N+2))" after an offset 1105 and supplies it to the orientation sensor 503.

[0100] Thereafter, no switching of the offset occurs because no setting change is made by the setting unit 508. The control unit 506 sets the synchronization reference timing in the generation unit 507 so that the synchronization signals to be supplied to the first imaging unit 501, the second imaging unit 502, and the attitude sensor 503 are generated based on the offset according to the setting change of the exposure time of the imaging unit by the setting unit 508.

[0101] By carrying out the above processing, even when the settings are changed, it is possible to match the exposure end time of the center line of the image captured by the first imaging unit 501, the exposure end time of the center line of the image captured by the second imaging unit 502, and the data acquisition timing of the attitude sensor 503. Furthermore, it is possible to realize a synchronous operation with the display unit 504 with a smaller delay time from imaging to display.

[0102] It should be noted here that when the synchronization timing is set to the exposure end time, even if the setting change is made to the first imaging unit 501 and the second imaging unit 502, the period of the synchronous output and image output of the first imaging unit 501 and the second imaging unit 502 and the data output of the attitude sensor 503 are constant. For example, in FIG. 9 and FIG. 10, when the first setting change is made to the first imaging unit 501 and the second setting change is made to the second imaging unit 502, the period of the synchronous output and image output of the first imaging unit 501 and the second imaging unit 502 fluctuates. If the output of the captured image is not of a constant period, it is necessary to select an IC (Integrated Circuit) corresponding to such an output period as the captured image processing unit in the subsequent stage. In addition, it is also possible to widen the range of options for the captured image processing unit in the subsequent stage by adding a separate frame memory or the like and controlling the read operation from the frame memory to adjust the period of the captured image output from the first imaging unit 501 and the second imaging unit 502 to a constant period. However, the processing load and the circuit scale will increase. This fluctuation in the data output period will also occur when the synchronization reference line is changed in Fig. 11. In this case, for example, by restricting the setting change of the synchronization reference line while the MR system 100 is operating, it is possible to suppress the increase in the processing load and the circuit scale required to deal with the period fluctuation.

[0103] 12 is a flowchart showing synchronous operation control of the first imaging unit 501, the second imaging unit 502, and the orientation sensor 503 by the HMD 101 according to this embodiment. A control method of the HMD 101 will be described below.

[0104] In step S1201, the control unit 506 determines whether or not the setting unit 508 has changed the setting of the synchronization reference timing, the first imaging unit 501 or the second imaging unit 502, or the synchronization timing in the exposure time, or whether or not the comparison unit 516 has detected a change in processing time for various image processes. If the result of this determination shows that there has been a change in setting or a change in processing time as described above, the process proceeds to step S1202. On the other hand, if there has been no change in setting or a change in processing time as described above, the process proceeds to step S1207.

[0105] In step S1202, the control unit 506 acquires from the setting unit 508 the settings of the first imaging unit 501 (including at least parameters related to the first imaging unit 501 necessary to obtain an offset).

[0106] In step S1203, the control unit 506 acquires from the setting unit 508 the settings of the second imaging unit 502 (including at least parameters related to the second imaging unit 502 necessary to obtain an offset).

[0107] In step S1204, the control unit 506 acquires the amount of variation in the processing time of each frame detected by the comparison unit 516. The comparison unit 516 compares, for each frame, the synchronization signal of the captured image output from the generation unit 512 with the synchronization signal of the composite image output from the synthesis unit 514, and detects the processing time of each frame.

[0108] In step S1205, the control unit 506 acquires from the setting unit 508 a setting as to which time during the exposure period of which line of the image captured by the first imaging unit 501 is to be set as the reference synchronization timing.

[0109] In step S1206, the control unit 506 performs the above-mentioned processing based on the information acquired in steps S1202 to S1205, thereby obtaining offsets corresponding to the first imaging unit 501, the second imaging unit 502, and the orientation sensor 503.

[0110] In step S1207, the generation unit 507 determines whether or not the detection unit 505 has detected a synchronization signal (synchronization input) of the composite image received from the image processing device 104 via the I / F 509. If the result of this determination is that the detection unit 505 has detected a synchronization signal (synchronization input), the process proceeds to step S1208. On the other hand, if the detection unit 505 has not detected a synchronization signal (synchronization output), the process proceeds to step S1201.

[0111] In step S1208, the generating unit 507 receives a notification from the detecting unit 505 that a synchronization signal (synchronization input) of a composite image has been received. Then, the generating unit 507 generates a "synchronization signal to be supplied to the first imaging unit 501" after an offset corresponding to the first imaging unit 501 from the detection timing of the synchronization signal, and generates a "synchronization signal to be supplied to the second imaging unit 502" after an offset corresponding to the second imaging unit 502. Then, the generating unit 507 supplies the generated synchronization signals to the first imaging unit 501 and the second imaging unit 502, respectively. Furthermore, the generating unit 507 generates a "synchronization signal to be supplied to the orientation sensor 503" after an offset corresponding to the orientation sensor 503 from the detection timing of the synchronization signal, and supplies the generated synchronization signal to the orientation sensor 503.

[0112] As described above, when the exposure time of the first imaging unit 501 or the exposure time of the second imaging unit 502 is changed, the generation unit 507 changes the generation timing of the synchronization signal of the first imaging unit 501 and the synchronization signal of the second imaging unit 502 so that a first time within the exposure time of the first imaging unit 501 matches a second time within the exposure time of the second imaging unit 502.

[0113] 9, a first time within the exposure time of the first imaging unit 501 is the center time of the exposure time of a specific line of the first imaging unit 501. A second time within the exposure time of the second imaging unit 502 is the center time of the exposure time of the second imaging unit 502.

[0114] 10, a first time within the exposure time of the first imaging unit 501 is an exposure start time of a specific line of the first imaging unit 501. A second time within the exposure time of the second imaging unit 502 is an exposure start time of the second imaging unit 502.

[0115] 11, a first time within the exposure time of the first imaging unit 501 is an exposure end time of a specific line of the first imaging unit 501. A second time within the exposure time of the second imaging unit 502 is an exposure end time of the second imaging unit 502.

[0116] In step S1204, if the processing time of the images captured by the first imaging unit 501 and the second imaging unit 502 varies between frames, the generation unit 507 changes the generation timing of the synchronization signal of the first imaging unit 501 or the synchronization signal of the second imaging unit 502.

[0117] In this manner, in this embodiment, the display unit 504, the first imaging unit 501, the second imaging unit 502, and the attitude sensor 503 are operated synchronously based on a synchronization signal of a composite image supplied to the display unit 504, and the delay time from imaging to display can be further reduced. Furthermore, it is also possible to accommodate changes in the setting of the exposure time of the first imaging unit 501 or the second imaging unit 502 and changes in the setting of the synchronization reference timing. By matching the imaging and data acquisition timing of each device to any timing during the exposure time, a more realistic MR experience without positional deviation between the captured image and the image in the virtual space is realized.

[0118] Second embodiment In the following embodiments including the second embodiment, differences from the first embodiment will be described, and unless otherwise noted below, they will be the same as the first embodiment. In the first embodiment, a configuration was described in which a synchronization signal is generated for each device based on the detection timing of the synchronization signal of the composite image in the detection unit 505, using the settings of the first imaging unit 501 and the second imaging unit 502 set by the setting unit 508, setting information of the synchronization reference timing, and fluctuation information of the image processing time detected by the comparison unit 516. In the second embodiment, a configuration will be described in which a more accurate synchronization operation between devices is realized by further using an area of ​​interest in an image of the wearer of the HMD 101.

[0119] Fig. 13 is a block diagram showing an example of the functional configuration of the HMD 101 and the image processing device 104 according to the second embodiment. In Fig. 13, a gaze sensor 1301 is added to Fig. 5. The HMD 101 further includes the gaze sensor 1301.

[0120] The gaze sensor 1301 detects the gaze of the wearer of the HMD 101 using a known method such as a corneal reflex method using an infrared camera to identify an area of ​​interest of the wearer of the HMD 101 within the imaging range of the first imaging unit 501 or within the display range of the display unit 504. The gaze sensor 1301 functions as an identifying unit and identifies the area of ​​interest in the imaging range of the first imaging unit 501 by gaze detection. Specifically, the gaze sensor 1301 functions as a detecting unit and detects the gaze position of the wearer of the HMD 101 within the imaging range of the first imaging unit 501 and identifies the gaze position as the area of ​​interest.

[0121] The control unit 506 controls the synchronization reference timing determined based on the line-of-sight information detected by the line-of-sight sensor 1301, changes in processing time required for various processes received from the image processing device 104 via the I / F 509, the exposure time of the first imaging unit 501 and the second imaging unit 502 set by the setting unit 508, and generation of a synchronization signal by the generation unit 507. Specifically, the control unit 506 first determines a synchronization reference line, based on the region of interest of the wearer of the HMD 101 identified by the line-of-sight sensor 1301, from among the lines of the first imaging unit 501 exposed at different timings, with which the exposure time of the second imaging unit 502 and the orientation sensor 503 should be synchronized. Next, the control unit 506 determines a synchronization reference timing that serves as a reference for synchronizing the exposure timing of the first imaging unit 501 and the second imaging unit 502 and the acquisition timing of the attitude sensor 503, depending on the changes in processing time required for various processes received from the image processing device 104 via the I / F 509, the exposure time setting of the first imaging unit 501, the exposure time setting of the second imaging unit 502, and the synchronization timing setting of which timing during the exposure time of the synchronization reference line (start of exposure, center of exposure, end of exposure, etc.) to synchronize, and sets it in the generation unit 507.

[0122] Upon receiving the notification from the detection unit 505, the generation unit 507 generates a synchronization signal to be supplied to the first imaging unit 501, the second imaging unit 502, and the orientation sensor 503, based on the synchronization signal of the composite image detected by the detection unit 505 and the synchronization reference timing determined by the control unit 506. Then, the generation unit 507 supplies the generated synchronization signal to the first imaging unit 501, the second imaging unit 502, and the orientation sensor 503.

[0123] FIG. 14 is a diagram for explaining an example of setting the synchronization reference timing in the control unit 506. In the mixed reality image 205 observed by the wearer of the HMD 101, a gaze point 1401, which is an area of ​​interest on which the wearer of the HMD 101 is gazing, is obtained based on gaze information detected by the gaze sensor 1301. The control unit 506 determines a synchronization reference timing 1403 from a synchronization reference line 1402 on which the gaze point 1401 is exposed during the entire exposure time of the first imaging unit 501 and a synchronization timing setting (for example, exposure center here) during the exposure time of the synchronization reference line 1402. Then, the control unit 506 sets the synchronization reference timing 1403 in the generation unit 507. Here, the synchronization timing setting may be set in advance in the control unit 506, or may be arbitrarily set from the outside via the setting unit 508, and may be configured to be changeable. The gaze point 1401 is an area of ​​interest in the imaging range of the first imaging unit 501. A synchronization reference line 1402 is set based on a gaze point 1401 .

[0124] 15 is a flowchart showing synchronous operation control of the display unit 504, the first imaging unit 501, the second imaging unit 502, and the attitude sensor 503 by the HMD 101 according to the second embodiment. A control method of the HMD 101 will be described below.

[0125] In step S1501, the gaze sensor 1301 detects gaze information of the wearer of the HMD 101.

[0126] In step S1502, the control unit 506 determines, based on the line-of-sight information detected by the line-of-sight sensor 1301, whether or not the line of sight of the wearer of the HMD 101 has moved since the first imaging unit 501 last acquired an image.

[0127] If it is determined that the line of sight of the person wearing the HMD 101 has moved, the process proceeds to step S1503. On the other hand, if the line of sight of the person wearing the HMD 101 has not moved, the process proceeds to step S1504.

[0128] In step S1503 , the control unit 506 sets the synchronization reference line 1402 of the first imaging unit 501 in the generation unit 507 based on the line-of-sight information detected by the line-of-sight sensor 1301 .

[0129] In step S1504, the control unit 506 determines whether or not there has been a change in settings of the first imaging unit 501 or the second imaging unit 502, or a change in settings that changes the synchronization timing in the exposure time, or whether or not there has been detection of a change in processing time for various image processing operations in the comparison unit 516.

[0130] As a result of this determination, if the setting change as described above has been made by setting unit 508, the process proceeds to step S1505. On the other hand, if the setting change as described above has not been made by setting unit 508, the process proceeds to step S1509.

[0131] In step S1505, the control unit 506 acquires from the setting unit 508 the settings of the first imaging unit 501 (including at least parameters related to the first imaging unit 501 necessary to obtain an offset).

[0132] In step S1506, the control unit 506 acquires from the setting unit 508 the settings of the second imaging unit 502 (including at least parameters related to the second imaging unit 502 necessary to obtain an offset).

[0133] In step S1507, the control unit 506 acquires the amount of variation in the processing time for each frame detected in the comparison unit 516.

[0134] In step S1508, the control unit 506 acquires from the setting unit 508 the synchronization timing setting (eg, exposure center) during the exposure time of the synchronization reference line 1402.

[0135] In step S1509, the control unit 506 determines the synchronization reference timing 1403 within the exposure time of the synchronization reference line 1402 of the first imaging unit 501 based on the information acquired in steps S1503 to S1508, and sets it in the generation unit 507.

[0136] In step S1510, the generation unit 507 performs the above processing to obtain offsets corresponding to the first imaging unit 501, the second imaging unit 502, and the orientation sensor 503.

[0137] In step S1511, the generation unit 507 determines whether or not the detection unit 505 has detected a synchronization signal (synchronization input) of the composite image received from the image processing device 104 via the I / F 509. If the result of this determination is that the detection unit 505 has detected a synchronization signal (synchronization input), the process proceeds to step S1512. On the other hand, if the detection unit 505 has not detected a synchronization signal (synchronization output), the process proceeds to step S1501.

[0138] In step S1512, upon receiving a notification from the detection unit 505 that a synchronization signal (synchronization input) for a composite image has been received, the generation unit 507 generates a “synchronization signal to be supplied to the first imaging unit 501” after an offset corresponding to the first imaging unit 501 from the detection timing of the synchronization signal, generates a “synchronization signal to be supplied to the second imaging unit 502” after an offset corresponding to the second imaging unit 502, and supplies the generated synchronization signals to the first imaging unit 501 and the second imaging unit 502. Furthermore, the generation unit 507 generates a “synchronization signal to be supplied to the orientation sensor 503” after an offset corresponding to the orientation sensor 503 from the detection timing of the synchronization signal, and supplies the generated synchronization signal to the orientation sensor 503.

[0139] In this manner, also in this embodiment, the display unit 504, the image capturing unit 501, the image capturing unit 502, and the attitude sensor 503 are operated synchronously based on the synchronization signal of the composite image supplied to the display unit 504, and the delay time from capturing to display can be further reduced. By considering the line of sight information of the wearer of the HMD 101 and the processing time of each device, etc., the image capturing exposure timing and the data acquisition timing of each device can be matched. Furthermore, it is also possible to accommodate setting changes of the exposure time of the first image capturing unit 501 and the second image capturing unit 502 and setting changes of the synchronization timing. By determining the image capturing exposure period and the synchronization reference timing 1403 of the data acquisition of each device in the region of interest of the wearer of the HMD 101, a more realistic MR experience without positional deviation between the captured image and the image in the virtual space is realized. Furthermore, if the gaze sensor 1301 has an external synchronization input function, it is also possible to use a similar control method to synchronize the timing of detection of gaze information by the gaze sensor 1301 with the operations of the first imaging unit 501, the second imaging unit 502, and the attitude sensor 503 based on a synchronization signal of a composite image supplied to the display unit 504.

[0140] In this case, the generation unit 507 generates a synchronization signal for controlling the acquisition timing of the gaze position detected by the gaze sensor 1301 based on the synchronization signal of the composite image so that a first time within the exposure time of the first imaging unit 501, a second time within the exposure time of the second imaging unit 502, the acquisition timing of the position and orientation detected by the orientation sensor 503, and the acquisition timing of the gaze position detected by the gaze sensor 1301 coincide with each other.

[0141] In addition, in this embodiment, the region of interest of the wearer of the HMD 101 is set to the gaze point detected by the gaze sensor 1301, but the method of setting the region of interest is not limited to this. For example, since the wearer of the HMD 101 generally observes the display image with the virtual object 204 as the center, it is also possible to use the rendering position information of the virtual object 204 included in the image 203 in the virtual space as the region of interest. This provides an advantage that there is no need to add the gaze sensor 1301. In this case, the control unit 506 functions as an identification unit and identifies the region of interest based on the rendering position of the virtual object 204 included in the image 203 in the virtual space.

[0142] (Third embodiment) In the above-described second embodiment, the control unit 506 uses the gaze point 1401 of the wearer of the HMD 101 detected by the gaze sensor 1301 to determine the synchronization reference line 1402. However, instead of the gaze point 1401 of the wearer of the HMD 101, the synchronization reference line 1402 may be determined using extraction information of the marker 202 extracted (recognized) from the captured image for the left eye and the captured image for the right eye in order to obtain position and orientation information in the generation unit 512. That is, the control unit 506 may specify a region of interest based on extraction information of the marker (feature) 202 extracted from the image captured by the second imaging unit 502, and determine the synchronization reference line 1402. Here, the center of gravity position of the extracted marker group may be used as the extraction information of the marker 202 extracted (recognized) from the captured image for the left eye and the captured image for the right eye in order to obtain position and orientation information in the generation unit 512. In addition, any region, such as the center of gravity position of the marker having the greatest influence on the position and orientation calculation result, may be used. The control unit 506 determines a synchronization reference line 1402 along which the extracted information area of ​​the marker 202 is exposed from the total exposure time of the first imaging unit 501, and a synchronization reference timing 1403 from the synchronization timing setting during the exposure time of the synchronization reference line 1402, and sets it in the generation unit 507.

[0143] In this manner, in the present embodiment, the display unit 504, the first imaging unit 501, the second imaging unit 502, and the orientation sensor 503 are operated in synchronization based on the synchronization signal of the composite image supplied to the display unit 504, and the delay time from imaging to display can be further reduced. By considering the line of sight information of the wearer of the HMD 101 and the processing time for each device, etc., the imaging exposure timing and the data acquisition timing of each device can be matched. By considering the extraction information of the marker 202 extracted (recognized) from the captured image for the left eye and the captured image for the right eye to obtain the position and orientation information in the generation unit 512, and the processing time for each device, etc., the imaging and data acquisition timing of each device can be matched to any timing during the exposure time. Furthermore, it is also possible to respond to setting changes in the exposure time of the first imaging unit 501 and the second imaging unit 502 and setting changes in the synchronization reference timing 1403. By matching the imaging and data acquisition timing of each device to any timing during the exposure time, a more realistic MR experience without positional deviation between the captured image and the image in the virtual space is realized. Also, as in the case where the rendering position information of the virtual object 204 included in the image 203 in the virtual space is used, there is an advantage that there is no need to add the gaze sensor 1301.

[0144] (Fourth embodiment) Each functional unit in the HMD 101 and the image processing device 104 shown in FIG. 5 and FIG. 13 may be implemented by hardware, or some of the functional units may be implemented by software (computer program).

[0145] In the latter case, in the HMD 101, the first imaging unit 501, the second imaging unit 502, the attitude sensor 503, the display unit 504, the I / F 509, and the gaze sensor 1301 may be implemented in hardware, and the remaining functional units may be implemented in software. In this case, the software is stored in a memory included in the HMD 101, and the processor included in the HMD 101 executes the software to realize the functions of the corresponding functional units.

[0146] 16(a) is a block diagram showing an example of a hardware configuration of an HMD 101 according to the fourth embodiment. The HMD 101 includes a processor 1610, a RAM 1620, a non-volatile memory 1630, an imaging unit 1640, a gaze sensor 1650, a posture sensor 1660, a display unit 1670, an I / F 1680, and a bus 1690.

[0147] The processor 1610 executes various processes using computer programs and data stored in the RAM 1620. As a result, the processor 1610 controls the overall operation of the HMD 101, and executes or controls each of the processes described above as being performed by the HMD 101.

[0148] The RAM 1620 has an area for storing computer programs and data loaded from the non-volatile memory 1630, and an area for storing data received from the image processing device 104 via the I / F 1680. Furthermore, the RAM 1620 has a work area used when the processor 1610 executes various processes. In this way, the RAM 1620 can provide various areas as appropriate.

[0149] The non-volatile memory 1630 stores computer programs and data for causing the processor 1610 to execute or control the above-mentioned operation of the HMD 101. The computer programs stored in the non-volatile memory 1630 include computer programs for causing the CPU 1601 to execute the functions of the functional units (excluding the first imaging unit 501, the second imaging unit 502, the attitude sensor 503, the display unit 504, the I / F 509, and the gaze sensor 1301) of the HMD 101 shown in Fig. 5 or Fig. 13. The computer programs and data stored in the non-volatile memory 1630 are loaded into the RAM 1620 as appropriate under the control of the processor 1610, and become targets for processing by the processor 1610.

[0150] The imaging unit 1640 includes the first imaging unit 501 and the second imaging unit 502. The gaze sensor 1650 includes the gaze sensor 1301. The attitude sensor 1660 includes the attitude sensor 503. The display unit 1670 includes the display unit 504. The I / F 1680 includes the I / F 509. The processor 1610, the RAM 1620, the non-volatile memory 1630, the imaging unit 1640, the gaze sensor 1650, the attitude sensor 1660, the display unit 1670, and the I / F 1680 are all connected to a bus 1690. Note that the configuration shown in FIG. 16(a) is an example of a configuration applicable to the HMD 101, and can be changed / modified as appropriate.

[0151] In addition, as for the image processing device 104, any computer device capable of executing software corresponding to each functional unit except for the I / F 510 and the content DB 513 can be applied to the image processing device 104. An example of the hardware configuration of a computer device applicable to the image processing device 104 will be described with reference to the block diagram of Fig. 16(b). The image processing device 104 has a CPU 1601, a RAM 1602, a ROM 1603, an operation unit 1604, a display unit 1605, an external storage device 1606, an I / F 1607, and a bus 1608.

[0152] The CPU 1601 executes various processes using computer programs and data stored in the RAM 1602 and the ROM 1603. As a result, the CPU 1601 controls the overall operation of the image processing device 104, and executes or controls each of the processes described above as being performed by the image processing device 104.

[0153] The RAM 1602 has an area for storing computer programs and data loaded from the ROM 1603 or the external storage device 1606, and an area for storing data received from the HMD 101 via the I / F 1607. The RAM 1602 also has a work area used when the CPU 1601 executes various processes. In this way, the RAM 1602 can provide various areas as appropriate. The ROM 1603 stores setting data, startup programs, and the like for the image processing device 104.

[0154] The operation unit 1604 is a user interface such as a keyboard, a mouse, or a touch panel, and the user can input various instructions to the CPU 1601 by operating it.

[0155] The display unit 1605 is configured with a liquid crystal screen, a touch panel screen, or the like, and can display the results of processing by the CPU 1601 as images, characters, etc. The display unit 1605 may be a projection device such as a projector that projects images and characters.

[0156] The external storage device 1606 is a large-capacity information storage device such as a hard disk drive. An OS (operating system) is stored in the external storage device 1606. The external storage device 1606 also stores computer programs and data for causing the CPU 1601 to execute the functions of each functional unit (excluding the I / F 510 and the content DB 513) of the image processing device 104 shown in Fig. 5 or 13. The external storage device 1606 also stores the above-mentioned content DB 513.

[0157] Computer programs and data stored in the external storage device 1606 are loaded into the RAM 1602 as appropriate under the control of the CPU 1601 and are processed by the CPU 1601 .

[0158] The I / F 1607 is a communication interface for performing data communication with the HMD 101, and functions as the above-mentioned I / F 510. That is, the image processing device 104 performs data communication with the HMD 101 via the I / F 1607.

[0159] The CPU 1601, RAM 1602, ROM 1603, operation unit 1604, display unit 1605, external storage device 1606, and I / F 1607 are all connected to a bus 1608. Note that the configuration shown in Fig. 16(b) is an example of a configuration applicable to the image processing device 104, and can be changed / modified as appropriate.

[0160] Fifth embodiment In each of the above embodiments, the markers 202 artificially placed in the real space are used to obtain the position and orientation of the imaging unit. However, in addition to or instead of the markers 202, the position and orientation of the imaging unit may be obtained using natural features that originally exist in the real space (for example, corners of furniture such as chairs and desks, corners of buildings and cars that make up the scenery).

[0161] 5 or 13 is an example. For example, each process described above as being performed by the HMD 101 may be shared and executed by a plurality of devices, and each process described above as being performed by the image processing device 104 may be shared and executed by a plurality of devices.

[0162] Also, instead of the head-mounted display device, a "portable device having a plurality of imaging units with different shutter methods, an orientation sensor 503, and a display unit 504" such as a smartphone may be used. Also, such a portable device may be added to the MR system in addition to the head-mounted display device. In such a case, the image processing device 104 generates an image of the mixed reality space according to the position and orientation of the head-mounted display device and delivers it to the head-mounted display device, and generates an image of the mixed reality space according to the position and orientation of the portable device and delivers it to the portable device. The method of generating the image of the mixed reality space is the same as in the above embodiment.

[0163] Furthermore, the HMD 101 and the image processing device 104 may be integrated together, or instead of a head-mounted display device, the above-mentioned portable device and the image processing device 104 may be integrated together.

[0164] In addition, in the above embodiment, the posture sensor 503 has been described as being possessed by the HMD 101, but this is not limited thereto. For example, the posture sensor 503 may be configured to obtain necessary information from an image captured by an objective camera installed around the wearer of the HMD 101.

[0165] In addition, the numerical values, calculation methods, processing execution timing, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and it is not intended that each embodiment be limited to these examples.

[0166] In addition, a part or all of the above-described embodiments may be used in appropriate combination. In addition, a part or all of the above-described embodiments may be used selectively.

[0167] (Other embodiments) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.

[0168] It should be noted that the above-described embodiments are merely illustrative of specific examples of implementing the present disclosure, and the technical scope of the present disclosure should not be interpreted as being limited by these embodiments. In other words, the present disclosure can be implemented in various forms without departing from its technical concept or main features.

[0169] The disclosure of this embodiment includes the following configurations, systems, and methods. (Configuration 1) A display unit; A first imaging unit; A second imaging unit; a generating unit that generates a first signal for controlling an exposure start of the first imaging unit and a second signal for controlling an exposure start of the second imaging unit based on a signal indicating an image input timing to the display unit; A head-mounted display device comprising: (Configuration 2) The head-mounted display device described in configuration 1, characterized in that the generation unit generates the first signal and the second signal so that a first time within an exposure time of the first imaging unit and a second time within an exposure time of the second imaging unit coincide with each other. (Configuration 3) The head-mounted display device described in configuration 1 or 2, characterized in that when the exposure time of the first imaging unit or the exposure time of the second imaging unit is changed, the generation unit changes the generation timing of the first signal or the second signal so that a first time within the exposure time of the first imaging unit and a second time within the exposure time of the second imaging unit coincide with each other. (Configuration 4) The head-mounted display device of any one of configurations 1 to 3, characterized in that when the processing time of the images captured by the first imaging unit and the second imaging unit varies between frames, the generation unit changes the generation timing of the first signal or the second signal. (Configuration 5) The head-mounted display device according to any one of configurations 1 to 4, further comprising an identification unit for identifying a region of interest within the imaging range of the first imaging unit. (Configuration 6) The head-mounted display device described in configuration 5, characterized in that the identification unit is a first detection unit that detects the gaze position of the wearer of the head-mounted display device within the imaging range of the first imaging unit, and identifies the gaze position as the area of ​​interest. (Configuration 7) A head-mounted display device as described in configuration 5 or 6, characterized in that a first time within the exposure time of the first imaging unit is set based on a region of interest in the imaging range of the first imaging unit. (Configuration 8) the head-mounted display device further includes a second detection unit that detects a position and orientation of the head-mounted display device, The head-mounted display device according to any one of configurations 1 to 7, characterized in that the generation unit generates a third signal for controlling the acquisition timing of the position and orientation detected by the second detection unit based on a signal indicating the timing of image input to the display unit. (Configuration 9) The head-mounted display device described in configuration 8, characterized in that the generation unit generates the first signal, the second signal, and the third signal so that a first time within an exposure time of the first imaging unit, a second time within an exposure time of the second imaging unit, and an acquisition timing of the position and orientation detected by the second detection unit coincide with each other. (Configuration 10) The head-mounted display device described in configuration 6, characterized in that the generation unit generates a fourth signal for controlling the acquisition timing of the gaze position detected by the first detection unit based on a signal indicating the timing of image input to the display unit. (Configuration 11) The head-mounted display device described in configuration 10, characterized in that the generation unit generates the first signal, the second signal, and the fourth signal so that a first time within an exposure time of the first imaging unit, a second time within an exposure time of the second imaging unit, and the acquisition timing of the gaze position detected by the first detection unit coincide with each other. (Configuration 12) the first imaging unit captures an image using a first shutter method; 12. The head-mounted display device according to any one of configurations 1 to 11, wherein the second imaging section captures images using a second shutter mode different from the first shutter mode. (Configuration 13) the first shutter method is a rolling shutter method, 13. The head-mounted display device according to configuration 12, wherein the second shutter method is a global shutter method. (Configuration 14) The head-mounted display device according to any one of configurations 1 to 13, characterized in that the display unit displays a composite image of an image captured by the first imaging unit and an image of a virtual space based on an image captured by the second imaging unit. (Configuration 15) The head-mounted display device of configuration 8 or 9, characterized in that the display unit displays a composite image of an image captured by the first imaging unit, an image captured by the second imaging unit, and an image of a virtual space based on the position and orientation detected by the second detection unit. (Configuration 16) the display unit displays a composite image of an image captured by the first imaging unit, an image captured by the second imaging unit, and an image of a virtual space based on the position and orientation detected by the second detection unit; The head-mounted display device according to configuration 5, wherein the identification unit identifies the region of interest based on a drawing position of a virtual object included in the image of the virtual space. (Configuration 17) The head-mounted display device described in configuration 5, characterized in that the identification unit identifies the region of interest based on extraction information of features extracted from an image captured by the second imaging unit. (Configuration 18) 18. The head-mounted display device according to claim 17, wherein the extracted features are markers. (Configuration 19) the generation unit generates the first signal and the second signal such that a first time within an exposure time of the first imaging unit and a second time within an exposure time of the second imaging unit coincide with each other; a first time within an exposure time of the first imaging unit is a center time of an exposure time of a specific line of the first imaging unit, 14. The head-mounted display device according to configuration 13, wherein the second time within the exposure time of the second imaging unit is a central time of the exposure time of the second imaging unit. (Configuration 20) the generation unit generates the first signal and the second signal such that a first time within an exposure time of the first imaging unit and a second time within an exposure time of the second imaging unit coincide with each other; a first time within an exposure time of the first imaging unit is an exposure start time of a specific line of the first imaging unit, 14. The head-mounted display device according to configuration 13, wherein the second time within the exposure time of the second imaging unit is an exposure start time of the second imaging unit. (Configuration 21) the generation unit generates the first signal and the second signal such that a first time within an exposure time of the first imaging unit and a second time within an exposure time of the second imaging unit coincide with each other; a first time within an exposure time of the first imaging unit is an exposure end time of a specific line of the first imaging unit; 14. The head-mounted display device according to configuration 13, wherein the second time within the exposure time of the second imaging unit is an exposure end time of the second imaging unit. (System 1) A head-mounted display device according to configuration 1; an image processing device; The image processing device includes: a synthesis unit that generates an image of a virtual space based on an image captured by the second imaging unit, and generates a synthetic image by synthesizing the image of the virtual space and the image captured by the first imaging unit; a transmission unit that transmits the composite image and a signal indicating the image input timing to the head-mounted display device; The system is characterized in that the display unit displays the composite image. (System 2) A head-mounted display device according to configuration 8; an image processing device; The image processing device includes: a synthesis unit that generates an image of a virtual space based on the image captured by the second imaging unit and the position and orientation detected by the second detection unit, and generates a synthetic image by synthesizing the image of the virtual space and the image captured by the first imaging unit; a transmission unit that transmits the composite image and a signal indicating the image input timing to the head-mounted display device; The system is characterized in that the display unit displays the composite image. (Method 1) A method for controlling a head-mounted display device having a display unit, a first imaging unit, and a second imaging unit, comprising: A control method for a head-mounted display device, comprising a generation step of generating a first signal for controlling the start of exposure of the first imaging unit and a second signal for controlling the start of exposure of the second imaging unit based on a signal indicating the timing of image input to the display unit. [Explanation of symbols]

[0170] 101 HMD, 501 first imaging unit, 502 second imaging unit, 503 attitude sensor, 504 display unit, 505 detection unit, 506 control unit, 507 generation unit, 508 setting unit, 509 I / F

Claims

1. A display unit; a first imaging unit; a second imaging unit; and a generating unit that generates a first signal for controlling the start of exposure of the first imaging unit and a second signal for controlling the start of exposure of the second imaging unit based on a signal indicating an image input timing to the display unit; A head-mounted display device comprising:

2. The head-mounted display device according to claim 1, characterized in that the generation unit generates the first signal and the second signal so that a first time within the exposure time of the first imaging unit and a second time within the exposure time of the second imaging unit coincide with each other.

3. The head-mounted display device according to claim 1, characterized in that when the exposure time of the first imaging unit or the exposure time of the second imaging unit is changed, the generation unit changes the generation timing of the first signal or the second signal so that a first time within the exposure time of the first imaging unit and a second time within the exposure time of the second imaging unit coincide with each other.

4. The head-mounted display device according to claim 1, characterized in that when the processing time of the images captured by the first imaging unit and the second imaging unit varies between frames, the generation unit changes the generation timing of the first signal or the second signal.

5. The head-mounted display device according to claim 1 , further comprising an identifying unit for identifying a region of interest within the imaging range of the first imaging unit.

6. The head-mounted display device described in claim 5, characterized in that the identification unit is a first detection unit that detects the gaze position of the wearer of the head-mounted display device within the imaging range of the first imaging unit, and identifies the gaze position as the area of ​​interest.

7. 6. The head-mounted display device according to claim 5, wherein the first time period within the exposure time of the first image capturing unit is set based on a region of interest in the image capturing range of the first image capturing unit.

8. the head-mounted display device further includes a second detection unit that detects a position and orientation of the head-mounted display device; The head-mounted display device according to claim 1, characterized in that the generation unit generates a third signal for controlling the timing of acquiring the position and orientation detected by the second detection unit based on a signal indicating the timing of image input to the display unit.

9. The head-mounted display device according to claim 8, characterized in that the generation unit generates the first signal, the second signal, and the third signal so that a first time within the exposure time of the first imaging unit, a second time within the exposure time of the second imaging unit, and the timing of acquiring the position and orientation detected by the second detection unit coincide with each other.

10. The head-mounted display device described in claim 6, characterized in that the generation unit generates a fourth signal for controlling the acquisition timing of the gaze position detected by the first detection unit based on a signal indicating the timing of image input to the display unit.

11. The head-mounted display device described in claim 10, characterized in that the generation unit generates the first signal, the second signal, and the fourth signal so that a first time within the exposure time of the first imaging unit, a second time within the exposure time of the second imaging unit, and the acquisition timing of the gaze position detected by the first detection unit coincide with each other.

12. the first imaging unit captures an image using a first shutter method, 2. The head-mounted display device according to claim 1, wherein the second image capturing unit captures images using a second shutter method different from the first shutter method.

13. the first shutter method is a rolling shutter method, 13. The head-mounted display device according to claim 12, wherein the second shutter method is a global shutter method.

14. The head-mounted display device according to claim 1, characterized in that the display unit displays a composite image of an image captured by the first imaging unit and an image of a virtual space based on an image captured by the second imaging unit.

15. The head-mounted display device according to claim 8, characterized in that the display unit displays a composite image of an image captured by the first imaging unit, an image captured by the second imaging unit, and an image of a virtual space based on the position and orientation detected by the second detection unit.

16. The head-mounted display device further has a second detection unit that detects a position and orientation of the head-mounted display device, the display unit displays a composite image of the image captured by the first imaging unit, the image captured by the second imaging unit, and an image of a virtual space based on the position and orientation detected by the second detection unit; The head-mounted display device according to claim 5 , wherein the specifying unit specifies the region of interest based on a rendering position of a virtual object included in the image of the virtual space.

17. The head-mounted display device according to claim 5 , wherein the specifying unit specifies the region of interest based on extracted information of features extracted from the image captured by the second imaging unit.

18. 18. The head-mounted display device according to claim 17, wherein the extracted features are markers.

19. A control method for a head-mounted display device having a display unit, a first imaging unit, and a second imaging unit, comprising: A control method for a head-mounted display device, comprising a generation step of generating a first signal for controlling the start of exposure of the first imaging unit and a second signal for controlling the start of exposure of the second imaging unit based on a signal indicating the timing of image input to the display unit.

20. A program for causing a computer to execute the control method for a head-mounted display device described in claim 19.

Images