Imaging apparatus and control method for the same

Abstract

To make it possible to achieve a stable frame rate on top of synchronizing a plurality of imaging units or synchronizing the imaging units and a sensing unit.SOLUTION: An imaging apparatus has a first imaging unit that takes an image by exposure during a first exposure period, a second imaging unit that takes an image by exposure during a second exposure period, a first timestamp generation unit for generating a first timestamp indicating an exposure timing within the first exposure period of the first imaging unit, and a second timestamp generation unit for generating a second timestamp indicating an exposure timing within the second exposure period of the second imaging unit.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an imaging device and a control method for the imaging device. [Background technology]

[0002] In recent years, mixed reality (MR) technology has become known as a technology that seamlessly blends the real world and the virtual world in real time. One MR technology is a technology that uses a video see-through HMD (Head Mounted Display) to capture an image of a subject that roughly matches the subject observed from the HMD user's pupil position using a video camera or the like. With this technology, the HMD user can view an image in which CG (Computer Graphics) is superimposed on the captured image.

[0003] This technology obtains positional information about the space experienced by the HMD user by calculating the position and orientation of the HMD through calculations using multiple captured images and information from various sensors. If the positional information of the HMD user can be obtained accurately, it becomes possible to render CG in the appropriate space, allowing the HMD user to be immersed in the MR experience.

[0004] In order to accurately obtain position information, it is desirable that the multiple imaging means and various sensors be operated in a synchronized state as much as possible. In response to this, Patent Document 1 describes a technique for synchronizing the multiple imaging means by aligning the centers of gravity of the exposure times. Furthermore, Patent Document 2 describes a technique for synchronizing the multiple imaging means and various sensors. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-5608 [Patent Document 2] Patent Publication No. 2021-117057 Summary of the Invention [Problem to be solved by the invention]

[0006] However, while the technologies described in Patent Documents 1 and 2 can synchronize multiple captured images and various sensors on a frame-by-frame basis, the frame rate fluctuates and is not constant in order to synchronize them, resulting in MR images that feel unnatural to the HMD user.

[0007] An object of the present disclosure is to make it possible to achieve a stable frame rate by synchronizing multiple image capturing units or an image capturing unit and a sensing unit. [Means for solving the problem]

[0008] The imaging device includes a first imaging unit that captures an image by exposure during a first exposure period, a second imaging unit that captures an image by exposure during a second exposure period, a first timestamp generation unit that generates a first timestamp indicating the exposure timing of the first imaging unit within the first exposure period, and a second timestamp generation unit that generates a second timestamp indicating the exposure timing of the second imaging unit within the second exposure period. [Effects of the Invention]

[0009] According to the present disclosure, a stable frame rate can be achieved by synchronizing multiple imaging units or an imaging unit and a sensing unit. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an image display system. [Figure 2] FIG. 2 is a diagram illustrating an example of the functional configuration of an HMD and an image processing device. [Figure 3] 4 is a timing chart showing processing of the imaging unit. [Figure 4] 10 is a flowchart showing a method for controlling an HMD. [Figure 5]10 is a timing chart showing a method for controlling an HMD. [Figure 6] FIG. 2 is a diagram illustrating an example of the functional configuration of an HMD and an image processing device. [Figure 7] 10 is a timing chart showing a method for controlling an HMD. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments will be described in detail below with reference to the drawings. (First embodiment) 1 is a diagram showing an example of the configuration of an image display system 100 according to the first embodiment. The image display system 100 has a head mounted display (hereinafter referred to as HMD) 101, a display device 102, an operation unit 104, and an image processing device 103. The display device 102 and the operation unit 104 are connected to the image processing device 103. The image processing device 103 inputs operation information of a user of the operation unit 104, and displays images and information on the display device 102.

[0012] The HMD 101 is worn on the user's head and displays an image generated by an image processing device 103. The HMD 101 communicates with the image processing device 103 wirelessly connected via a small-scale network. The small-scale network is, for example, a wireless local area network (WLAN) or a wireless personal area network (WPAN). Communication between the HMD 101 and the image processing device 103 is not limited to wireless communication, and a wired communication method may also be used. Furthermore, although the image processing device 103 and the HMD 101 have separate hardware configurations, it is also possible to integrate them by implementing all of the functions of the image processing device 103 within the HMD 101.

[0013] Fig. 2 is a block diagram showing an example of the functional configuration of the HMD 101 and the image processing device 103 shown in Fig. 1. The HMD 101 includes imaging units 201 and 202, a display unit 203, a sensing unit 204, a motion calculation unit 205, an imaging processing unit 206, a calculation unit 207, an image synthesis unit 208, a synchronization unit 220, and a timestamp management unit 221.

[0014] The imaging units 201 and 202 each have an objective optical system and an image sensor for capturing images of the outside world. The display unit 203 has an eyepiece optical system and a display for presenting images to the user of the HMD 101. The sensing unit 204 senses the movement of the HMD 101. The movement calculation unit 205 calculates the movement of the HMD 101 from the sensing results of the sensing unit 204. The imaging processing unit 206 processes the captured images from the imaging units 201 and 202. The calculation unit 207 performs calculations. The image synthesis unit 208 synthesizes the captured images with CG images. The synchronization unit 220 synchronizes the imaging units 201 and 202 and the sensing unit 204. The timestamp management unit 221 manages timestamps.

[0015] The image processing device 103 has a position and orientation calculation unit 209, a CG generation unit 210, and a content DB (database) 211. The position and orientation calculation unit 209 calculates the position and orientation of the HMD 101 from the captured image and the movement of the HMD 101. The content DB 211 stores CG data. The CG generation unit 210 reads the CG data from the content DB 211 based on the position and orientation information calculated by the position and orientation calculation unit 209, and then generates a CG image.

[0016] Next, a method for a user to experience MR images will be described with reference to FIG. 2. The image capturing units 201 and 202 capture images of the external world. Taking into consideration various factors such as the number of pixels, image quality, noise, sensor size, power consumption, and cost, the image capturing units 201 and 202 may selectively use either a rolling shutter image capturing element or a global shutter image capturing element, or may combine them depending on the application. For example, when capturing an image to be combined with an image of a virtual space, the image capturing unit 201 uses a rolling shutter image capturing element capable of acquiring higher-quality images. When capturing an image of the external world to calculate the position and orientation of the HMD 101, the image capturing unit 202 uses a global shutter image capturing element that does not suffer from image streaming. Image streaming is a phenomenon that occurs due to the operating principle of the rolling shutter method, in which exposure processing is initiated sequentially for each line in the scanning direction.

[0017] Fig. 3 is a timing chart showing an example of processing by the rolling shutter imaging unit 201 and the global shutter imaging unit 202. The horizontal axis in Fig. 3 represents time. An example will be described in which the imaging unit 201 uses an imaging element of the rolling shutter type, and the imaging unit 202 uses an imaging element of the rolling shutter type.

[0018] The exposure (rolling) of the imaging unit 201 indicates the exposure time of each line captured by a rolling shutter type imaging element. In the rolling shutter type imaging unit 201, a time lag occurs in the exposure timing of each line, and when the imaging unit 201 or the subject moves during the exposure time, a phenomenon occurs in which the subject is recorded in a distorted, flowing manner. This phenomenon is called image flow.

[0019] The exposure (global) of the imaging unit 202 indicates the exposure time of each line captured by a global shutter type imaging element. In the global shutter type imaging unit 202, exposure processing is performed on all lines simultaneously, so there is no time lag in the exposure timing of each line and image blurring does not occur. In this embodiment, the imaging unit 201 will be described as using a rolling shutter type imaging element and the imaging unit 202 as using a global shutter type imaging element.

[0020] 2, imaging units 201 and 202 output captured images of the outside world to timestamp management unit 221. When the captured images are input, timestamp management unit 221 issues a timestamp. The detailed operation of this will be described later.

[0021] The image capture processing unit 206 performs image processing on the captured images captured by the image capture units 201 and 202. The image processing performed by the image capture processing unit 206 includes AE (Auto Exposure) control, demosaic processing, shading correction, noise reduction, distortion correction, etc. The image capture processing unit 206 performs image processing on the captured images captured by the image capture units 201 and 202 to improve the image quality. Thereafter, the image capture processing unit 206 outputs an image obtained by image processing the captured image captured by the image capture unit 202 to a position and orientation calculation unit 209 to specify the rendering position of the CG after image processing. The image capture processing unit 206 also outputs an image obtained by image processing the captured image captured by the image capture unit 201 to an image synthesis unit 208 to synthesize it with a CG image.

[0022] The sensing unit 204 senses the movement of the HMD 101. The sensors of the sensing unit 204 include an IMU (Inertial Measurement Unit), an acceleration sensor, and an angular velocity sensor. In this embodiment, the sensing unit 204 is, for example, an IMU. The movement calculation unit 205 calculates the movement, tilt, rotation, etc. of the HMD 101 from the sensing data output from the IMU, and outputs the calculated movement, tilt, rotation, etc. to the position and orientation calculation unit 209.

[0023] The position and orientation calculation unit 209 calculates the relationship between a world coordinate system representing the real world and a camera coordinate system representing an image observed through the HMD 101 based on the image captured by the imaging unit 202 after image processing and the calculation result of the motion calculation unit 205. Then, based on this relationship, the position and orientation calculation unit 209 calculates the position and orientation of the HMD 101 relative to the real world. For example, the position and orientation calculation method may include placing reference markers or the like in the real world, acquiring images using the imaging unit 202 configured with a stereo camera, and calculating the position and orientation of the HMD 101 based on the positional relationship of the markers in the images. Another method may include calculating feature amounts of stationary objects in the real world from the images, and estimating the self-position of the HMD 101 and creating an environmental map based on changes in the feature amounts over time. In this embodiment, the method for calculating the position and orientation is not particularly limited. The position and orientation calculation unit 209 outputs the position and orientation information calculated in this manner to the CG generation unit 210.

[0024] The CG generation unit 210 calculates the position and orientation of the captured image to superimpose CG on, based on the position and orientation information from the position and orientation calculation unit 209. The CG generation unit 210 also performs CG rendering by reading CG data from the content DB 211. As a result, the CG generation unit 210 generates a CG image that matches the orientation of the captured image captured by the imaging unit 202, and outputs the CG image to the image synthesis unit 208.

[0025] The image synthesis unit 208 generates an MR image by synthesizing the image captured by the imaging unit 201 after image processing with the CG image obtained from the CG generation unit 210, and presents (displays) the MR image on the display unit 203. By displaying the MR image on the display unit 203, the user of the HMD 101 can visually recognize objects that do not exist in the outside world as CG and experience MR.

[0026] At this time, as described above, the CG image generated by the CG generating unit 210 is generated based on the image captured by the imaging unit 202 after image synthesis. On the other hand, the image synthesized by the image synthesis unit 208 is the image captured by the imaging unit 201 after image processing. In other words, if the timing of capturing images by the imaging units 201 and 202 differs, an unnatural MR image will be generated when the image synthesis unit 208 synthesizes the images.

[0027] 3, a method for aligning the centers of gravity of exposure of the multiple image capturing units 201 and 202 will be described. If the centers of gravity of exposure of the image capturing units 201 and 202 can be aligned, there will be no time lag between the time when the image capturing units 201 and 202 capture images.

[0028] 3 shows a state in which the external synchronization input from the synchronization unit 220 to the two imaging units 201 and 202 is changed to match the centers of gravity of exposure for the imaging units 201 and 202. Fig. 3 shows a timing chart of the imaging exposure for the imaging unit 201 and a timing chart of the imaging exposure for the imaging unit 202.

[0029] When external synchronization is input from the synchronization unit 220, the imaging unit 201 starts exposure after an exposure start wait time A1_f0 that varies depending on the set shutter speed (for example, 1 / 120 s in the case of Frame 0). At this time, the imaging unit 201 uses a rolling shutter type image sensor and therefore starts exposure sequentially for each line. Then, after the imaging unit 201 has been exposed for the set shutter speed time, it ends exposure sequentially for each line. Then, when exposure ends, the imaging unit 201 outputs image data in raster order. When outputting image data, the imaging unit 201 also outputs synchronization signals such as Vsync indicating the start of a frame and Hsync indicating the start of a line, in addition to the image data. When image data is input, the imaging processing unit 206 acquires the image data based on the synchronization signals such as Vsync and Hsync.

[0030] Similarly to the imaging unit 201, when external synchronization input is received from the synchronization unit 220, the imaging unit 202 starts exposure after an exposure start wait time A2_f0 that varies depending on the set shutter speed (for example, 1 / 120 seconds in the case of Frame 0). At this time, the imaging unit 202 uses a global shutter type image sensor, so it starts exposure for all lines simultaneously. Then, once the imaging unit 202 has been exposed for the set shutter speed time, it ends exposure for all lines simultaneously. Then, once exposure is completed, the imaging unit 202 outputs image data in raster order.

[0031] As described above, the image capture unit 201 uses a rolling shutter image sensor, and the image capture unit 202 uses a global shutter image sensor. Therefore, to align the centers of exposure of the image capture units 201 and 202, it is necessary to relatively shift the exposure start times of the image capture units 201 and 202 in consideration of the shutter speed settings of the image capture units 201 and 202. Therefore, in FIG. 3 , the alignment of the centers of exposure is achieved by changing the timing of the external synchronization input to the image capture unit 202 from that of the external synchronization input to the image capture unit 201. In this way, the synchronizer 220 changes the external synchronization input timing for each of the image capture units 201 and 202 so as to align the centers of exposure of the multiple image capture units 201 and 202.

[0032] However, with the above method, the external synchronization input timing changes depending on the shutter speed of the image capturing units 201 and 202. Since the time from external synchronization input to image data output is constant, if the external synchronization input timing is not constant, the image data output time between frames (frame rate) will fluctuate. If the frame rate of the captured images output from the image capturing units 201 and 202 fluctuates, the frame rate for updating the images displayed on the display unit 203 will also fluctuate, posing a problem that the user will not be able to enjoy a comfortable MR experience.

[0033] Therefore, in this embodiment, the HMD 101 uses a synchronization unit 220 and a timestamp management unit 221 to share external synchronization input among the multiple image capture units 201 and 202. The HMD 101 then attempts to solve the above problem by linking the deviation of the exposure centers of gravity of the image capture units 201 and 202 to image data as timestamp information.

[0034] Next, the synchronization unit 220 of this embodiment will be described. As shown in FIG. 2, the synchronization unit 220 generates an external synchronization signal to synchronize the imaging units 201 and 202. When the vertical synchronization signal Vsync is input from the image composition unit 208, the synchronization unit 220 generates the external synchronization signal after a certain offset time has elapsed. This is because, by making the imaging rates of the imaging units 201 and 202 and the update rate of the display unit 203 the same, it is possible to provide MR images smoothly and with low delay. However, the method for generating the external synchronization signal is not particularly limited to this. The synchronization unit 220 may automatically generate a timing signal internally and output the timing signal as an external synchronization signal, or may generate and output an external synchronization signal triggered by the vertical synchronization signal Vsync of a specific imaging unit 201 or 202.

[0035] Next, the timestamp management unit 221 of this embodiment will be described with reference to Figures 4 and 5. Figure 4 is a flowchart showing an example of processing by the timestamp management unit 221. The timestamp management unit 221 has an internal time counter Tcnt.

[0036] In step S401, the timestamp management unit 221 starts the operation of the time counter Tcnt. The time counter Tcnt starts counting a reference time without being affected by other operations. Note that the timestamp management unit 221 may use the value of an internal counter as the value of the time counter Tcnt, or may generate the value of the time counter Tcnt based on a reference signal input from an external device.

[0037] In step S402, the timestamp management unit 221 determines whether or not it has received an external synchronization signal output by the synchronization unit 220. If the timestamp management unit 221 has not received an external synchronization signal output by the synchronization unit 220, it returns to step S402, and if it has received an external synchronization signal output by the synchronization unit 220, it proceeds to step S403.

[0038] In step S403, the timestamp management unit 221 holds the value of the time counter Tcnt at the time when the external synchronization signal is input from the synchronization unit 220 as the external synchronization time T(n). The synchronization unit 220 also outputs this external synchronization signal to the image capture units 201 and 202. The image capture units 201 and 202 enter a preparation state for exposure at the same time as the external synchronization time T(n).

[0039] In step S404, the timestamp management unit 221 acquires exposure time parameters set in the image capture units 201 and 202. The exposure time parameters include shutter speed values ​​and exposure mode settings set in registers for the image capture units 201 and 202. This enables the timestamp management unit 221 to acquire information related to the exposure time in each of the image capture units 201 and 202.

[0040] In step S405, the timestamp management unit 221 identifies an offset value Toffset1 of the exposure center of gravity of the image capture unit 201 and an offset value Toffset2 of the exposure center of gravity of the image capture unit 202 from the exposure time parameters of the image capture units 201 and 202. The offset values ​​Toffset1 and Toffset2 indicate the time from the external synchronization time T(n) to the exposure centers of gravity of the image capture units 201 and 202, respectively, and are values ​​uniquely determined by the characteristics of the image capture units 201 and 202 and the set shutter speeds. The offset value Toffset1 can be expressed, for example, as Toffset1(n) = A1_f(n) + B1_f(n). The offset value Toffset2 can be expressed, for example, as Toffset2(n) = A2_f(n) + B2_f(n). A1_f(n) and A2_f(n) are the times from the input of the external synchronization signal to the start of exposure, as in A1_f0 and A2_f0 in FIG. 5, respectively. B1_f(n) and B2_f(n) are the times from the start of exposure to the arrival at the center of exposure, as in B1_f0 and B2_f0 in FIG. 5, respectively. n indicates the frame number. The times A1_f(n), A2_f(n), B1_f(n), and B2_f(n) are determined by the characteristics of the image capture unit 201 or 202 and the set shutter speed. Therefore, the times A1_f(n), A2_f(n), B1_f(n), and B2_f(n) may be calculated based on the shutter speed set by the calculation unit 207 for each frame, or may be obtained by referencing a table of possible shutter speeds. In this way, the timestamp management unit 221 identifies the offset value Toffset of the exposure center of gravity of the imaging unit 201 and the offset value Toffset2 of the exposure center of gravity of the imaging unit 202 from the exposure time parameters of the imaging units 201 and 202. The method of identification is not particularly limited.

[0041] In step S406, the timestamp management unit 221 issues a timestamp S1(n) for the image capture unit 201 and a timestamp S2(n) for the image capture unit 202, and stores the timestamps S1(n) and S2(n). The timestamp management unit 221 calculates the timestamp S1(n) by S1(n) = T(n) + Toffset1(n) and calculates the timestamp S2(n) by S2(n) = T(n) + Toffset2(n). The timestamps S1(n) and S2(n) indicate the exposure center times of the image capture units 201 and 202, respectively.

[0042] In step S407, the timestamp management unit 221 determines whether or not a vertical synchronization signal Vsync has been input from the imaging unit 201 or 202. If the timestamp management unit 221 has not input a vertical synchronization signal Vsync from the imaging unit 201 or 202, the process returns to step S407, and if the timestamp management unit 221 has input a vertical synchronization signal Vsync from the imaging unit 201 or 202, the process proceeds to step S408.

[0043] In step S408, when the timestamp management unit 221 receives a vertical synchronization signal Vsync from the imaging unit 201, it outputs the timestamp S1(n) of the imaging unit 201 in association with the image data D1(n) received from the imaging unit 201. Furthermore, when the timestamp management unit 221 receives a vertical synchronization signal Vsync from the imaging unit 202, it outputs the timestamp S2(n) of the imaging unit 202 in association with the image data D2(n) received from the imaging unit 202. This allows the timestamp management unit 221 to complete image processing in units of one frame, and the process returns to step S402.

[0044] Thereafter, the timestamp management unit 221 repeats the same process for the next frame, thereby outputting the image data D1(n) and D2 for each frame in correspondence with the timestamps S1(n) and S2(n) indicating the exposure center of gravity.

[0045] Fig. 5 is a timing chart showing an example of a control method for the HMD 101 according to this embodiment. The timestamp management unit 221 performs the processing of the flowchart in Fig. 4. Below, Fig. 5 will be used to supplement the explanation of the processing of the flowchart described in Fig. 4.

[0046] Time E0 is the time when the synchronizer 220 outputs an external synchronization signal to the imaging units 201 and 202, the sensing unit 204, and the timestamp manager 221. A common external synchronization signal is input to the imaging units 201, 202, the sensing unit 204, and the timestamp manager 221. The timestamp manager 221 holds the value of the time counter Tcnt at time E0 as the external synchronization time T0.

[0047] At time T0, the shutter speed setting of the imaging unit 201 is 1 / 120 s, and the imaging unit 201 uses a rolling shutter type image sensor. Time A1_f0 is the time from when the imaging unit 201 receives an external synchronization signal until exposure begins. Time B1_f0 is the time from when exposure of the imaging unit 201 begins until the exposure reaches the center of gravity. Times A1_f0 and B1_f0 are values ​​that are uniquely determined by the characteristics of the imaging unit 201 and the shutter speed that is set. The timestamp management unit 221 calculates timestamp S1(0) by S1(0) = T0 + A1_f0 + B1_f0.

[0048] Time R0 is the time when the imaging unit 201 outputs the vertical synchronization signal Vsync and starts outputting image data D1(0). At time R0, the timestamp management unit 221 associates timestamp S1(0) with image data D1(0) and outputs them.

[0049] Furthermore, at time T0, the shutter speed setting of the imaging unit 202 is 1 / 120 s, and the imaging unit 202 uses a global shutter type image sensor. Time A2_f0 is the time from when the imaging unit 202 receives an external synchronization signal to when exposure begins. Time B2_f0 is the time from when exposure of the imaging unit 202 begins to when it reaches the exposure center of gravity. Times A2_f0 and B2_f0 are values ​​that are uniquely determined by the characteristics of the imaging unit 202 and the shutter speed that is set. The timestamp management unit 221 calculates the timestamp S2(0) by S2(0)=T0+A2_f0+B2_f0.

[0050] Time R1 is the time when the imaging unit 202 outputs the vertical synchronization signal Vsync and starts outputting image data D2(0). At time R1, the timestamp management unit 221 associates timestamp S2(0) with image data D2(0) and outputs them.

[0051] The image capturing unit 201 uses a rolling shutter type image capturing element, and the image capturing unit 202 uses a global shutter type image capturing element. Therefore, even though the shutter speeds of the image capturing units 201 and 202 are the same, it can be seen that the times B1_f0 and B2_f0 are different from each other.

[0052] Next, the effects of this embodiment will be described. As described above, the timestamp management unit 221 receives an external synchronization signal common to the image capture units 201 and 202, and generates timestamps S1(n) and S2(n) according to a set value. The time difference ΔT(n) is the difference between the timestamp S1(n) and the timestamp S2(n), and is the time difference between the exposure centers of the image capture units 201 and 202.

[0053] The CG image generated by the CG generation unit 210 is generated based on the image captured by the imaging unit 202 after image processing. On the other hand, the captured image input to the image synthesis unit 208 is the image captured by the imaging unit 201 after image processing. In other words, if the exposure timings of the imaging units 201 and 202 differ, the image synthesis performed by the image synthesis unit 208 will generate an MR image that looks unnatural. In this embodiment, the HMD 101 calculates the time difference ΔT(n) from the timestamps S1(n) and S2(n) and generates a CG image taking the time difference ΔT(n) into consideration.

[0054] 5, the timestamp value of Frame 0 of the imaging unit 201 is S1(0). The timestamp value of Frame 0 of the imaging unit 202 is S2(0). The time difference ΔT0 is the time difference between the exposure center of gravity of the imaging unit 201 and the exposure center of gravity of the imaging unit 202. For example, the imaging processing unit 206 calculates the time difference ΔT0 using ΔT0=|S1(0)-S2(0)|, and outputs the time difference ΔT0 and the image captured by the imaging unit 202 after image processing to the position and orientation calculation unit 209.

[0055] The position and orientation calculation unit 209 calculates the position and orientation of the HMD 101, taking into account the deviation in the exposure center of gravity between the image capture unit 201 and the image capture unit 202, based on the time difference ΔT0, the image captured by the image capture unit 202 after image processing, and the calculation result of the motion calculation unit 205. As described above, the position and orientation calculation unit 209 calculates the relationship between the world coordinate system representing the real world and the camera coordinate system representing the image observed through the HMD 101, and calculates the position and orientation of the HMD 101 relative to the real world. By using the time difference ΔT and the calculation result of the motion calculation unit 205, the position and orientation calculation unit 209 can correct the time error of the time difference ΔT0, taking into account the movement of the HMD 101. Specifically, the position and orientation calculation unit 209 can know the state of the HMD 101, such as the movement, tilt, rotation, acceleration, and angular velocity, based on the calculation result of the motion calculation unit 205. That is, the position and orientation calculation unit 209 knows the difference between the camera coordinates represented by the image captured by the imaging unit 201 and the camera coordinates represented by the image captured by the imaging unit 202 based on the change in the HMD 101 in the real world and the time difference ΔT0, and takes this into consideration to calculate the position and orientation of the HMD 101. The position and orientation calculation unit 209 outputs information about the position and orientation of the HMD 101 to the CG generation unit 210.

[0056] The CG generation unit 210 reads CG data from the content DB 211 based on the information on the position and orientation of the HMD 101 calculated by the position and orientation calculation unit 209, and generates a CG image that matches the direction of the image captured by the imaging unit 202. Then, the CG generation unit 210 outputs the CG image to the image synthesis unit 208.

[0057] The image synthesis unit 208 receives the processed image captured by the imaging unit 201 from the imaging processing unit 206, and receives the CG image from the CG generation unit 210. The image synthesis unit 208 generates an MR image by synthesizing the processed image captured by the imaging unit 201 with the CG image, and displays the MR image on the display unit 203. By displaying the MR image on the display unit 203, the user of the HMD 101 can visually recognize objects that do not exist in the outside world as CG, and can experience MR.

[0058] As described above, the position and orientation calculation unit 209 calculates the position and orientation of the HMD 101 using the time difference ΔT0 and the calculation result of the motion calculation unit 205. As a result, the position and orientation calculation unit 209 can calculate the position and orientation of the HMD 101 after correcting the time error of the time difference ΔT0 in consideration of the movement of the HMD 101. In this way, even if the exposure centers of gravity of the image capture units 201 and 202 do not match, the image synthesis unit 208 can synthesize the image captured by the image capture unit 201 after image processing with the CG image at an accurate position.

[0059] Based on the characteristics of the multiple image capturing units 201 and 202 and the set shutter speed value, the HMD 101 can accurately calculate the time difference ΔT0 between the exposure centers of gravity of the image capturing units 201 and 202. This allows the HMD 101 to synchronize the multiple image capturing units 201 and 202 and the sensing unit 204 using the synchronization unit 220, thereby achieving a stable frame rate.

[0060] As described above, the imaging device includes the HMD 101 and the image processing device 103. The HMD 101 is a head-mounted device that includes the display unit 203.

[0061] The imaging unit 201 captures an image by exposure in a first exposure period. The first exposure period is, for example, B1_f0+C1_f0 in FIG. 5. The imaging unit 202 captures an image by exposure in a second exposure period. The second exposure period is, for example, B2_f0+C2_f0 in FIG. 5. The timestamp management unit 221 is a timestamp generation unit, and generates a first timestamp indicating the exposure timing within the first exposure period of the imaging unit 201. The first timestamp is, for example, S1(0) in FIG. 5. The timestamp management unit 221 also generates a second timestamp indicating the exposure timing within the second exposure period of the imaging unit 202. The second timestamp is, for example, S2(0) in FIG. 5.

[0062] Specifically, the timestamp management unit 221 generates a first timestamp indicating the center of gravity or the center of the first exposure period of the image capture unit 201. The timestamp management unit 221 also generates a second timestamp indicating the center of gravity or the center of the exposure period of the image capture unit 202.

[0063] The imaging unit 201 captures an image using a first shutter method. The imaging unit 202 captures an image using a second shutter method that is different from the first shutter method. The first shutter method is, for example, a rolling shutter method. The second shutter method is, for example, a global shutter method.

[0064] 5, the first exposure period of the imaging unit 201 is, for example, B1_f1+C1_f1. The second exposure period of the imaging unit 202 is, for example, B2_f1+C2_f1. In this case, the lengths of the first and second exposure periods are different.

[0065] The sensing unit 204 acquires sensor data. For example, the sensing unit 204 acquires sensor data indicating the movement of the HMD 101.

[0066] The image synthesis unit 208, the position and orientation calculation unit 209, and the CG generation unit 210 are image processing units that perform image processing based on the image captured by the imaging unit 201, the image captured by the imaging unit 202, the first timestamp, and the second timestamp.

[0067] The position and orientation calculation unit 209 and the CG generation unit 210 generate a computer graphics image based on the image captured by the imaging unit 202, the first timestamp, the second timestamp, and the sensor data. The image synthesis unit 208 performs image synthesis based on the image captured by the imaging unit 201 and the computer graphics image.

[0068] The imaging unit 201 may perform exposure in the first exposure period based on the vertical synchronization signal Vsync. In this case, the synchronization unit 220 generates an external synchronization signal based on the vertical synchronization signal Vsync of the imaging unit 201. The imaging unit 202 performs exposure in the second exposure period based on the external synchronization signal generated by the synchronization unit 220.

[0069] As described above, according to this embodiment, the HMD 101 and the image processing device 103 can synchronize the image capturing unit 201, the image capturing unit 202, and the sensing unit 204, and can achieve a stable frame rate.

[0070] (Second embodiment) The second embodiment will be described below with reference to Fig. 6. Fig. 6 is a block diagram showing an example of the functional configuration of an HMD 101 and an image processing device 103 according to the second embodiment. The HMD 101 in Fig. 6 is obtained by adding an image deformation unit 601 to the HMD 101 in Fig. 2. The image processing device 103 in Fig. 6 is the same as the image processing device 103 in Fig. 2. Differences between the second embodiment and the first embodiment will be described below.

[0071] The CG generation unit 210 may experience fluctuations in throughput and increased latency depending on the rendering load of the CG data read from the content DB. In this case, the image synthesis unit 208 may experience a delay in inputting the CG image from the CG generation unit 210, which may result in the position of the CG image not matching the position of the captured image, resulting in an unnatural MR image.

[0072] Therefore, the image deformation unit 601 deforms the CG image generated by the CG generation unit 210 based on the calculation result of the movement calculation unit 205. Since the image captured by the imaging unit 201 changes in accordance with the movement of the HMD 101, the image deformation unit 601 deforms the CG image so as to correct the delay of the CG image. The image deformation method is, for example, an image deformation method such as homography transformation. The image deformation unit 601 outputs the deformed CG image to the image synthesis unit 208.

[0073] The image synthesis unit 208 receives the processed image captured by the imaging unit 201 from the imaging processing unit 206, and receives the deformed CG image from the image transformation unit 601. The image synthesis unit 208 generates an MR image by combining the processed image captured by the imaging unit 201 with the deformed CG image, and presents (displays) the MR image on the display unit 203.

[0074] As a result, even if the latency increases due to the generation of CG images by the CG generation unit 210, the HMD 101 can display MR images on the display unit 203 that do not cause discomfort to the user of the HMD 101.

[0075] The movement calculation unit 205 calculates the movement, tilt, rotation, etc. of the HMD 101 based on the sensor data output from the sensing unit 204. The image deformation unit 601 deforms the CG image based on the calculation results of the movement calculation unit 205. The image synthesis unit 208 synthesizes the image captured by the imaging unit 201 after image processing with the deformed CG image. Therefore, the HMD 101 needs to accurately grasp the data acquisition timing of the imaging units 201 and 202 and the sensing unit 204. Therefore, the timestamp management unit 221 associates a timestamp with the sensor data output by the sensing unit 204 and outputs it. Details of this will be explained below.

[0076] Fig. 7 is a timing chart showing an example of a control method for the HMD 101 according to the second embodiment. The operation of the second embodiment will be described using the timing chart of Fig. 7. The timing chart of the image capturing units 201 and 202 in Fig. 7 is the same as the timing chart of the image capturing units 201 and 202 in Fig. 5. The timing chart of the sensing unit 204 in Fig. 7 will be mainly described below.

[0077] Time E0 is the time when the synchronizer 220 outputs an external synchronization signal to the imaging units 201 and 202, the sensing unit 204, and the timestamp manager 221. A common external synchronization signal is input to the imaging units 201, 202, the sensing unit 204, and the timestamp manager 221. The timestamp manager 221 holds the value of the time counter Tcnt at time E0 as the external synchronization time T0.

[0078] The sensing unit 204 samples sensor data at the sampling timing based on the settings. The time from time E0 to the sampling timing is a value uniquely determined by the settings of the sensing unit 204, and is A3_f0+B3_f0. The timestamp management unit 221 calculates the timestamp S3(0) by S3(0)=T0+A3_f0+B3_f0. After a time C3_f0 from the sampling timing, the sensing unit 204 starts outputting the sampled sensor data D3(0). The timestamp management unit 221 associates the timestamp S3(0) with the sensor data D3(0) and outputs them to the operation calculation unit 205.

[0079] The operation calculation unit 205 calculates the deviation amounts ΔT0_13 and ΔT0_23 of the timing of acquiring sensor data based on the timestamp S1(0) of the imaging unit 201, the timestamp S2(0) of the imaging unit 202, and the timestamp S3(0) of the sensing unit 204.

[0080] For example, the motion calculation unit 205 calculates the amount of deviation ΔT0_13 between the exposure center of the imaging unit 201 and the data acquisition timing of the sensing unit 204 using ΔT0_13 = |S1(0) - S3(0)|. The motion calculation unit 205 also calculates the amount of deviation ΔT0_23 between the exposure center of the imaging unit 202 and the data acquisition timing of the sensing unit 204 using ΔT0_23 = |S2(0) - S3(0)|. The motion calculation unit 205 outputs the amounts of deviation ΔT0_13 and ΔT0_23 to the image transformation unit 601.

[0081] The image deformation unit 601 deforms the CG image generated by the CG generation unit 210 based on the deviation amounts ΔT0_13 and ΔT0_23 so as to correct the delay of the CG image relative to the captured image.

[0082] The image synthesis unit 208 receives the processed image captured by the imaging unit 201 from the imaging processing unit 206, and receives the deformed CG image from the image deformation unit 601. The image synthesis unit 208 generates an MR image by combining the processed image captured by the imaging unit 201 with the deformed CG image, and displays the MR image on the display unit 203. This allows the display unit 203 to present (display) an image that is less uncomfortable for the user of the HMD 101.

[0083] As described above, the sensing unit 204 acquires sensor data by sampling at the sampling time. The timestamp management unit 221 generates a third timestamp indicating the sampling time of the sensing unit 204. The third timestamp is, for example, S3(0) in FIG. 7.

[0084] The position and orientation calculation unit 209 and the CG generation unit 210 generate a computer graphics image based on the image captured by the imaging unit 202, the first timestamp, the second timestamp, and the sensor data. The first timestamp is, for example, S1(0) in FIG. 7. The second timestamp is, for example, S2(0) in FIG. 7.

[0085] The image transformation unit 601 transforms the computer graphics image based on the first timestamp, the second timestamp, and the third timestamp. The image synthesis unit 208 performs image synthesis based on the image captured by the imaging unit 201 and the computer graphics image transformed by the image transformation unit 601.

[0086] As described above, according to this embodiment, the HMD 101 and the image processing device 103 can synchronize the image capturing unit 201, the image capturing unit 202, and the sensing unit 204, and can achieve a stable frame rate.

[0087] (Other embodiments) The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0088] Although the embodiments have been described above, the present disclosure is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

[0089] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) a first imaging unit that captures an image by exposure during a first exposure period; a second imaging unit that captures an image by exposure during a second exposure period; a first timestamp generation unit that generates a first timestamp indicating an exposure timing within the first exposure period of the first imaging unit; a second timestamp generation unit that generates a second timestamp indicating an exposure timing within the second exposure period of the second imaging unit; An imaging device comprising: (Configuration 2) the first timestamp generation unit generates a first timestamp indicating a center of gravity or a center of the first exposure period of the first imaging unit; 2. The imaging device according to claim 1, wherein the second timestamp generation unit generates a second timestamp that indicates the center of gravity or the center of the second exposure period of the second imaging unit. (Configuration 3) the first imaging unit captures an image using a first shutter method, 3. The imaging device according to configuration 1 or 2, wherein the second imaging section captures images using a second shutter method different from the first shutter method. (Configuration 4) the first shutter method is a rolling shutter method, 4. The imaging device according to configuration 3, wherein the second shutter method is a global shutter method. (Configuration 5) 5. The imaging device according to any one of configurations 1 to 4, wherein the length of the first exposure period and the length of the second exposure period are different. (Configuration 6) 6. The imaging device according to any one of configurations 1 to 5, further comprising a sensing section that acquires sensor data. (Configuration 7) 7. The imaging device according to configuration 6, wherein the sensing unit acquires sensor data indicating a movement of the imaging device. (Configuration 8) The imaging device according to any one of configurations 1 to 7, further comprising an image processing unit that performs image processing based on an image captured by the first imaging unit, an image captured by the second imaging unit, the first timestamp, and the second timestamp. (Configuration 9) a generation unit that generates a computer graphics image based on the image captured by the second imaging unit, the first timestamp, the second timestamp, and the sensor data; 8. The imaging device according to configuration 6 or 7, further comprising an image synthesis unit that performs image synthesis based on the image captured by the first imaging unit and the computer graphics image. (Configuration 10) the sensing unit acquires sensor data by sampling at a sampling time; 8. The imaging device according to configuration 6 or 7, further comprising a third timestamp generation unit that generates a third timestamp that indicates the sampling time of the sensing unit. (Configuration 11) a generation unit that generates a computer graphics image based on the image captured by the second imaging unit, the first timestamp, the second timestamp, and the sensor data; an image transformation unit that transforms the computer graphics image based on the first timestamp, the second timestamp, and the third timestamp; The imaging device according to configuration 10, further comprising an image synthesis unit that performs image synthesis based on the image captured by the first imaging unit and the computer graphics image transformed by the image transformation unit. (Configuration 12) the first imaging unit performs exposure during the first exposure period based on a first synchronization signal; the second imaging unit performs exposure during the second exposure period based on a second synchronization signal; 12. The imaging device according to any one of configurations 1 to 11, further comprising a synchronization section that generates the second synchronization signal based on the first synchronization signal. (Configuration 13) 13. The imaging device according to any one of configurations 1 to 12, wherein the imaging device is a head-mounted device having a display unit. (Configuration 14) an imaging unit that captures an image by exposure during an exposure period; a sensing unit that acquires sensor data by sampling at a sampling time; a first timestamp generation unit that generates a first timestamp indicating an exposure timing within the exposure period of the imaging unit; a second timestamp generation unit that generates a second timestamp indicating the sampling time of the sensing unit; An imaging device comprising: (Configuration 15) 15. The imaging device of claim 14, wherein the first timestamp generation unit generates a first timestamp that indicates a center of gravity or a center of the exposure period of the imaging unit. (Configuration 16) 16. The imaging device according to configuration 14 or 15, wherein the sensing unit acquires sensor data indicating a movement of the imaging device. (Configuration 17) 17. The imaging device according to any one of configurations 14 to 16, wherein the imaging device is a head-mounted device having a display unit. (Method 1) a first imaging unit that captures an image by exposure during a first exposure period; a second imaging unit that performs imaging by exposure during a second exposure period, a first timestamp generating step of generating a first timestamp indicating an exposure timing within the first exposure period of the first imaging unit; a second timestamp generating step of generating a second timestamp indicating an exposure timing within the second exposure period of the second imaging unit; 10. A method for controlling an imaging device, comprising: (Method 2) an imaging unit that captures an image by exposure during an exposure period; a sensing unit that acquires sensor data by sampling at a sampling time, a first timestamp generating step of generating a first timestamp indicating an exposure timing within the exposure period of the imaging unit; a second timestamp generating step of generating a second timestamp indicating the sampling time of the sensing unit; 10. A method for controlling an imaging device, comprising: [Explanation of symbols]

[0090] 101 HMD, 102 display device, 103 image processing device, 104 operation unit, 201 imaging unit, 202 imaging unit, 203 display unit, 204 sensing unit, 205 movement calculation unit, 206 imaging processing unit, 207 calculation unit, 208 image synthesis unit, 209 position and orientation calculation unit, 210 CG generation unit, 211 content DB, 220 synchronization unit, 221 timestamp management unit, 601 image transformation unit

Claims

1. a first imaging unit that captures an image by exposure during a first exposure period; a second imaging unit that captures an image by exposure during a second exposure period; a first timestamp generation unit that generates a first timestamp indicating an exposure timing within the first exposure period of the first imaging unit; a second timestamp generation unit that generates a second timestamp indicating an exposure timing within the second exposure period of the second imaging unit; An imaging device comprising:

2. the first timestamp generation unit generates a first timestamp indicating a center of gravity or a center of the first exposure period of the first imaging unit; 2. The imaging device according to claim 1, wherein the second timestamp generation unit generates a second timestamp that indicates a center of gravity or a center of the second exposure period of the second imaging unit.

3. the first imaging unit captures an image using a first shutter method, 2. The imaging device according to claim 1, wherein the second imaging unit captures images using a second shutter method different from the first shutter method.

4. the first shutter method is a rolling shutter method, 4. The imaging device according to claim 3, wherein the second shutter method is a global shutter method.

5. 2. The imaging device according to claim 1, wherein the length of the first exposure period is different from the length of the second exposure period.

6. 2. The imaging device according to claim 1, further comprising a sensing unit that acquires sensor data.

7. The imaging device according to claim 6 , wherein the sensing unit acquires sensor data indicating a movement of the imaging device.

8. 2. The imaging device according to claim 1, further comprising an image processing unit that performs image processing based on the image captured by the first imaging unit, the image captured by the second imaging unit, the first timestamp, and the second timestamp.

9. a generation unit that generates a computer graphics image based on the image captured by the second imaging unit, the first timestamp, the second timestamp, and the sensor data; 7. The imaging device according to claim 6, further comprising an image synthesis unit that synthesizes an image based on the image captured by the first imaging unit and the computer graphics image.

10. the sensing unit acquires sensor data by sampling at a sampling time; 7. The imaging device according to claim 6, further comprising a third timestamp generation unit that generates a third timestamp that indicates the sampling time of the sensing unit.

11. a generation unit that generates a computer graphics image based on the image captured by the second imaging unit, the first timestamp, the second timestamp, and the sensor data; an image transformation unit that transforms the computer graphics image based on the first timestamp, the second timestamp, and the third timestamp; The imaging device according to claim 10 , further comprising an image synthesis unit that performs image synthesis based on the image captured by the first imaging unit and the computer graphics image transformed by the image transformation unit.

12. the first imaging unit performs exposure during the first exposure period based on a first synchronization signal; the second imaging unit performs exposure during the second exposure period based on a second synchronization signal; 2. The imaging device according to claim 1, further comprising a synchronization unit that generates the second synchronization signal based on the first synchronization signal.

13. 2. The imaging device according to claim 1, wherein the imaging device is a head-mounted device having a display unit.

14. an imaging unit that captures an image by exposure during an exposure period; a sensing unit that acquires sensor data by sampling at a sampling time; a first timestamp generation unit that generates a first timestamp indicating an exposure timing within the exposure period of the imaging unit; a second timestamp generation unit that generates a second timestamp indicating the sampling time of the sensing unit; An imaging device comprising:

15. 15. The imaging device according to claim 14, wherein the first timestamp generation unit generates a first timestamp that indicates a center of gravity or a center of the exposure period of the imaging unit.

16. The imaging device according to claim 14 , wherein the sensing unit acquires sensor data indicating a movement of the imaging device.

17. 15. The imaging device according to claim 14, wherein the imaging device is a head-mounted device having a display unit.

18. a first imaging unit that captures an image by exposure during a first exposure period; a second imaging unit that captures an image by exposure during a second exposure period, a first timestamp generating step of generating a first timestamp indicating an exposure timing within the first exposure period of the first imaging unit; a second timestamp generating step of generating a second timestamp indicating an exposure timing within the second exposure period of the second imaging unit; 10. A method for controlling an imaging device, comprising:

19. an imaging unit that captures an image by exposure during an exposure period; a sensing unit that acquires sensor data by sampling at a sampling time, a first timestamp generating step of generating a first timestamp indicating an exposure timing within the exposure period of the imaging unit; a second timestamp generating step of generating a second timestamp indicating the sampling time of the sensing unit; 10. A method for controlling an imaging device, comprising:

Images