Electronic devices, methods for controlling electronic devices, programs, and storage media

Abstract

To enable stereoscopic display suitable for a user.SOLUTION: An electronic apparatus of the present invention has: first acquisition means that acquires a first image picked up through a first optical system of an imaging apparatus, and a second image picked up through a second optical system of the imaging apparatus and having parallax with respect to the first image; second acquisition means that acquires distance information indicating the distance from the imaging apparatus to a subject; and control means that controls the parallax between the first image and the second image through mechanical adjustment to adjust an angle between an optical axis of the first optical system and an optical axis of the second optical system when the speed of change of the distance indicated by the distance information is smaller than a threshold, and through soft adjustment to perform image processing on the first image and the second image when the speed of change of the distance is larger than the threshold.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to an electronic device, a control method for an electronic device, a program, and a storage medium.

Background Art

[0002] In recent years, devices capable of capturing and displaying images in real time to expand human visual information have become widespread. Using a head-mounted display (HMD) or the like for experiencing virtual reality (AR; Augmented Reality), a user can experience as if virtual objects with size and sense of distance are actually present in the real space.

[0003] Humans view an object from different viewpoints of the right eye and the left eye, and perceive a sense of depth from the difference in the appearance of the object seen with the right eye and the appearance of the object seen with the left eye (binocular parallax), thereby stereoscopically viewing the object. Therefore, when imaging a subject and displaying it on an HMD or the like, by performing stereoscopic display with a parallax corresponding to the binocular parallax of the user, the user can suitably stereoscopically view the displayed subject. The parallax for stereoscopic display varies depending on the subject distance, which is the distance from the HMD to the subject, and becomes larger as the subject distance becomes shorter. Therefore, it is required to adjust the parallax of the stereoscopic display according to the subject distance. In the technique disclosed in Patent Document 1, the parallax of the stereoscopic display is adjusted by adjusting the angle formed by the optical axis of the imaging unit or by performing a process of converting video data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technology disclosed in Patent Document 1, depending on the amount of parallax adjustment for stereoscopic display, there may be cases where the parallax adjustment is not fast enough. If the parallax adjustment is not fast enough and the parallax of the object in the stereoscopic display deviates from the parallax when the user directly views the object (for example, with the naked eye, without using an HMD), the user may have difficulty viewing the displayed object in 3D and may experience discomfort or a sense of unease.

[0006] Therefore, the present invention aims to enable a three-dimensional display suitable for the user. [Means for solving the problem]

[0007] A first aspect of the present invention is an electronic device comprising: a first acquisition means for acquiring a first image captured through a first optical system of an imaging device and a second image captured through a second optical system of the imaging device, which has parallax with respect to the first image; a second acquisition means for acquiring distance information indicating the distance from the imaging device to a subject; and a control means for controlling the parallax between a first image and a second image by mechanical adjustment, which adjusts the angle between the optical axis of the first optical system and the optical axis of the second optical system when the rate of change of the distance indicated by the distance information is less than a threshold, and by soft adjustment, which performs image processing on the first image and the second image when the rate of change of the distance is greater than the threshold.

[0008] A second aspect of the present invention provides an acquisition means for acquiring a first image captured through a first optical system of an imaging device and a second image having parallax to the first image, captured through a second optical system of the imaging device, and when the distance from the imaging device to the subject changes at a first speed, the angle between the optical axis of the first optical system and the optical axis of the second optical system is adjusted so that the distance changes at a speed faster than the first speed. First The electronic device is characterized by having a control means that controls the parallax between the first image and the second image by performing image processing on the first image and the second image when the parallax changes at a speed of 2.

[0009] A third aspect of the present invention is a control method for electronic equipment, comprising: a first acquisition step of acquiring a first image captured through a first optical system of an imaging device and a second image captured through a second optical system of the imaging device, which has parallax with respect to the first image; a second acquisition step of acquiring distance information indicating the distance from the imaging device to a subject; and a control step of controlling the parallax between a first image and a second image by mechanical adjustment, which adjusts the angle between the optical axis of the first optical system and the optical axis of the second optical system when the rate of change of the distance indicated by the distance information is less than a threshold, and by soft adjustment, which performs image processing on the first image and the second image when the rate of change of the distance is greater than the threshold.

[0010] A fourth aspect of the present invention is a program for causing a computer to function as one of the means of the electronic devices described above.

[0011] A fifth aspect of the present invention is a computer-readable storage medium that stores a program for causing the computer to function as one of the electronic devices described above. [Effects of the Invention]

[0012] According to the present invention, a three-dimensional display suitable for the user becomes possible. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram of a head-mounted display according to the first embodiment. [Figure 2] This is a diagram showing an example of how to wear a head-mounted display according to the first embodiment. [Figure 3] This figure illustrates the difference in convergence angles according to the first embodiment. [Figure 4] This is a diagram illustrating the mechanical adjustment according to the first embodiment. [Figure 5] This is a diagram illustrating the soft adjustment according to the first embodiment. [Figure 6] It is a flowchart showing the parallax adjustment process according to the first embodiment. [Figure 7] It is a diagram showing the mechanical adjustment amount according to the first embodiment. [Figure 8] It is a diagram showing the shift amount of the image according to the first embodiment. [Figure 9] It is a flowchart showing the determination process of the adjustment method according to the first embodiment. [Figure 10] It is a flowchart showing the focus adjustment process according to the second embodiment.

Mode for Carrying Out the Invention

[0014] (First Embodiment) Hereinafter, the first embodiment of the present invention will be described. In the first embodiment, an example in which an electronic device to which the present invention is applied is a wearable device such as a head-mounted display (HMD) will be described.

[0015] <Configuration> FIG. 1 is a block diagram showing a configuration example of the HMD 100. The HMD 100 includes an imaging display unit 250R for the right eye and an imaging display unit 250L for the left eye. First, the imaging display units 250R and 250L will be described. The imaging display unit 250R includes an image imaging unit 200R for the right eye, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, an eyepiece 116, and the like. Similarly, the imaging display unit 250L includes an image imaging unit 200L for the left eye, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, an eyepiece 116, and the like. Each of the image imaging unit 200R and the image imaging unit 200L has an aperture 201, a lens 202, an aperture drive circuit 203, an AF (auto focus) drive circuit 204, a lens system control circuit 205, a shutter 210, and an imaging unit 211.

[0016] The lens 202 is composed of a plurality of lenses, but in FIG. 1, it is simply shown as only one lens for simplicity. The system control unit 218 communicates with the lens system control circuit 205 and controls the aperture 1 via the aperture drive circuit 203. Also, the system control unit 218 focuses on the target by displacing the focus lens included in the lens 202 via the AF drive circuit 204.

[0017] The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging unit​​​​​​​The memory control unit 213 controls the transmission and reception of data between the A / D converter 212, the image processing unit 214, the memory 215, and the D / A converter 216. Image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213, or via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, as well as image data for display on the EVF 217. The memory 215 has sufficient storage capacity to store a predetermined number of still images, a predetermined amount of video footage, and audio. The memory 215 also serves as a memory for image display (video memory).

[0020] The D / A converter 216 converts the image display data stored in the memory 215 into an analog signal and supplies it to the EVF 217. The display image data written to the memory 215 is displayed on the EVF 217 via the D / A converter 216. The EVF 217 displays according to the analog signal from the D / A converter 216. The EVF 217 is a display such as an LCD or an organic EL. By converting the digital signal that has been A / D converted by the A / D converter 212 and stored in the memory 215 into an analog signal in the D / A converter 216 and sequentially transferring it to the EVF 217 for display, live view display (through image display) can be performed.

[0021] The system control unit 218 is a control unit consisting of at least one processor or circuit. That is, the system control unit 218 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 218 controls the entire HMD 100. The system control unit 218 implements the processes described later by executing the program recorded in the non-volatile memory 220. The system control unit 218 also controls the display by controlling the memory 215, D / A converter 216, EVF 217, etc. conduct.

[0022] The HMD100 also includes a system memory 219, a non-volatile memory 220, a system timer 221, a communication unit 222, an attitude detection unit 223, and an eyepiece detection unit 118.

[0023] The system memory 219 is, for example, RAM, and the system control unit 218 loads constants, variables, and programs read from the non-volatile memory 220 into the system memory 219 for the operation of the system control unit 218.

[0024] The non-volatile memory 220 is an electrically erasable and recordable memory. For example, an EEPROM is used for the non-volatile memory 220, and constants and programs for the operation of the system control unit 218 are stored there. The program referred to here is a program for executing the various flowcharts described later.

[0025] The system timer 221 is a timekeeping unit that measures the time used for various controls and the time of the built-in clock.

[0026] The communication unit 222 transmits and receives video and audio signals to and from external devices connected wirelessly or via wired cables. The communication unit 222 can also connect to wireless LAN (Local Area Network) and the internet. Furthermore, the communication unit 222 can communicate with external devices using Bluetooth® and Bluetooth Low Energy. The communication unit 222 can transmit images captured by the imaging unit 211 (including live images) and images recorded on the recording medium 228, and can receive image data and other various information from external devices.

[0027] The attitude detection unit 223 detects the attitude of the HMD 100 relative to the direction of gravity. Based on the attitude detected by the attitude detection unit 223, it is possible to determine whether the image captured by the imaging unit 211 was taken with the HMD 100 held horizontally or vertically. The system control unit 218 can add orientation information corresponding to the attitude detected by the attitude detection unit 223 to the image file of the image captured by the imaging unit 211, or rotate the image before recording. An acceleration sensor or a gyroscope sensor can be used as the attitude detection unit 223. The attitude detection unit 223 can detect the movement of the HMD 100 (pan, tilt, roll, whether it is stationary or not, etc.).

[0028] The eyepiece detection unit 118 is an eyepiece detection sensor that detects the approach (eye-sight) and retraction (eye-away) of an eye (object) to the eyepiece unit 116. The eyepiece detection unit 118 may also be configured to detect the approach of some object to the eyepiece unit 116 using, for example, an infrared proximity sensor. When an object approaches, infrared light emitted from the light emitter of the eyepiece detection unit 118 is reflected by the object and received by the light receiver of the infrared proximity sensor. The distance from the eyepiece unit 116 to the object can be determined by the amount of infrared light received. In this way, the eyepiece detection unit 118 performs eyepiece detection to detect the proximity distance of an object to the eyepiece unit 116. The eyepiece detection unit 118 detects that an eye has been placed on the eyepiece when an object that approaches within a predetermined distance from the eyepiece unit 116 is detected from a non-eyepiece state (non-approach state). On the other hand, the eyepiece detection unit 118 detects that the eye has been removed when the object that was detected as approaching moves more than a predetermined distance away from the eyepiece state (close state). The threshold for detecting eyepiece contact and the threshold for detecting eye removal may be different, for example, by providing hysteresis. Furthermore, after detecting eyepiece contact, the eyepiece detection unit 118 determines that the eyepiece is in a state until eye removal is detected. After detecting eye removal, it determines that the eyepiece is not in a state until eyepiece contact is detected again. The system control unit 218 can switch the display (display state) / hidden (hidden state) of the EVF 217 according to the state detected by the eyepiece detection unit 118. Specifically, when the system control unit 218 detects eyepiece contact, it displays the EVF 217, and when eyepiece contact is detected, it displays the EVF 217. If an eye is detected, the EVF217 is turned off. Note that the eyepiece detection unit 118 is not limited to an infrared proximity sensor; other sensors that can detect a state that can be considered as an eyepiece may be used.

[0029] The HMD100 also includes an external viewfinder display unit 107, an external viewfinder display drive circuit 224, a power control unit 225, a power supply unit 226, a recording medium interface 227, a recording medium 228, and an operation unit 229.

[0030] The external viewfinder display unit 107 displays various settings of the HMD100, such as shutter speed and aperture, via the external viewfinder display drive circuit 224.

[0031] The power control unit 225 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which blocks are energized, and detects whether a battery is installed, the type of battery, and the remaining battery level. The power control unit 225 also controls the DC-DC converter based on the detection results and instructions from the system control unit 218, supplying the necessary voltage to each part, including the recording medium 228, for the required period. The power supply unit 226 consists of primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, and lithium-ion batteries, and an AC adapter.

[0032] The recording medium I / F 227 is an interface to a recording medium 228, such as a memory card or hard disk. The recording medium 228 is a recording medium (storage medium) such as a memory card for recording captured images, and is composed of semiconductor memory or magnetic disks. The recording medium 228 may be removable or built-in.

[0033] The operation unit 229 is an input unit that receives user input (user operation) and is used to input various instructions to the system control unit 218. The operation unit 229 includes a shutter button 101, a power switch 102, a mode selector switch 103, and other operating components 230. The other operating components 230 include an electronic dial, directional keys, and menu buttons.

[0034] The shutter button 101 is equipped with a first shutter switch 121 and a second shutter switch 122. The first shutter switch 121 turns on during the operation of the shutter button 101, so-called half-press (shooting preparation instruction), and generates a first shutter switch signal SW1. The system control unit 218 starts shooting preparation processing such as AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and EF (flash pre-flash) processing in response to the first shutter switch signal SW1. The second shutter switch 122 turns on when the operation of the shutter button 101 is completed, so-called full-press (shooting instruction), and generates a second shutter switch signal SW2. The system control unit 218 starts a series of shooting processes from reading the signal from the imaging unit 211 to writing the captured image as an image file to the recording medium 228 in response to the second shutter switch signal SW2.

[0035] The mode switch 103 is an operating component for switching the operating mode of the system control unit 218 to one of the following: shooting / display mode, playback mode, AR display mode, etc. The user can directly switch to one of the above-mentioned shooting modes using the mode switch 103. Alternatively, the user can switch to the operating mode list screen using the mode switch 103, and then selectively switch to one of the displayed modes using the operation unit 229.

[0036] Figure 2 shows an example of the HMD100 being worn by a user. Of the HMD100 shown in Figure 2, components identical to those described in Figure 1 are denoted by the same reference numerals, and their explanations are omitted as appropriate.

[0037] The rotation adjustment unit 231R adjusts the rotation angle of the image acquisition unit 200R around the Z axis (Yaw direction; parallax direction). The rotation adjustment unit 231L adjusts the rotation angle of the image acquisition unit 200L around the Z axis (Yaw direction; parallax direction).

[0038] The optical axis spacing 232 is the distance (inter-optical axis distance) between the optical axis of the image acquisition unit 200R (image display unit 250R) and the optical axis of the image acquisition unit 200L (image display unit 250L). The HMD100 is configured to allow adjustment of the optical axis spacing 232 by an adjustment mechanism (not shown).

[0039] The interocular distance 234 is the distance between the user's right and left eyes. The EVF interval 233 is the distance between the EVF 217R for the right eye and the EVF 217L for the left eye. Since the interocular distance 234 varies from person to person, it is preferable that the EVF interval 233 of the HMD 100 can also be adjusted to match the interocular distance 234. The HMD 100 is configured to allow the EVF interval 233 to be changed (adjusted) by an adjustment mechanism (not shown).

[0040] <Differences in convergence angles> Figures 3(a) and 3(b) illustrate the differences (changes) in the convergence angle due to differences in the distance from the eye to an object, and the differences in the convergence angle due to individual differences. Figure 3(a) shows the difference in the convergence angle due to differences in the distance from the eye to an object for a person whose interocular distance 234a is narrower than the interocular distance 234b in Figure 3(b). Figure 3(b) shows the difference in the convergence angle due to differences in the distance from the eye to an object for a person whose interocular distance 234b is wider than the interocular distance 234a in Figure 3(a).

[0041] Convergence angles 301 and 303 are the convergence angles when a person is looking at an object that is close to them. Convergence angles 302 and 304 are the convergence angles when a person is looking at an object that is far away. Comparing convergence angles 301 and 302 in Figure 3(a), it can be seen that the convergence angle is larger when looking at an object that is close than when looking at an object that is far away. Humans perceive the difference in convergence angles due to changes in the distance from the eye to an object as a sense of distance (depth).

[0042] As shown in Figures 3(a) and 3(b), the convergence angle is determined by the interocular distance 234 and the distance from the eye to the object. Therefore, if the interocular distance 234 is different, the convergence angle will also be different. There are individual differences in interocular distance 234, and the interocular distance 234 for adults is approximately 5 cm to 7 cm. For this reason, even with the same 3D image (stereoscopic image), the perceived distance may differ from user to user.

[0043] In AR (Augmented Reality) video experiences, the sense of distance from the user to virtual objects is sometimes important. For example, when using AR for sports or sightseeing experiences, accurately reproducing the desired sense of distance to objects allows users to enjoy a more realistic and less jarring video experience. Therefore, the HMD100 is required to provide 3D images that reproduce the desired sense of distance (3D images that allow the user to perceive a sense of distance close to the desired sense). To provide 3D images that reproduce the desired sense of distance, it is desirable to adjust the parallax between the right-eye image and the left-eye image after aligning the optical axis spacing 232 with the interpupillary distance 234 shown in Figure 2. The optical axis spacing 232 may be manually adjusted to match the user's interpupillary distance 234, or it may be automatically adjusted by acquiring information on the gaze positions of the left and right eyes and calculating the interpupillary distance 234 from the gaze position information. The right-eye image is the image captured by the right-eye image acquisition unit 200R and displayed in the EVF217R. Furthermore, the left-eye image is the image captured by the left-eye image acquisition unit 200L and displayed on the EVF217L.

[0044] <How to adjust image parallax> In the following explanation, the difference between how an object appears to the right eye and how it appears to the left eye is defined as "parallax (binocular parallax)." Therefore, the difference between how an object appears to the right eye in the image and how it appears to the left eye in the image is defined as the parallax between the right eye image and the left eye image, and may simply be referred to as "image parallax." There are two methods for adjusting image parallax: mechanical adjustment of the image acquisition units 200R and 200L (left and right imaging optical systems) and soft adjustment through image processing. The HMD100 adjusts image parallax through mechanical or soft adjustment and displays a stereoscopic image with a parallax equivalent to the user's binocular parallax.

[0045] Mechanical adjustment is a method of adjusting image parallax by adjusting the rotation angle of the left and right imaging optical systems (imaging devices) according to the distance (subject distance) from the left and right imaging optical systems to the subject the user is focusing on. Here, the larger the convergence angle when the user views an object (subject), the greater the user's binocular parallax, and the larger the angle formed by the optical axes of the left and right imaging optical systems (convergence angle of the imaging optical system), the greater the image parallax. Therefore, in mechanical adjustment, the rotation angle of the left and right imaging optical systems is adjusted so that the convergence angle of the imaging optical system matches the convergence angle when the user views the subject directly (without using an HMD, for example, with the naked eye). In mechanical adjustment, the optical axis of the left eye optical system is made parallel to the line of sight of the left eye when the user is directly viewing the subject, and the optical axis of the right eye optical system is made parallel to the line of sight of the right eye when the user is directly viewing the subject. Mechanical adjustment adjusts the difference between the image acquired by the image acquisition unit 200R and the image acquired by the image acquisition unit 200L.

[0046] Figures 4(a) and 4(b) illustrate mechanical adjustment. Figure 4(a) shows an example where the rotation adjustment units 231R and 231L are adjusted when the subject 235 moves from infinity to near, thereby adjusting the convergence angle of the imaging optical system. In Figure 4(b), images 236R and 236L are images taken when the subject 235 is at infinity, and images 237R and 237L are images taken when the subject 235 is at near. Images 236R and 237R are examples of images for the right eye, and images 236L and 237L are examples of images for the left eye.

[0047] Soft adjustment is a method of adjusting image parallax by performing image processing that changes the display position of the image acquired through the left and right imaging optical systems on a display device (e.g., an EVF) from the position without soft adjustment. In soft adjustment, for example, when the distance to the subject changes, the display position of the image is adjusted, shifting the position of the subject on the image horizontally (parallax direction) as if the left and right imaging optical systems had been moved to match the subject of interest. When the subject approaches the user and increases the image parallax, the horizontal distance between the subject position in the right-eye image and the subject position in the left-eye image is increased. On the other hand, when the subject moves away from the user and decreases the image parallax, the horizontal distance between the subject position in the right-eye image and the subject position in the left-eye image is decreased. Note that the image acquisition units 200R and 200L may capture an area wider than the display range shown on the EVF 217R and 217L, or they may capture an area about the same as the display range. When performing soft adjustments, it is acceptable that some parts of the captured image may not be visible depending on the user's perception of the distance at which they are viewing the subject displayed on the EVF217R or 217L.

[0048] Figures 5(a) and 5(b) illustrate soft adjustment. Figure 5(a) shows an example where soft adjustment is performed without mechanical adjustment when the subject 235 moves from infinity to near. In Figure 5(b), images 238R and 238L are images taken when the subject 235 is at infinity. Images 239R and 239L are images taken when the subject 235 moves from infinity to near without mechanical or soft adjustment. Images 240R and 240L are images softly adjusted to increase the horizontal distance between the subject position in the right-eye image and the subject position in the left-eye image. Perform soft adjustment. The parallax of the image can then be adjusted so that the subject is displayed in the same position as when mechanical adjustment is performed. In addition, with software adjustment, image processing (e.g., projection transformation) that geometrically deforms the image may be performed to achieve a view that is closer to the user's direct view (a view without an HMD, e.g., a view with the naked eye).

[0049] Distance information indicating the distance to the subject can be obtained in various ways. For example, distance information may be obtained using methods such as image plane phase difference imaging or active distance measurement methods such as lasers, as exemplified by LIDAR. The method of obtaining distance information is not particularly limited.

[0050] <Switching the parallax adjustment method> The amount of image parallax adjustment per predetermined time varies depending on the subject distance, becoming larger as the subject distance decreases. Also, the amount of image parallax adjustment per predetermined time increases as the rate of change of the subject distance increases. If the amount of image parallax adjustment per predetermined time is greater than a predetermined amount, mechanical adjustment cannot keep up with the image parallax adjustment, so the system control unit 218 switches to soft adjustment to adjust the image parallax. In the first embodiment, the system control unit 218 mainly adjusts the image parallax using mechanical adjustment according to the subject distance, and switches to soft adjustment to adjust the image parallax when the rate of change of the subject distance is greater than or equal to a threshold.

[0051] <Parallax adjustment process> Figure 6 is a flowchart showing an example of image parallax adjustment processing.

[0052] In step S601, the system control unit 218 adjusts the EVF interval 233 to match the interocular distance 234 shown in Figure 2.

[0053] In step S602, the system control unit 218 adjusts the optical axis spacing 232 to match the interocular spacing 234 shown in Figure 2. In steps S601 and S602, the system control unit 218 may use, for example, the average value for adult men and women as the interocular spacing 234. Alternatively, if the system control unit 218 can acquire information on the user's gaze position, the system control unit 218 may calculate the interocular spacing 234 based on the acquired information on the user's gaze position.

[0054] In step S603, the system control unit 218 determines the subject that the user is focusing on (the subject of interest) and acquires distance information (subject distance information) indicating the distance from the HMD 100 to the subject of interest. The system control unit 218 acquires the subject distance information based, for example, on the amount of defocus output from the imaging unit 211. The system control unit 218 may determine the subject of interest to be, for example, a subject that is in the center of the image. Alternatively, if the system control unit 218 can acquire information on the user's line of sight, the system control unit 218 may determine the subject of interest based on the user's line of sight.

[0055] In step S604, the system control unit 218 acquires subject distance information in the same manner as in step S603, after a predetermined time has elapsed since the processing in step S603.

[0056] In step S605, the system control unit 218 calculates the rate of change of subject distance from the subject distance information acquired in steps S603 and S604. The system control unit 218 also calculates the amount of adjustment for image parallax. The mechanical adjustment amount (the amount of adjustment for image parallax due to mechanical adjustment, and the rotation angle in the parallax direction of the image capturing units 200R and 200L) is calculated from the following equation 1. In equation 1, d1 is the subject distance indicated by the subject distance information acquired in step S603, and d2 is the subject distance indicated by the subject distance information acquired in step S604. θ is the mechanical adjustment amount, and φ is the rotation angle of the image capturing units 200R and 200L with respect to the X-axis direction of the HMD100.

number

[0057] Figure 7 shows the relationship between the amount of mechanical adjustment and the change in subject distance when adjusting the parallax of an image by mechanical adjustment. In Figure 7, the difference X is the difference between the subject distance acquired in step S603 and the subject distance acquired in step S607. When the subject distance changes by the difference X, the system control unit 218 adjusts the parallax of the image by rotating the image capturing units 200R and 200L so that the rotation angle of the image capturing units 200R and 200L becomes φ+θ.

[0058] The amount of image shift in soft adjustment (the amount of image parallax adjustment by soft adjustment) is calculated from the following equation 2. In equation 2, d1 is the subject distance indicated by the subject distance information acquired in step S603, and d2 is the subject distance indicated by the subject distance information acquired in step S604. S is the amount of shift, and φ is the rotation angle of the image acquisition unit 200R,200L of the HMD100 with respect to the X axis.

number

[0059] Figure 8 shows the relationship between the change in subject distance and the shift amount when adjusting the image parallax by software adjustment. In Figure 8, difference X is the difference between the subject distance acquired in step S603 and the subject distance acquired in step S607. When the subject distance changes by difference X, the system control unit 218 adjusts (moves) the display position of images 241R and 241L by the shift amount S, so that they are displayed as images 242R and 242L.

[0060] Returning to the explanation of Figure 6, in step S606, the system control unit 218 performs a process to determine the method for adjusting the parallax and adjusts the parallax of the image.

[0061] Figure 9 is a flowchart showing an example of the process for determining the method of adjusting for parallax, which is performed in step S606 of Figure 6.

[0062] In step S901, the system control unit 218 determines whether the rate of change of the subject distance is greater than or equal to a threshold. If the rate of change of the subject distance is greater than or equal to the threshold, the system control unit 218 proceeds to step S902; otherwise, it proceeds to step S903. The threshold may be determined based on the operating speed of the mechanism that performs mechanical adjustment (for example, the upper limit speed of rotation in the yaw direction). The threshold may also be a value that can be arbitrarily set by the user. The threshold may also be a value suitable for the user that has been calibrated in advance by the user. Furthermore, the threshold may be different depending on the subject distance. For example, the threshold may be smaller the shorter the subject distance. By setting a smaller threshold for shorter subject distances, it becomes easier to adjust the image parallax by soft adjustment when the subject distance is shorter.

[0063] In step S902, the system control unit 218 performs soft adjustment to adjust the image parallax. In step S903, the system control unit 218 performs mechanical adjustment to adjust the image parallax.

[0064] As described above, according to this embodiment, the image is mechanically adjusted according to the subject distance. Parallax is adjusted, and if the rate of change in subject distance exceeds a threshold, the image parallax is adjusted by software. This makes it possible to display a 3D image suitable for the user, even for subjects whose rate of change in subject distance is faster than the threshold.

[0065] In this embodiment, mechanical adjustment and soft adjustment are switched depending on whether the rate of change of the subject distance is above a threshold. However, mechanical adjustment and soft adjustment may also be switched depending on whether the subject distance is above a predetermined distance. For example, if the subject distance is above a predetermined distance, the image parallax may be adjusted by mechanical adjustment, and if the subject distance is below a predetermined distance, the image parallax may be adjusted by soft adjustment. This makes it possible to display a stereoscopic image suitable for the user even for subjects whose distance is shorter than a predetermined distance.

[0066] (Second embodiment) In the second embodiment, the HMD100 further adjusts the focus of the imaging optical system for the subject of interest. Note that the parts that overlap with the first embodiment will not be explained.

[0067] Figure 10 is a flowchart showing an example of the focus adjustment process. Note that the processes in steps S1001 to S1006 are the same as the processes in steps S601 to S606 in Figure 6.

[0068] In step S1007, the system control unit 218 controls the AF drive circuit 204 based on the amount of defocus, and adjusts the focus of the imaging optical system for the subject of interest. As a result, the HMD100 can provide the user with an image that is in focus on the subject of interest.

[0069] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0070] The above embodiments are merely examples, and configurations obtained by appropriately modifying or changing the configuration of the above embodiments within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the configurations of the above embodiments are also included in the present invention.

[0071] This embodiment includes the following configurations and methods. (Composition 1) A first acquisition means for acquiring a first image captured through a first optical system of the imaging device and a second image having parallax with respect to the first image, captured through a second optical system of the imaging device, A second acquisition means for acquiring distance information indicating the distance from the imaging device to the subject, If the rate of change of the distance indicated by the distance information is less than a threshold, control means control the parallax between the first image and the second image by mechanical adjustment of the angle between the optical axis of the first optical system and the optical axis of the second optical system; if the rate of change of the distance is greater than the threshold, control means control the parallax between the first image and the second image by soft adjustment of image processing of the first image and the second image. An electronic device characterized by having the following features. (Configuration 2) The control means adjusts the rotation angle in the parallax direction of the first optical system and the second optical system in the mechanical adjustment. The electronic device according to configuration 1, characterized by the features described above. (Composition 3) The control means, in the software adjustment, changes the display position in which the first image and the second image are displayed on the display device from the position in which the software adjustment is not performed. The electronic device according to configuration 1 or 2, characterized by the above. (Composition 4) The control means, in the soft adjustment, shifts the first image and the second image in the parallax direction. The electronic device according to configuration 3, characterized by the features described above. (Composition 5) The control means further adjusts the distance between the optical axes of the first optical system and the second optical system. An electronic device according to any one of configurations 1 to 4, characterized by the features described herein. (Composition 6) The control means further controls the focus of the imaging device. An electronic device according to any one of configurations 1 to 5, characterized by the features described herein. (Composition 7) The threshold is based on the operating speed of the mechanism that performs the mechanical adjustment. An electronic device according to any one of configurations 1 to 6, characterized by the features described herein. (Composition 8) The threshold value is a value that can be arbitrarily set by the user. An electronic device according to any one of configurations 1 to 6, characterized by the features described herein. (Composition 9) The threshold value varies depending on the distance from the imaging device to the subject. An electronic device according to any one of configurations 1 to 6, characterized by the features described herein. (Composition 10) The second acquisition means acquires distance information indicating the distance from the imaging device to the subject that is captured in the center of the first image and the second image. An electronic device according to any one of configurations 1 to 9, characterized by the features described herein. (Composition 11) It further includes a detection means for detecting the user's gaze position, The second acquisition means acquires distance information indicating the distance from the imaging device to the subject determined based on the line of sight. An electronic device according to any one of configurations 1 to 10, characterized by the features described herein. (Composition 12) Acquisition means for acquiring a first image captured through a first optical system of an imaging device and a second image having parallax with respect to the first image, captured through a second optical system of the imaging device, If the distance from the imaging device to the subject changes at a first speed, the angle between the optical axis of the first optical system and the optical axis of the second optical system is adjusted so that the distance changes at a speed faster than the first speed. First When the speed changes at 2, a control means controls the parallax between the first image and the second image by performing image processing on the first image and the second image. An electronic device characterized by having the following features. (method) A first acquisition step involves acquiring a first image captured through a first optical system of the imaging device and a second image captured through a second optical system of the imaging device, which has parallax with respect to the first image. A second acquisition step involves acquiring distance information indicating the distance from the imaging device to the subject, A control step to control the parallax between the first image and the second image by mechanical adjustment, which adjusts the angle between the optical axis of the first optical system and the optical axis of the second optical system, if the rate of change of the distance indicated by the distance information is less than a threshold, and by soft adjustment, which performs image processing on the first image and the second image, if the rate of change of the distance is greater than the threshold, and A method for controlling electronic equipment, characterized by having the following features. (program) A program for causing a computer to function as one of the electronic devices described in any one of the configurations 1 to 12. (medium) A computer-readable storage medium containing a program for causing the computer to function as one of the electronic devices described in any one of items 1 to 12. [Explanation of symbols]

[0072] 100: Head-mounted display (HMD) 200R, 200L: Image acquisition unit 218: System control unit

Claims

1. A first acquisition means for acquiring a first image captured through a first optical system of the imaging device and a second image having parallax with respect to the first image, captured through a second optical system of the imaging device, A second acquisition means for acquiring distance information indicating the distance from the imaging device to the subject, If the rate of change of the distance indicated by the distance information is less than a threshold, control means control the parallax between the first image and the second image by mechanical adjustment of the angle between the optical axis of the first optical system and the optical axis of the second optical system; if the rate of change of the distance is greater than the threshold, control means control the parallax between the first image and the second image by soft adjustment of image processing of the first image and the second image. An electronic device characterized by having the following features.

2. The control means adjusts the rotation angle in the parallax direction of the first optical system and the second optical system in the mechanical adjustment. The electronic device according to feature 1.

3. The control means, in the software adjustment, changes the display position in which the first image and the second image are displayed on the display device from the position in which the software adjustment is not performed. The electronic device according to feature 1.

4. The control means, in the soft adjustment, shifts the first image and the second image in the parallax direction. The electronic device according to feature 3.

5. The control means further adjusts the distance between the optical axes of the first optical system and the second optical system. The electronic device according to feature 1.

6. The control means further controls the focus of the imaging device. The electronic device according to feature 1.

7. The threshold is based on the operating speed of the mechanism that performs the mechanical adjustment. The electronic device according to feature 1.

8. The threshold value is a value that can be arbitrarily set by the user. The electronic device according to feature 1.

9. The threshold value varies depending on the distance from the imaging device to the subject. The electronic device according to feature 1.

10. The second acquisition means acquires distance information indicating the distance from the imaging device to the subject captured in the center of the first image and the second image. The electronic device according to feature 1.

11. It further includes a detection means for detecting the user's gaze position, The second acquisition means acquires distance information indicating the distance from the imaging device to the subject determined based on the line of sight. The electronic device according to feature 1.

12. Acquisition means for acquiring a first image captured through a first optical system of an imaging device and a second image having parallax with respect to the first image, captured through a second optical system of the imaging device, When the distance from the imaging device to the subject changes at a first speed, the control means adjusts the angle between the optical axis of the first optical system and the optical axis of the second optical system, and when the distance changes at a second speed faster than the first speed, it performs image processing on the first and second images to control the parallax between the first and second images. An electronic device characterized by having the following features.

13. A first acquisition step involves acquiring a first image captured through a first optical system of the imaging device and a second image captured through a second optical system of the imaging device, which has parallax with respect to the first image. A second acquisition step involves acquiring distance information indicating the distance from the imaging device to the subject, A control step to control the parallax between the first image and the second image by mechanically adjusting the angle between the optical axis of the first optical system and the optical axis of the second optical system if the rate of change of the distance indicated by the distance information is less than a threshold, and by soft adjustment of image processing of the first image and the second image if the rate of change of the distance is greater than the threshold. A method for controlling electronic equipment, characterized by having the following features.

14. A program for causing a computer to function as one of the means of an electronic device according to any one of claims 1 to 12.

15. A computer-readable storage medium storing a program for causing a computer to function as one of the means of an electronic device according to any one of claims 1 to 12.

Images