Image processing device, display device, and image processing method

Abstract

To cause a display device to show good 3D and 2D images that correspond to the luminance of the subject to be image-captured.SOLUTION: Image processing devices 214, 218 comprise: processing means for causing display elements 217R, 217L to show images using signals from imaging elements 211R, 211L that capture images of a subject through a first optical system 202R and a second optical system 202L that are arranged so as to cause parallax to occur; and acquisition means for acquiring the luminance information of the subject. The processing means includes a first process for causing the display elements to show a first and a second image that mutually have parallax and a second process for causing the display elements to show two third images that do not have parallax, the first and second processes being switched in accordance with the luminance information.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device that displays an image on a display device such as a head-mounted display (HMD). [Background technology]

[0002] Display devices such as HMDs that allow users to experience virtual reality (VR) and augmented reality (AR) include those that generate three-dimensional (3D) images that can be viewed stereoscopically by capturing images using image sensors and then display these 3D images.

[0003] Patent Document 1 also discloses a display device that switches between displaying a stereoscopic 3D image and a two-dimensional (2D) image with an extended depth of field according to information on the shooting conditions. The shooting conditions referred to here are optical zoom value, electronic zoom value, user operation information, parallax information, and distance information. Patent Document 2 further discloses an imaging device that performs focus detection by a pupil division method using an imaging element in which a microlens is formed on each of a plurality of pixels arranged two-dimensionally. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6908039 [Patent Document 2] JP 2014-182360 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the display device disclosed in Patent Document 1, the display image is automatically switched between a 3D image and a 2D image according to information on the shooting situation, regardless of the brightness of the shooting environment (subject). Therefore, even if the user wants to display a 3D image, a 2D image may be displayed, which reduces usability (visibility).

[0006] Furthermore, when performing focus detection as disclosed in Patent Document 2, if the brightness of the shooting environment decreases, the sensitivity of the image sensor is increased or the charge accumulation time is lengthened. If the charge accumulation time is lengthened, the interval (period) of acquiring images by shooting becomes longer, the image display rate decreases, and it becomes difficult to display images smoothly.

[0007] The present invention provides an image processing device and the like that is capable of performing good 3D image display and 2D image display according to the luminance of a subject to be imaged. [Means for solving the problem]

[0008] An image processing device according to one aspect of the present invention includes a processing means for displaying an image on a display element using a signal from an imaging element that images an object through a first optical system and a second optical system arranged to generate parallax, and an acquisition means for acquiring luminance information of the object. The processing means is characterized in that it switches between a first process for displaying a first image and a second image having a parallax on the display element and a second process for displaying two third images having no parallax on the display element according to the luminance information. Note that a display device having the image processing device also constitutes another aspect of the present invention.

[0009] An image processing method according to another aspect of the present invention includes a processing step of displaying an image on a display element using a signal from an imaging element that images an object through a first optical system and a second optical system arranged to generate parallax, and a step of acquiring luminance information of the object. In the processing step, a first process of displaying a first image and a second image having a parallax on the display element and a second process of displaying two third images having no parallax on the display element are switched according to the luminance information. Note that a program for causing a computer to execute processing according to the image processing method also constitutes another aspect of the present invention. Effect of the Invention

[0010] According to the present invention, it is possible to perform good 3D image display and 2D image display according to the luminance of the subject to be imaged. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing a configuration of a head mounted display according to an embodiment. [Diagram 2] FIG. 1 is a top view showing a configuration of a head mounted display according to an embodiment. [Diagram 3] FIG. 2 is a diagram showing a pixel array of an imaging element used in the head mounted display of the embodiment. [Figure 4] 4A and 4B are diagrams for explaining focus detection using the split-pupil method. [Diagram 5] 4 is a flowchart showing a main process executed by the head mounted display of the embodiment. [Figure 6] FIG. 13 is a diagram showing switching between 3D and 2D image displays according to brightness. [Figure 7] 11 is a flowchart showing a focus adjustment process in the main process. [Figure 8] 11 is a flowchart showing a display switching process in the main process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. EXAMPLES

[0013] <Head-mounted display configuration> 2 shows a schematic configuration of a head mounted display (HMD) 100 as a display device according to an embodiment of the present invention, viewed from above. The HMD 100 is worn on the head of a user U, and presents a right eye image and a left eye image to the user's right eye RE and left eye LE, respectively. In the figure, the Z direction corresponds to the up-down direction, the X direction corresponds to the left-right direction, and the Y direction corresponds to the front-back direction.

[0014] The HMD 100 has a right eye imaging display unit 250R and a left eye imaging display unit 250L. The right eye imaging display unit 250R has a right eye imaging unit 200R including a right eye optical system (first optical system) 202R and a right eye imaging element (first imaging element, first imaging region) 211R, and a right eye display element (first display element, first display region) 217R. The left eye imaging display unit 250L has a left eye imaging unit 200L including a left eye optical system (second optical system) 202L and a left eye imaging element (second imaging element, second imaging region) 211L, and a left eye display element (second display element) 217L.

[0015] The right eye imaging element 211R and the left eye imaging element 211L may be different imaging regions in a single imaging element. In this case, the right eye optical system 202R and the left eye optical system 202L may be bent using two reflecting surfaces so that the distance between the optical systems 202R and 202L on the image side becomes narrow. The right eye display element 217R and the left eye display element 217L may be different display regions in a single display element.

[0016] The right-eye optical system 202R and the left-eye optical system 202L are arranged in parallel to generate parallax with respect to the subject. The right-eye optical system 202R and the left-eye optical system 202L are each composed of one or more optical members such as a lens, a prism, and a mirror, and form an image of light from the subject on the right-eye imaging element 211R and the left-eye imaging element 211L. The right-eye imaging element 211R and the left-eye imaging element 211L are each composed of a photoelectric conversion element such as a CMOS sensor or a CCD sensor, and photoelectrically convert (image) the subject image formed on their imaging surfaces to output an imaging signal as an electrical signal. The imaging signal output from the right-eye imaging element 211R corresponds to a first signal, and the imaging signal output from the left-eye imaging element 211L corresponds to a second signal. Right eye image data and left eye image data are generated from the imaging signals output from the right eye imaging element 211R and the left eye imaging element 211L, respectively, to display a right eye image (first image) and a left eye image (second image) that have parallax with each other and can be observed as stereoscopic (3D) images.

[0017] The right eye display element 217R and the left eye display element 217L are each configured with a display device such as an LCD panel or an organic EL panel, and display a right eye image corresponding to the right eye image data and a left eye image corresponding to the left eye image data. Although not shown in Fig. 2 but shown in Fig. 1, the right eye imaging display unit 250R and the left eye imaging display unit 250L each have an eyepiece optical system 116 for magnifying and observing the right eye image and the left eye image displayed on the right eye display element 217R and the left eye display element 217L with the user's right eye RE and left eye LE, respectively.

[0018] The right eye imaging display unit 250R and the left eye imaging display unit 250L each have an imaging adjustment unit 320L, 320R. The imaging adjustment units 320L, 320R rotate the right eye imaging unit 200R and the left eye imaging unit 200L in the X direction around an axis parallel to the Z direction to adjust the convergence angle (or convergence distance) of the right eye imaging unit 200R and the left eye imaging unit 200L, respectively. This makes it possible to obtain a convergence state according to the distance to the subject that the user wishes to stereoscopically view. In addition, the imaging adjustment units 320L, 320R move the right eye imaging unit 200R and the left eye imaging unit 200L in the X direction to adjust the base line length 323, which is the distance in the X direction between the imaging optical axes (shown by a dashed line in the figure). This makes it possible to adjust the base line length to match the distance 321 between the right eye RE and the left eye LE of the user.

[0019] The HMD 100 is also provided with a shutter button 101, a power switch 102, and a mode changeover switch 103, which will be described later.

[0020] Fig. 1 shows the internal configuration of the HMD 100. In Fig. 1, one of the right eye imaging display unit 250R and the left eye imaging display unit 250L shown in Fig. 2 is shown as the imaging display unit 250. In addition, one of the right eye imaging unit 200R (right eye optical system 202R, right eye imaging element 211R) and the left eye imaging unit 200L (left eye optical system 202L, left eye imaging element 211L) is shown as the imaging unit 200 (optical system 202, imaging element 211). In addition, one of the right eye display element 217R and the left eye display element 217L is shown as the display element 217.

[0021] The imaging unit 200 has an aperture 201, an optical system 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens control circuit 205, a shutter 210, and an image sensor 211. The imaging unit 200 also has an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, a display element 217, and a non-volatile memory 220.

[0022] The aperture 201 is an optical member for adjusting the amount of light incident from a subject by changing its aperture diameter. The aperture drive circuit 203 drives the aperture 201 to change its aperture diameter. The optical system 202 forms an image from the light from the subject as described above. The AF drive circuit 204 drives the optical system 202 for focus adjustment. The lens control circuit 205 controls the aperture drive circuit 203 and the AF drive circuit 204 in response to an aperture drive command and a focus drive command from a system control unit 218 (described later), thereby controlling the drive of the aperture 201 and the optical system 202.

[0023] The shutter 210 is a mechanical shutter such as a focal plane shutter, and operates in response to an instruction from the system control unit 218 to control the exposure amount of the image sensor 211. As described above, the image sensor 211 converts the subject image into an image signal. The image sensor 211 also has a microlens that divides the subject image into two for each pixel to form a pair of phase difference images, and photoelectrically converts each of the pair of phase difference images for each pixel to output a pair of phase difference image signals. The image sensor 211 includes a circuit that synthesizes a pair of phase difference image signals from a plurality of pixels within a set focus detection range to generate and output a pair of focus detection signals. The system control unit 218 detects the phase difference between the pair of focus detection signals from the image sensor 211, and obtains a defocus amount from the phase difference. Furthermore, the system control unit 218 calculates the drive amount of the optical system 202 to obtain a focus state for the subject from the obtained defocus amount, and outputs a focus drive command to the lens control circuit 205. This makes it possible to perform focus detection (phase difference AF) using an image sensor phase difference detection method.

[0024] The A / D converter 212 converts the analog image signal output from the image sensor 211 into digital image data and outputs the digital image data to the image processing unit 214 and the memory control unit 213. The image processing unit 214 performs image processing such as pixel interpolation, reduction / enlargement (resizing) processing, and color conversion processing on the image data from the A / D converter 212 or the image data from the memory control unit 213 to generate image data. The image processing unit 214 also performs AWB (auto white balance) processing on the image data based on the results of calculations using the image data. The image data is written into the memory 215 directly or via the memory control unit 213, and is also output to the system control unit 218.

[0025] The memory 215 stores image data (image data for recording and display). The memory 215 also serves as an image display memory (video memory). The system control unit 218 performs automatic exposure (AE) processing using luminance information of the imaging environment acquired from the image data, and performs phase difference AF processing using the pair of focus detection signals described above.

[0026] The D / A converter 216 converts the display image data read from the memory 215 into an analog image signal and supplies it to the display element 217. As a result, display images (right eye image and left eye image) are displayed on the display element 217. Live view display is performed by sequentially displaying display images corresponding to image data generated from an imaging signal from the imaging element 211 at a predetermined frame rate on the display element 217.

[0027] The system control unit 218 includes at least one processor such as a CPU, and controls the overall operation of the HMD 100, including the above-mentioned AE and AF, by executing programs stored in the non-volatile memory 220. The system control unit 218 also performs display control, controlling the memory 215, the D / A converter 216, the display element 217, and the like.

[0028] The HMD 100 also includes a system memory 219 , a system timer 221 , a communication unit 222 , a posture detection unit 223 , and an eyepiece detection unit 118 .

[0029] The system memory 219 is configured, for example, by a RAM, and stores constants and variables for operation of the system control unit 218, and also develops programs read from the nonvolatile memory 220. The nonvolatile memory 220 is an electrically erasable and recordable memory, and is configured, for example, by an EEPROM. The nonvolatile memory 220 stores constants, programs, etc. for operation of the system control unit 218.

[0030] The system timer 221 measures the time used for various controls and the time of a built-in clock. The communication unit 222 transmits and receives image data, audio data, and various information to and from an external device connected wirelessly or by wire. The communication unit 222 can also connect to a wireless LAN (Local Area Network) or the Internet. The communication unit 222 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy.

[0031] The orientation detection unit 223 detects the orientation of the HMD 100 with respect to the direction of gravity using an acceleration sensor, a gyro sensor, etc. The system control unit 218 can acquire the movement (pan, tilt, etc.) of the HMD 100 when the imaging unit 200 captures an image via the orientation detection unit 223.

[0032] The eyepiece detection unit 118 uses an infrared proximity sensor or the like, and detects whether the user's eye is placed near or far from the eyepiece optical system 116 (that is, whether the user has worn the HMD 100 on his / her head). The system control unit 218 switches the display element 217 between display and non-display depending on whether the user's eye is placed near or far from the eyepiece optical system 116.

[0033] The HMD 100 further includes a power supply control unit 225 , a power supply unit 226 , a recording medium I / F 227 , an operation unit 229 , and an eyeball information acquisition unit 240 .

[0034] The power supply control unit 225 is configured with a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to be energized, and detects whether a battery is installed, the type of battery, and the remaining battery level. The power supply unit 226 is a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, or an AC adapter. The recording medium I / F 227 is an interface with the recording medium 228. The recording medium 228 is configured with a semiconductor memory, an optical disk, etc., and records image data for recording.

[0035] The operation unit 229 is operated by a user to input various instructions to the system control unit 218. The operation unit 229 includes the above-mentioned shutter button 101, power switch 102, and mode changeover switch 103, as well as other operation members 230 such as dials, directional keys, and menu buttons.

[0036] The shutter button 101 has a first shutter switch 231 and a second shutter switch 232. The first shutter switch 231 is turned on when the shutter button 101 is pressed halfway, and generates a first shutter switch signal SW1. The system control unit 218, which has received the first shutter switch signal SW1, performs shooting preparation processing including AF processing, AE processing, and AWB processing. The second shutter switch 232 is turned on when the shutter button 101 is pressed all the way, and generates a second shutter switch signal SW2. The system control unit 218, which has received the second shutter switch signal SW2, performs a series of shooting processing from reading out the imaging signal from the imaging element 211 to generating image data and recording the image data for recording on the recording medium 228.

[0037] The mode changeover switch 103 switches the operation mode of the system control unit 218 between a capture / display mode, a playback mode, an AR display mode, and the like.

[0038] The eyeball information acquisition unit 240 acquires an image of the eyeballs (RE, LE) of the user wearing the HMD 100, and calculates information about the user's line of sight from the eyeball image. The system control unit 218 sets a focus detection range using the information about the line of sight.

[0039] <Configuration of the image sensor 211> Fig. 3 shows a two-dimensional pixel array on the imaging surface of a CMOS sensor, which is the image sensor 211. Fig. 3 shows an array of imaging pixels in an area of ​​4 columns x 4 rows, but the actual image sensor 211 includes a greater number of columns x rows of imaging pixels in both the vertical (y direction) and horizontal (x direction) directions.

[0040] One pixel group 400 is made up of 2 columns x 2 rows of imaging pixels, and these imaging pixels are provided with color filters in a Bayer array. In the pixel group 400, an imaging pixel 400R having a spectral sensitivity of R (red) is arranged at the top left, imaging pixels 400G having a spectral sensitivity of G (green) are arranged at the top right and bottom left, and an imaging pixel 400B having a spectral sensitivity of B (blue) is arranged at the bottom right. Furthermore, each imaging pixel has one microlens 401 for performing focus detection using an imaging surface phase difference detection method, and two sub-pixels (focus detection pixels) 402, 403 arranged in a 2 column x 1 row array.

[0041] The microlens 401 splits the light beam that has passed through the pupil of the optical system 202 into two (i.e., performs pupil division). One of the two split light beams is received by the sub-pixel 402, and the other light beam is received by the sub-pixel 403. An A+B signal obtained by adding together an A signal and a B signal generated by photoelectric conversion in each of the two sub-pixels 402, 403 is used to generate an imaging signal, and the A signal and the B signal (or a signal obtained by subtracting the A signal from the A+B signal) are used to generate one and the other of a pair of focus detection signals. Note that the number of sub-pixels provided for each imaging pixel may be three or more. Also, it is not necessary for all imaging pixels to have multiple sub-pixels, and only a portion of the imaging pixels (for example, imaging pixels that are discretely arranged) may have multiple sub-pixels.

[0042] <Relationship between phase difference and defocus amount> Next, the relationship between the phase difference (image shift amount) of a pair of focus detection signals acquired from the image sensor 211 and the defocus amount will be described with reference to FIG. 4. The defocus amount is used for AF processing and acquiring distance information. Here, it is assumed that the center of the image sensor and the position of the optical axis of the optical system coincide with each other. Here, the case where the defocus amount is calculated by the right eye imaging unit 200R will be described. However, the defocus amount may be calculated by the left eye imaging unit 200L, or the defocus amount calculated by each of the right eye imaging unit 200R and the left eye imaging unit 200L and combined may be used.

[0043] 4, the imaging surface of the image sensor 211R is disposed on an image plane 1300. The exit pupil of the optical system 202R is divided into two by a microlens for each pixel of the image sensor 211R, into a first pupil region 1303 through which a first light beam from a subject (1301, 1302) passes and a second pupil region 1304 through which a second light beam passes.

[0044] The defocus amount d is defined as |d|, where the distance from the imaging position of the light beam from the subject to the imaging plane (1300) is d<0 for a front-focus state where the imaging position is on the subject side of the imaging plane, and d>0 for a back-focus state where the imaging position of the subject is on the opposite side of the imaging plane. The in-focus state where the imaging position of the subject is on the imaging plane is d=0. Figure 4 shows the in-focus state for subject 1301 and the front-focus state for subject 1302. The front-focus state and the back-focus state together are defined as a defocus state (|d|>0).

[0045] In the front focus state, the light beams from the subject 1302 that pass through the first pupil region 1303 and the second pupil region 1304 are each focused once and then spread to have blur widths Γ1 and Γ2 centered on the center of gravity positions G1 and G2 of the light beams, forming blurred images on the imaging surface. These blurred images are received by two sub-pixels (402 and 403 in FIG. 3) in each of the multiple imaging pixels included in the range in which the blurred images are formed. A signals and B signals from the two sub-pixels of the multiple imaging pixels are each combined to generate a pair of focus detection signals.

[0046] The blur widths Γ1 and Γ2 of the subject image increase almost in proportion to an increase in the magnitude |d| of the defocus amount d. Similarly, the magnitude |p| of the image shift amount p (center of gravity position difference G1-G2) between a pair of focus detection signals also increases almost in proportion to an increase in the magnitude |d| of the defocus amount d. In the back focus state, the image shift direction between a pair of focus detection signals is opposite to that in the front focus state, but the same is true.

[0047] The defocus amount d can be calculated by multiplying the image shift amount p by a known conversion coefficient K. The conversion coefficient K is a value that depends on the optical system, and more specifically, on the angle of incidence of the light beam into the optical system, the F-number of the optical system, and the position of the optical axis.

[0048] <Display switching according to the brightness of the shooting environment> In this embodiment, in order to obtain a good focus detection result (that is, an AF result) while presenting a 3D image to the user, display control is performed according to the brightness of the shooting environment. The brightness of the shooting environment is the brightness of the environment in which the subject is placed, and more specifically, the brightness of the subject. The subject includes a main subject and background objects. In the following, shooting environments with different brightnesses are explained by dividing them into four cases (brightness levels), Case 1 to Case 4, but the number of cases may be changed.

[0049] 5 shows a main process executed in accordance with a program by system control unit 218. System control unit 218 and image processing unit 214 correspond to a processing means and an acquiring means, and constitute an image processing device.

[0050] In step S1, the system control unit 218 determines the luminance of the shooting environment. The charge accumulation time (shutter time) of the image sensor set as a result of AE processing is used for the luminance determination. As described above, in this embodiment, the AE processing is performed using luminance information obtained from image data. However, a photometric unit that measures the luminance of the shooting environment may be provided in the HMD 100, and the AE processing may be performed using the photometric result of the photometric unit as luminance information. Furthermore, the luminance determination may be performed directly based on whether the luminance information is bright (predetermined luminance) or darker, rather than based on the result of AE processing.

[0051] FIG. 6 shows cases 1, 2, 3, and 4 from the brightest side, and further shows an example of a shooting scene (Image) for each case. FIG. 6 also shows whether or not 3D images and two-dimensional (2D) images can be displayed (upper row) for each case, and the judgment conditions (lower row). In the following description, a mode that displays 3D images is called a 3D display mode, and a mode that displays 2D images is called a 2D display mode. In the judgment conditions, tv3d and tv2d respectively indicate the charge accumulation time of the image sensor in the 3D display mode and the 2D display mode. Also, Tv_1, Tv_2, and Tv_3 are thresholds of the charge accumulation time, and increase in this order. Tv_1, Tv_2, and Tv_3 correspond to the predetermined times in Cases 1, 2, and 3, respectively.

[0052] Case 1 is the brightest shooting environment, and signals can be read from the image sensor at a high frame rate (FR) and high speed to obtain right eye image data and left eye image data for 3D display. In Case 1, 3D image display and 2D image display are always possible. A short time equal to or less than the first threshold value Tv_1 is set as the charge accumulation times tv3d and tv2d in the 3D and 2D display modes. Tv_1 is, for example, a time shorter than 1 / 60 seconds.

[0053] In this embodiment, the charge accumulation time tv2d in the 2D display mode is set to a time equal to or shorter than Tv_1 in all of Case 1 to Case 4. This is because it is possible to select a method in which an image for 2D image display is acquired by only one of the right eye imaging unit 200R and the left eye imaging unit 200L, or an image is acquired alternately by the right eye imaging unit 200R and the left eye imaging unit 200L. Image data with a wide dynamic range may be generated from right eye image data acquired through the right eye imaging unit 200R and left eye image data acquired through the left eye imaging unit 200L with exposure conditions (charge accumulation time, etc.) different from those of the right eye imaging unit 200R. Then, two identical images (third images) corresponding to this wide dynamic range image data may be displayed in 2D on the right eye display element 217R and the left eye display element 217L.

[0054] In Case 1, the system control unit 218 selects the 3D display mode. However, the display mode may be made selectable (changeable) between the 2D display mode and the 3D display mode by a selection operation by the user.

[0055] Case 2 is a darker (lower brightness) shooting environment than Case 1 but relatively bright, and signals can be read from the imaging element at high to medium FR and high to medium speed to obtain an image required for 3D image display. In Case 2, 2D image display is always possible. On the other hand, 3D image display is possible when the charge accumulation time tv3d is set to a time equal to or greater than the first threshold Tv_1 and equal to or less than the second threshold Tv_2. Tv_2 is a time during which a high FR can be maintained, such as 1 / 30 seconds. The system control unit 218 selects the 3D display mode when tv3d is equal to or less than Tv_2, and selects the 2D display mode when tv3d is longer than Tv_2.

[0056] However, when tv3d is equal to or smaller than Tv_2, the display mode may be made selectable (changeable) between 2D and 3D by a user selection operation.

[0057] Case 3 is a shooting environment darker (relatively darker) than Case 2, and a signal can be read from the imaging element at a medium FR and medium speed to obtain an image required for 3D image display. In Case 3, 2D image display is always possible. On the other hand, 3D image display is possible when the charge accumulation time tv3d is set to a time equal to or greater than the second threshold Tv_2 and equal to or less than the third threshold Tv_3. Tv_3 is, for example, a time longer than 1 / 30 seconds. For this reason, shooting in Case 3 is limited to when the user is stationary or moving slowly. When tv3d is equal to or less than Tv_3, the system control unit 218 selects the 3D display mode, and when tv3d is longer than Tv_3, the system control unit 218 selects the 2D display mode.

[0058] However, only when the user permits 3D image display with the charge accumulation time tv3d corresponding to the third threshold Tv_3, the user may be allowed to select (change) between the 2D display mode and the 3D display mode through a selection operation.

[0059] Case 4 is a darker shooting environment than Case 3, and the image required for 3D image display can only be acquired with a medium to low FR and medium to low speed signal readout from the image sensor. For this reason, 3D image display is not possible, and only 2D image display is possible.

[0060] However, also in Case 4, if the user permits 3D image display with a charge accumulation time longer than the third threshold Tv_3, the 3D display mode may be selected by a selection operation by the user.

[0061] Even if 3D image display is possible in Case 1 to Case 3, 2D display mode may be selected if the main subject is located at a long distance. Whether or not the main subject is the main subject can be determined using line-of-sight information from eyeball information acquisition unit 240. The distance of the main subject can be obtained from a distance map generated from the distribution of defocus amounts within the imaging angle of view.

[0062] Next, in step S2, the system control unit 218 sets focus detection parameters and acquires information according to the luminance of the shooting environment determined in step S1. The luminance of Cases 1, 2, and 3 corresponds to the first luminance, and the luminance of Case 4 corresponds to the second luminance.

[0063] When the determination luminance in step S1 is the luminance of Case 1, the system control unit 218 sets a time equivalent to the first threshold value Tv_1 as the charge accumulation time tvAF of the image sensor for acquiring the focus detection signal. In this case, the system control unit 218 sets the acquisition range (focus detection range) of the focus detection signal to the entire image sensor (imaging surface). The system control unit 218 also detects the main subject Target1 shown in FIG. 6 from the image data, and acquires subject information such as the type, position, and distance of the main subject Target1. Furthermore, the system control unit 218 acquires information regarding the user's line of sight through the eyeball information acquisition unit 240, and detects the user's gaze area EyePoint1 from the gaze information. In this case, it may be determined from the subject information and the gaze area that the user is gazing at the main subject, and the focus detection result in the vicinity of EyePoint1 may be used for AF processing.

[0064] When the determined luminance in step S1 is the luminance of Case 2, the system control unit 218 sets the charge accumulation time tvAF for acquiring the focus detection signal as follows. When the 2D display mode is set, a time corresponding to the first threshold Tv_1 is set as tvAF, and when the 3D display mode is set, a time corresponding to the second threshold Tv_2 is set. The system control unit 218 also sets the focus detection range to a range narrower than that of Case 1. The system control unit 218 also detects the main subject Target2 shown in FIG. 6 from the image data and acquires subject information of the main subject Target2. The system control unit 218 also acquires the user's line of sight information through the eyeball information acquisition unit 240, and detects the user's gaze area EyePoint2 from the line of sight information. In this case, it may be determined that the user is gazing at the main subject from the subject information and the gaze area, and the focus detection result in the vicinity of EyePoint2 may be used for AF processing.

[0065] When the determined luminance in step S1 is the luminance of Case 3, the system control unit 218 sets the following time as the charge accumulation time tvAF for acquiring the focus detection signal. When the 2D display mode is set, a time corresponding to the first threshold Tv_1 is set as the tvAF, and when the 3D display mode is set, a time corresponding to the third threshold Tv_3 is set. The system control unit 218 also sets the focus detection range to a range narrower than that of Case 2. The system control unit 218 also detects the main subject Target3 shown in FIG. 6 from the image data and acquires subject information of the main subject Target3. The system control unit 218 also acquires the user's line of sight information through the eyeball information acquisition unit 240, and detects the user's gaze area EyePoint3 from the line of sight information. In this case, it may be determined that the user is gazing at the main subject from the subject information and the gaze area, and the focus detection result in the vicinity of EyePoint3 may be used for AF processing.

[0066] When the determination luminance in step S1 is the luminance of Case 4, the system control unit 218 sets a time equivalent to the first threshold value Tv_1 in the 2D display mode as the charge accumulation time tvAF for acquiring the focus detection signal. At this time, the focus detection signals obtained in a plurality of frames may be added and used for focus detection. The system control unit 218 also sets the focus detection range to a range narrower than that of Case 2, as in Case 3. The system control unit 218 also detects the main subject Target4 shown in FIG. 6 from the image data and acquires subject information of the main subject Target4. Furthermore, the system control unit 218 acquires the user's line of sight information through the eyeball information acquisition unit 240, and detects the user's gaze area EyePoint4 from the line of sight information. In this case, it may be determined from the subject information and the gaze area that the user is gazing at the main subject, and the focus detection result in the vicinity of EyePoint4 may be used for AF processing.

[0067] Next, in step S3, the system control unit 218 sets display parameters for the right eye display element 217R and the left eye display element 217L according to the charge accumulation time (tv3d, tv2d) for acquiring an imaging signal (image data) according to the luminance determination result in step S1. Then, the process proceeds to step 4.

[0068] In this embodiment, a 3D image is displayed by displaying a right eye image and a left eye image having a parallax on the right eye display element 217R and the left eye display element 217L, and focus detection is performed using focus detection signals acquired from the right eye imaging unit 200R and the left eye imaging unit 200L. At this time, the charge accumulation time tvAF when acquiring the focus detection signal and the charge accumulation time tv3d when acquiring the imaging signal for 3D image display match each other, so that 3D image display with high FR and high-speed focus detection are possible.

[0069] However, in the particularly dark Case 4, it may be difficult to maintain a high FR. In such a case, the charge accumulation time tvAF when acquiring the focus detection signal is set to the charge accumulation time tv2d when acquiring the imaging signal for 2D image display. Then, one of the right eye imaging unit 200R and the left eye imaging unit 200L acquires an imaging signal for 2D image display to display the same image (third image) on the right eye display element 217R and the left eye display element 217L, and acquires a focus detection signal in the other imaging unit. This allows image display and focus detection to continue. Although the frequency of focus detection is reduced, focus detection may be performed using a focus detection signal added over multiple frames in order to continue 3D image display. Furthermore, if the shooting environment is so dark that focus detection is impossible, focus detection may not be performed and the user may be notified by display or the like that focus detection is impossible.

[0070] In step S4, the system control unit 218 performs focus detection based on the focus detection parameters set in step S2, and performs focus adjustment (AF processing) by controlling the driving of the optical system based on the defocus amount obtained as a result. The AF processing will be described in detail later.

[0071] Next, in step S5, system control unit 218 performs image processing for image display (image display processing) in accordance with the display parameters set in step S3. Details of this image processing will be described later.

[0072] In step S6, the system control unit 218 that has performed the image processing determines whether or not to end the image display and focus detection in response to a stop command from the user, powering off of the HMD 100, detection of eye movement away from the eye by the eye contact detection unit 118, etc. If the image display and focus detection are to be continued, the process returns to step S1, and if they are to be ended, the process ends.

[0073] The flowchart in Fig. 7 shows the focus adjustment process executed in step S4 in Fig. 5. The right eye optical system 202R and the left eye optical system 202L may have a difference in focus position due to mechanical factors, but there is no difference in focus detection process. For this reason, the right eye optical system 202R and the left eye optical system 202L are collectively referred to as the optical system 202, and the right eye imaging element 211R and the left eye imaging element 211L are collectively referred to as the imaging element 211.

[0074] In this embodiment, signals from one sub-pixel 402 of multiple imaging pixels within the acquisition range of focus detection signals in the image sensor 211 are collected (combined) to generate a first focus detection signal, and signals from the other sub-pixel 403 are collected to generate a second focus detection signal. At this time, both the first and second focus detection signals are generated as a Y signal obtained by adding signals from four sub-pixels of green (G), red (R), blue (B) and green (G). The system control unit 218 calculates an image shift amount p of these first and second focus detection signals (a pair of focus detection signals) by correlation calculation, and calculates a defocus amount d from the image shift amount p.

[0075] In step S501, the system control unit 218 sets the acquisition range (focus detection range) of the focus detection signal acquired in step S2 of FIG.

[0076] Next, in step S502, the system control unit 218 acquires a first focus detection signal (image A signal) and a second focus detection signal (image B signal) generated from signals from the sub-pixels 402 and 403 within the focus detection range.

[0077] Next, in step S503, the system control unit 218 performs pixel addition processing on each of the first and second focus detection signals. Specifically, addition processing is performed in the column direction to reduce the amount of signal data of the first and second focus detection signals, and then addition processing of G, R, B, and G is performed to obtain a Y signal. At this time, when the number of pixel additions is two, the pixel pitch is doubled, so the Nyquist frequency is 1 / 2 of that when no addition is performed, and when the number of pixel additions is three, the pixel pitch is tripled, so the Nyquist frequency is 1 / 3 of that when no addition is performed.

[0078] Next, in step S504, the system control unit 218 performs shading correction processing (optical correction processing) on ​​the first and second focus detection signals after the pixel addition processing to make these signal intensities uniform. The shading correction value is a value that depends on the incident angle of the light beam to the optical system 202, the F-number of the optical system 202, and the optical axis position.

[0079] Next, in step S505, the system control unit 218 performs bandpass filtering having a specific pass frequency band on the first and second focus detection signals after the shading correction processing in order to improve correlation (degree of agreement of the signals) and improve focus detection accuracy. Examples of bandpass filters include differential filters such as {1, 4, 4, 4, 0, -4, -4, -4, -1} that cut DC components and perform edge extraction, and additive filters such as {1, 2, 1} that suppress high-frequency noise components.

[0080] Next, in step S506, the system control unit 218 performs shift processing to relatively shift the first and second focus detection signals after the filter processing in the pupil division direction, and calculates a correlation amount that indicates the degree of coincidence of the signals. The k-th first focus detection signal after the filter processing is denoted as A(k), the second focus detection signal is denoted as B(k), and the range of number k corresponding to the focus detection range is denoted as W. If the shift amount due to the shift processing is denoted as s and the shift range of the shift amount s is denoted as Γ, the correlation amount COR is calculated by formula (1).

[0081]

number

[0082] In this shift process, a shift subtraction signal is generated by corresponding the k-th first focus detection signal A(k) and the ks-th second focus detection signal B(ks) and subtracting them. Then, the absolute value of the generated shift subtraction signal is calculated, and the correlation amount COR(s) is calculated by taking the sum of the number k within the range W corresponding to the focus detection range. If necessary, the correlation amount calculated for each row may be added across multiple rows for each shift amount. In addition, when the correlation amount COR(s) is calculated, it is possible to evaluate the reliability of the defocus amount calculated later by looking at the amount of change, peak bottom, etc.

[0083] Next, in step S507, the system control unit 218 calculates a real-valued shift amount at which the correlation amount becomes a minimum value by sub-pixel calculation from the calculated correlation amount, and sets the shift amount as the image shift amount p. Then, the system control unit 218 multiplies the image shift amount p by a conversion coefficient K to calculate a defocus amount (Def)d. The reliability of the defocus amount can also be evaluated depending on the magnitude of the conversion coefficient K.

[0084] Next, in step S508, the system control unit 218 converts the defocus amount acquired in step S507 into a driving amount of the focus lens in the optical system 202, and drives the focus lens. When this process ends in this manner, the system control unit 218 proceeds to step S5 in FIG. The flowchart in Fig. 8 shows the image display process (image processing method) executed in step S5. In step S601, the system control unit 218 acquires the result of the luminance determination in step S1 in Fig. 5 and the selection status of the 2D / 3D display mode by the user.

[0085] Next, in step S602, the system control unit 218 determines whether to display a 3D image in the 3D display mode or to display a 2D image in the 2D display mode, depending on the brightness determination result acquired in step S601 and the user selection status. If a 3D image is to be displayed, the process proceeds to step S603, and if a 2D image is to be displayed, the process proceeds to step S604.

[0086] In step S603, the system control unit 218 causes the image processing units 214 of the right eye imaging display unit 250R and the left eye imaging display unit 250L to generate right eye image data and left eye image data having mutual parallax from the imaging signals from the right eye imaging unit 200R and the left eye imaging unit 200L, respectively. Then, the process proceeds to step S605.

[0087] On the other hand, in step S604, the system control unit 218 copies the image data to display the same image corresponding to the image data generated by the imaging signal from the right eye imaging unit 200R or the left eye imaging unit 200L on both the right eye display element 217R and the left eye display element 217L. That is, the system control unit 218 generates the same image data without parallax as the right eye image data and the left eye image data. Then, the system control unit 218 proceeds to step S605.

[0088] In step S605, the system control unit 218 performs image adjustment processing on the right eye image data and left eye image data generated in step S603 or step S604 to display them on the right eye display element 217R and the left eye display element 217L. Specifically, the system control unit 218 performs processing to adjust the image display position and brightness.

[0089] Next, in step S606, the system control unit 218 supplies the right eye image data and left eye image data after the image adjustment process to the right eye display element 217R and the left eye display element 217L, respectively, via the D / A converter 216. This allows 3D image display in which a right eye image and a left eye image having parallax are displayed on the right eye display element 217R and the left eye display element 217L, or 2D image display in which a right eye image and a left eye image having no parallax are displayed. After this, the process proceeds to step S6 in FIG. 5.

[0090] As described above, by switching between 3D image display and 2D image display depending on the brightness of the shooting environment, it is possible to achieve both good image display and focus detection.

[0091] In the present embodiment, the optical system 202 is assumed to have a fixed focal length or a fixed zoom state. However, the display may be switched between 2D and 3D images in response to a change in the zoom state of the optical system 202.

[0092] Furthermore, in this embodiment, a case has been described in which the image processing device is built into the HMD, but the image processing device may be a device separate from the HMD, such as a personal computer or an image controller.

[0093] The above embodiment includes the following configurations. (Configuration 1) a processing means for displaying an image on a display element using a signal from an image pickup element which picks up an image of a subject through a first optical system and a second optical system which are arranged so as to generate parallax; An acquisition means for acquiring luminance information of the subject, The image processing device is characterized in that the processing means switches between a first process of causing the display element to display a first image and a second image having parallax with each other, and a second process of causing the display element to display two third images having no parallax, depending on the luminance information. (Configuration 2) The processing means includes: setting a charge accumulation time of the imaging element in accordance with the luminance information; When the charge accumulation time is shorter than a predetermined time, the first process is performed; 2. The image processing device according to configuration 1, wherein the second process is performed when the charge accumulation time is longer than the predetermined time. (Configuration 3) The image processing device according to configuration 1, characterized in that the processing means performs the first processing when the luminance information indicates that the luminance is brighter than a predetermined luminance, and performs the second processing when the luminance information indicates that the luminance is darker than the predetermined luminance. (Configuration 4) The processing means includes: Dividing the brightness level of the subject into a plurality of levels; 4. The image processing device according to any one of configurations 1 to 3, wherein the first process and the second process are switched for each luminance level. (Configuration 5) 5. The image processing device according to any one of configurations 1 to 4, wherein the processing means performs the first and second processes and also performs focus detection using a signal from the image sensor. (Configuration 6) the image sensor has a first image capturing area for capturing an image of the subject through the first optical system and a second image capturing area for capturing an image of the subject through the second optical system; The image processing device according to configuration 5, characterized in that, when performing the first processing, the processing means performs the focus detection using a signal from the first imaging area and a signal from the second imaging area. (Configuration 7) the image sensor has a first image capturing area for capturing an image of the subject through the first optical system and a second image capturing area for capturing an image of the subject through the second optical system; When performing the second processing, the processing means displaying the two third images on the display element using a signal from one of the first and second imaging regions; 6. The image processing device according to configuration 5, wherein the focus detection is performed using a signal from the other of the first and second imaging regions.

[0094] (Configuration 8) the image sensor has a first image capturing area for capturing an image of the subject through the first optical system and a second image capturing area for capturing an image of the subject through the second optical system; The image processing device according to any one of configurations 1 to 7, characterized in that, when performing the second processing, the processing means causes the display element to display the two third images generated using a signal from the first imaging area and a signal from the second imaging area that captures an image under exposure conditions different from those of the first imaging area. (Configuration 9) 9. The image processing device according to any one of configurations 1 to 8, wherein the processing means switches from the first processing to the second processing depending on the distance of a main subject that a user is gazing at. (Configuration 10) 9. The image processing device according to any one of configurations 1 to 8, wherein the processing means switches from one of the first process and the second process to the other in response to a user selection. (Configuration 11) An image processing device according to any one of configurations 1 to 10; the first and second optical systems; The imaging element; A display device comprising the display element.

[0095] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0096] The embodiments described above are merely representative examples, and various modifications and alterations are possible for each embodiment when implementing the present invention. [Explanation of symbols]

[0097] 202R Right Eye Optics 202L Left Eye Optics 211R right eye pixel capture 211L left eye pixel capture 217R Right eye indicates element 217L left eye represents element 218 システムcontrol department 214 Image Processing Department

Claims

1. Processing means for displaying an image on a display element using signals obtained by imaging a subject through a first optical system and a second optical system arranged so as to cause parallax, and acquisition means for acquiring luminance information of the subject, wherein the processing means switches between a first process of displaying a first image and a second image having parallax with each other on the display element according to the luminance information, and a second process of displaying two third images having no parallax on the display element. An image processing apparatus characterized by performing the above.

2. The processing means, sets the charge accumulation time of the imaging element used when imaging the subject through the first optical system and the second optical system according to the luminance information, performs the first process when the charge accumulation time is shorter than a predetermined time, The image processing apparatus according to claim 1, wherein the second process is performed when the charge accumulation time is longer than the predetermined time.

3. The processing means performs the first process when the luminance information indicates that the subject is brighter than a predetermined luminance, and performs the second process when the luminance information indicates that the subject is darker than the predetermined luminance. The image processing apparatus according to claim 1, characterized in that

4. The processing means, divides the luminance level of the subject into a plurality of levels, The image processing apparatus according to claim 1, characterized in that the first process and the second process are switched and performed for each luminance level.

5. The processing means performs the first and second processes, and performs focus detection using signals obtained by imaging a subject through the first optical system and the second optical system. The image processing apparatus according to claim 1, characterized in that

6. The processing means has a first imaging area for imaging the subject through the first optical system and a second imaging area for imaging the subject through the second optical system, When performing the first process, the focus detection is performed using signals from the first imaging area and signals from the second imaging area. The image processing apparatus according to claim 5, characterized in that

7. The processing means has a first imaging area for imaging the subject through the first optical system and a second imaging area for imaging the subject through the second optical system, When performing the second process, two third images are displayed on the display element using signals from one of the first and second imaging areas. The image processing apparatus according to claim 5, wherein the focus detection is performed using a signal from the other imaging area of the first and second imaging areas.

8. The processing means causes an image to be displayed on the display element using signals from an imaging element including a first imaging area that images the subject through the first optical system and a second imaging area that images the subject through the second optical system. The image processing apparatus according to claim 1, wherein when performing the second processing, the two third images generated using a signal from the first imaging area and a signal from the second imaging area that images under exposure conditions different from those of the first imaging area are displayed on the display element.

9. The image processing apparatus according to claim 1, wherein the processing means switches from the first processing to the second processing according to the distance of a main subject on which the user is focusing.

10. The image processing apparatus according to any one of claims 1 to 8, wherein the processing means switches from one of the first processing and the second processing to the other according to a selection by the user.

11. The display element includes a first display element and a second display element. The processing means, when performing the first processing, causes the first image to be displayed on the first display element and the second image to be displayed on the second display element. The image processing apparatus according to claim 1, wherein when performing the second processing, the third image is displayed on the first display element and the second display element.

12. The display element includes a first display area and a second display area. The processing means, when performing the first processing, causes the first image to be displayed on the first display area and the second image to be displayed on the second display area. The image processing apparatus according to claim 1, wherein when performing the second processing, the third image is displayed on the first display area and the second display area.

13. An image processing apparatus according to any one of claims 1 to 12, the first and second optical systems, the processing means, and a display apparatus characterized by including the display element.

14. A processing step of causing an image to be displayed on a display element using signals obtained by imaging a subject through a first optical system and a second optical system arranged so as to cause parallax, and a step of acquiring luminance information of the subject. In the above processing step, according to the luminance information, a first process of causing the display element to display a first image and a second image having a parallax with each other, and a second process of causing the display element to display two third images having no parallax are switched and performed, and an image processing method characterized by this.

15. A program characterized by causing a computer to execute processing according to the image processing method described in Claim 14.

Images