Imaging control device, imaging device, imaging control method, and imaging control program

Abstract

Provided are an imaging control device, an imaging device, an imaging control method, and an imaging control program.　This imaging control device comprises a processor that controls an imaging element having a plurality of pixels and including a photoelectric conversion unit that converts light incident on an imaging optical system and focused by a microlens into electric charges. The plurality of pixels include a first pixel that shares the microlens with adjacent pixels, and a second pixel different from the first pixel. The processor controls the ratio between a first exposure time, which is the exposure time of the first pixel, and a second exposure time, which is the exposure time of the second pixel, on the basis of a condition related to the angle of incidence of light to the pixels.

Description

Imaging control device, imaging device, imaging control method, and imaging control program

[0001] The technology of the present disclosure relates to an imaging control device, an imaging device, an imaging control method, and an imaging control program.

[0002] Patent document 1 describes an imaging control device that includes a control unit that controls pixels so that the reading of focus pixel signals from focus detection pixels used for focus adjustment and the reading of main line pixel signals from pixels used for image generation are performed independently, and an output unit that outputs the focus pixel signals and the main line pixel signals independently of each other.

[0003] Patent Document 2 describes a solid-state imaging device that includes a pixel array section in which a plurality of pixels are arranged two-dimensionally, and the pixel array section has an arrangement pattern in which pixel groups each including neighboring pixels of the same color are regularly arranged, and among the plurality of pixels, a phase difference detection pixel including one of a plurality of photoelectric conversion elements formed for one on-chip lens is an adjacent pixel of the same color and is included in a different adjacent pixel group.

[0004] Patent Document 3 describes an imaging element that includes imaging pixels whose accumulation time is controlled by a first accumulation control signal to generate a first image signal, and focus detection pixels whose accumulation time is controlled by a second accumulation control signal that is independent of the first accumulation control signal to generate a second image signal.

[0005] Patent Document 4 describes an imaging device comprising an imaging element having imaging pixels that output imaging pixel signals used to generate an image signal based on light beams from a subject, and function pixels that output function pixel signals that are used for predetermined functions other than generating the captured image; exposure amount detection means that detects the exposure amount of the imaging pixels based on the imaging pixel signals and detects the exposure amount of the function pixels based on the function pixel signals; exposure change means that changes the exposure of the imaging element to the light beams from the subject so that the exposure amount of one of the imaging pixels or the function pixels is appropriate; difference detection means that detects the difference between the exposure amount of the imaging pixels and the exposure amount of the function pixels; and signal processing means that performs predetermined signal processing on the imaging pixel signals and the function pixel signals.

[0006] Japanese Patent Publication No. 2017-005443 Japanese Patent Publication No. 2019-029985 Japanese Patent Publication No. 2008-085535 Japanese Patent Publication No. 2017-135521

[0007] An imaging control device, an imaging device, an imaging control method, and an imaging control program according to one embodiment of the technology of the present disclosure are shown below. Note that, although the parentheses indicate corresponding components in the following embodiments, the present disclosure is not limited to these.

[0008] (1) An imaging control device including a processor for controlling an imaging element having a plurality of pixels including a photoelectric conversion unit that converts light that is incident on an imaging optical system and collected by a microlens into an electric charge, wherein the plurality of pixels include a first pixel that shares the microlens with an adjacent pixel, and a second pixel that is different from the first pixel, and the processor controls the ratio between a first exposure time that is the exposure time of the first pixel and a second exposure time that is the exposure time of the second pixel based on a condition related to the angle of incidence of light on the pixel.

[0009] (2) The imaging control device according to (1), wherein the condition regarding the incident angle includes a condition regarding the imaging optical system.

[0010] (3) The imaging control device according to (2), wherein the condition related to the imaging optical system is a condition that determines the range of angles of incidence of light incident on the imaging element.

[0011] (4) The imaging control device according to (3), wherein the condition for determining the range of incident angles includes at least one of an aperture value of the imaging optical system, a focal length of the imaging optical system, a focal position of the imaging optical system, and a type of the imaging optical system.

[0012] (5) The imaging control device according to (4), wherein the condition for determining the incident angle range includes an aperture value of the imaging optical system, and the processor makes a first ratio, which is the ratio when the aperture value is a first aperture value, smaller than a second ratio, which is the ratio when the aperture value is a second aperture value that is larger than the first aperture value.

[0013] (6) The imaging control device according to any one of (1) to (5), wherein the condition regarding the incident angle includes a condition regarding the position of the pixel on the light receiving surface of the imaging element.

[0014] (7) The imaging control device according to (6), in which the position of the pixel is a position based on an image height of the imaging optical system.

[0015] (8) The imaging control device according to (6) or (7), wherein the processor differentiates a third ratio between the first exposure time and the second exposure time at the first pixel and the second pixel located at a first position from a fourth ratio between the first exposure time and the second exposure time at the first pixel and the second pixel located at a second position different from the first position.

[0016] (9) The imaging control device according to (8), wherein the first position is closer to the center of the light receiving surface of the imaging element than the second position, and the second ratio is smaller than the first ratio.

[0017] (10) An imaging control device described in any one of (2) to (5), wherein the processor controls a first ratio between the first exposure time and the second exposure time at the first pixel and the second pixel at a second position to a value based on the incident angle characteristics of the first pixel and the second pixel determined by the condition, and controls a second ratio between the first exposure time and the second exposure time at the first pixel and the second pixel at a first position closer to the center of the light receiving surface of the imaging element than the second position to the first ratio.

[0018] (11) An imaging control device according to any one of (2) to (5), wherein the processor controls a fifth ratio between the first exposure time and the second exposure time at the first pixel and the second pixel at a first position to a value based on the incident angle characteristics of the first pixel and the second pixel determined by the condition, and controls a sixth ratio between the first exposure time and the second exposure time at the first pixel and the second pixel at a second position closer to the edge of the light receiving surface of the imaging element than the first position to the fifth ratio.

[0019] (12) The imaging control device according to any one of (1) to (11), wherein the second pixel is a pixel that has the microlens alone.

[0020] (13) An imaging device including: the imaging control device according to any one of (1) to (12); and the imaging element.

[0021] (14) An imaging control method for controlling an imaging element having a plurality of pixels including a photoelectric conversion unit that converts light that is incident on an imaging optical system and collected by a microlens into an electric charge, wherein the plurality of pixels include a first pixel that shares the microlens with an adjacent pixel, and a second pixel that is different from the first pixel, and the imaging control method includes a step of controlling a ratio between a first exposure time that is an exposure time of the first pixel and a second exposure time that is an exposure time of the second pixel based on a condition related to an incident angle of light to the pixel.

[0022] (15) An imaging control program for controlling an imaging element having a plurality of pixels including a photoelectric conversion unit that converts light that is incident on an imaging optical system and collected by a microlens into an electric charge, wherein the plurality of pixels include a first pixel that shares the microlens with an adjacent pixel, and a second pixel that is different from the first pixel, and the imaging control program causes a processor to execute a step of controlling a ratio between a first exposure time that is an exposure time of the first pixel and a second exposure time that is an exposure time of the second pixel based on a condition related to an incident angle of light to the pixel.

[0023] FIG. 1 is a diagram showing a schematic configuration of a digital camera 100 that is an embodiment of an imaging device according to the technology of the present disclosure. FIG. 2 is a plan view schematic diagram showing a schematic configuration of the imaging element 5 shown in FIG. 1. FIG. 3 is a schematic diagram for explaining the angle of light incident on a pixel 61 when viewed in the column direction Y. FIG. 4 is a schematic diagram showing a partially enlarged light receiving surface 60 of the imaging element 5 shown in FIG. 2. FIG. 5 is a schematic cross-sectional view of a range A1 shown in FIG. 4. FIG. 6 is a schematic cross-sectional view of a range A2 shown in FIG. 4. FIG. 7 is a diagram showing output characteristics of a pixel 61G, a first phase difference detection pixel 61FA, and a second phase difference detection pixel 61FB. FIG. 8 is a diagram showing an example of a range HC when the F-number is smaller than that in FIG. 7. FIG. 9 is a diagram showing a first modified example of the imaging element 5 and corresponds to FIG. 4. FIG. 10 is a diagram showing a second modified example of the imaging element 5 and corresponds to FIG. 4. FIG. 11 shows the external appearance of a smartphone 200. FIG. 12 is a block diagram showing the configuration of the smartphone 200 shown in FIG.

[0024] Fig. 1 is a diagram showing a schematic configuration of a digital camera 100, which is an embodiment of an imaging device relating to the technique of the present disclosure. The digital camera 100 shown in Fig. 1 includes a lens device 40 having an imaging lens 1, an aperture 2, a lens driver 8 that drives the imaging lens 1, an aperture driver 9 that drives the aperture 2, and a lens controller 4 that controls the lens driver 8 and the aperture driver 9, and a main body 100A.

[0025] The main body 100A includes an image sensor 5, a system control unit 11 that controls the overall electrical control system of the digital camera 100, an operation unit 14, a display device 22, a memory 16 that includes RAM (Random Access Memory) and ROM (Read Only Memory), a memory control unit 15 that controls the storage of data in the memory 16 and the reading of data from the memory 16, a digital signal processing unit 17, and an external memory control unit 20 that controls the storage of data in a storage medium 21 and the reading of data from the storage medium 21.

[0026] The lens device 40 may be detachable from the main body 100 A, or may be integrated with the main body 100 A. The imaging lens 1 includes a focus lens.

[0027] The focus lens is a lens for adjusting the focus of an imaging optical system including the imaging lens 1 and the aperture 2, and is composed of a single lens or multiple lenses. When the focus lens moves in the optical axis direction, the position of the principal point of the focus lens (hereinafter also referred to as the focus lens position) changes along the optical axis, thereby changing the focal position on the subject side. Note that a liquid lens whose principal point position in the optical axis direction can be changed by electrical control may also be used as the focus lens.

[0028] The lens control unit 4 of the lens device 40 controls the lens drive unit 8 based on the lens drive signal sent from the system control unit 11 to change the focus lens position. The lens control unit 4 of the lens device 40 controls the diaphragm drive unit 9 based on the drive control signal sent from the system control unit 11 to change the aperture size (hereinafter, the size is expressed as the F-number) of the diaphragm 2. The larger the F-number, the larger the aperture size of the diaphragm 2 (the smaller the opening size of the diaphragm 2).

[0029] The image sensor 5 captures an image of the subject through the imaging optical system provided between the image sensor 5 and the subject. The image sensor 5 has a light-receiving surface 60 (see FIG. 2 ) on which a plurality of pixels are arranged two-dimensionally, and converts an image of the subject formed on the light-receiving surface 60 by the imaging optical system into an image signal using the plurality of pixels, and outputs the image signal. The output of the pixels included in the image sensor 5 is referred to as a pixel signal, and a collection of pixel signals is referred to as an image signal.

[0030] For example, a CMOS (complementary metal-oxide semiconductor) image sensor is used as the imaging element 5. The imaging element 5 is driven by a driver (not shown), and this driver is controlled by the system control unit 11.

[0031] The system control unit 11 controls the entire digital camera 100, and its hardware configuration consists of various processors that execute programs, including an imaging control program, and processes the programs. The programs executed by the system control unit 11 are stored in ROM (non-transitory storage medium) of the memory 16.

[0032] The various types of processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs to perform various processes, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), or a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to perform specific processing. More specifically, the structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0033] The system control unit 11 may be configured with one of various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0034] The system control unit 11 controls the image sensor 5 and the lens device 40 in accordance with an imaging control program, and outputs a subject image captured through the imaging optical system of the lens device 40 as a group of pixel signals. The system control unit 11 and the memory 16 constitute an imaging control device. The group of pixel signals output from the image sensor 5 are processed by the digital signal processing unit 17, and captured image data is generated, which is data suitable for display on the display device 22 or data suitable for storage in the storage medium 21.

[0035] An instruction signal from the user is input to the system control unit 11 through the operation unit 14. The operation unit 14 includes a touch panel integrated with the display surface 22b, various buttons, and the like.

[0036] The display device 22 includes a display surface 22b configured by an organic EL (electroluminescence) panel, a liquid crystal panel, or the like, and a display controller 22a that controls the display on the display surface 22b.

[0037] The memory control unit 15 , digital signal processing unit 17 , external memory control unit 20 , and display controller 22 a are interconnected by a control bus 24 and a data bus 25 , and are controlled by commands from the system control unit 11 .

[0038] 2 is a plan view schematically illustrating the configuration of the image sensor 5 shown in FIG. 1 . The image sensor 5 includes a light-receiving surface 60 on which pixel rows 62, each consisting of a plurality of pixels 61 arranged in a row direction X, are arranged in a column direction Y intersecting the row direction X, a drive circuit 63 that drives the pixels 61 arranged on the light-receiving surface 60, and a signal processing circuit 64 that processes pixel signals read out to signal lines from each pixel 61 in the pixel rows 62 arranged on the light-receiving surface 60. In the example of FIG. 2 , the row direction X and the column direction Y are orthogonal to each other. One side of the row direction X is referred to as the right direction XR, and the other side of the row direction X is referred to as the left direction XL. One side of the column direction Y is referred to as the upward direction YU, and the other side of the column direction Y is referred to as the downward direction YD.

[0039] FIG. 3 is a schematic diagram illustrating the angle of light incident on a pixel 61 when viewed in the column direction Y. A line L in the figure represents a line parallel to the optical axis of the imaging optical system or a line perpendicular to the light-receiving surface 60. Hereinafter, the angle between the line L and a ray of light incident on the pixel 61 (indicated by a dashed arrow in FIG. 3 ) is defined as the angle of incidence θ of the light on the pixel 61. The angle of incidence θ between the line L and light incident obliquely from the right direction XR with respect to the line L is defined as a positive value, and the angle between the line L and light incident obliquely from the left direction XL with respect to the line L is defined as a negative value. The absolute value of the angle of incidence θ is larger at the right and left ends of the light-receiving surface 60 than at the center of the light-receiving surface 60 (near the point where the light intersects with the optical axis of the imaging optical system). In other words, if the position of pixel 61 at the intersection with the optical axis on light receiving surface 60 is taken as the reference position, the absolute value of the incident angle θ of pixel 61 increases as the position of pixel 61 moves away from the reference position in the row direction X.

[0040] 4 is a partially enlarged schematic diagram of the light receiving surface 60 of the image sensor 5 shown in FIG. 2 . The plurality of pixels 61 arranged on the light receiving surface 60 includes pixels corresponding to each of a plurality of wavelength bands (three in this embodiment). Specifically, the light receiving surface 60 includes a pixel 61R (a block marked with "R" in the figure) corresponding to the wavelength band of red light, a pixel 61G (a block marked with "G" in the figure) corresponding to the wavelength band of green light, a pixel 61B (a block marked with "B" in the figure) corresponding to the wavelength band of blue light, a first phase difference detection pixel 61FA (a block marked with "FA" in the figure) corresponding to the wavelength band of green light, and a second phase difference detection pixel 61FB (a block marked with "FB" in the figure) corresponding to the wavelength band of green light. Each pixel 61 arranged on the light receiving surface 60 receives light of the corresponding wavelength band and outputs a pixel signal corresponding to the amount of light.

[0041] Pixels 61R, 61G, and 61B are arranged based on a Bayer pattern on the light-receiving surface 60. That is, on the light-receiving surface 60, RG pixel rows in which pixels 61R and pixels 61G are alternately arranged in the row direction X, and GB pixel rows in which pixels 61G and pixels 61B are alternately arranged in the row direction X are alternately arranged in the column direction Y. The arrangement pattern of the pixels 61 arranged on the light-receiving surface 60 is not limited to the Bayer pattern, and various arrangement patterns can be adopted.

[0042] In some GB pixel rows among these multiple pixel rows, some pixels 61B are replaced with first phase difference detection pixels 61FA, and some pixels 61G are replaced with second phase difference detection pixels 61FB, with the first phase difference detection pixels 61FA and the second phase difference detection pixels 61FB adjacent to each other. In these GB pixel rows, a plurality of phase difference detection pairs, each consisting of a first phase difference detection pixel 61FA and its adjacent second phase difference detection pixel 61FB, are arranged side by side at intervals in the row direction X. These GB pixel rows may be composed solely of phase difference detection pairs. The phase difference detection pairs may also be provided in the RG pixel rows. In this case, the adjacent pixels 61G and 61R may be replaced with the phase difference detection pair.

[0043] The system control unit 11 performs focus detection processing based on the pixel signals of the first phase difference detection pixels 61FA and the second phase difference detection pixels 61FB. Specifically, the system control unit 11 detects a phase difference by performing a correlation calculation between a group of pixel signals output from the first phase difference detection pixels 61FA and a group of pixel signals output from the second phase difference detection pixels 61FB included in the same pixel row, and performs focus detection processing to derive a focus lens position required to focus on a target subject based on the phase difference.

[0044] Fig. 5 is a schematic cross-sectional view of range A1 shown in Fig. 4. Fig. 6 is a schematic cross-sectional view of range A2 shown in Fig. 4. As shown in Fig. 5 and Fig. 6, each pixel 61 provided on the light receiving surface 60 includes a microlens ML that collects light from a subject, a photoelectric conversion unit PD that converts the light collected by the microlens ML into an electric charge, and a color filter CF that is provided between the photoelectric conversion unit PD and the microlens ML and that transmits light in a specific wavelength band.

[0045] Although not shown, each pixel 61 is provided with a readout circuit that converts the charges generated in the photoelectric conversion unit PD into a pixel signal and reads it out. Furthermore, a light-shielding film that defines the light-receiving area of ​​the photoelectric conversion unit PD and a light-shielding film that shields the readout circuit disposed close to the photoelectric conversion unit PD from light are provided between the photoelectric conversion unit PD and the color filter CF. The photoelectric conversion unit PD is a photodiode formed in a semiconductor substrate such as silicon, but may also be composed of an organic material film disposed above the semiconductor substrate.

[0046] The color filter CF (referred to as an R filter in FIG. 5 ) included in the pixel 61R transmits red light, the color filter CF (referred to as a G filter in FIGS. 5 and 6 ) included in the pixel 61G transmits green light, and the color filter CF (referred to as a B filter in FIG. 6 ) included in the pixel 61B transmits blue light. The color filters CF included in each of the first phase difference detection pixel 61FA and the second phase difference detection pixel 61FB are G filters that transmit green light, but are not limited to this. For example, they may be R filters or B filters, neutral density filters or white filters, or a configuration without a color filter CF.

[0047] The first phase difference detection pixel 61FA shares a microlens ML with its adjacent second phase difference detection pixel 61FB. That is, the microlens ML included in the first phase difference detection pixel 61FA and the microlens ML included in the adjacent second phase difference detection pixel 61FB are the same. The planar shape of the microlens ML provided in the phase difference detection pair is an ellipse with its major axis extending in the row direction X, as partially shown in FIG. 4 . This ellipse-shaped microlens ML is provided above two photoelectric conversion units PD arranged in the row direction X in the phase difference detection pair, straddling the two photoelectric conversion units PD. On the other hand, the pixel 61R, the pixel 61G, and the pixel 61B each have their own microlens ML, and the microlens ML is not shared with other pixels 61. It can also be said that the phase difference detection pair shares a microlens ML. The first phase difference detection pixel 61FA and the second phase difference detection pixel 61FB each constitute a first pixel. The pixel 61R, the pixel 61G, and the pixel 61B each constitute a second pixel different from the first pixel. It can also be said that the second pixel has optical characteristics different from those of the first pixel.

[0048] 7 is a diagram showing output characteristics (incident angle characteristics) with respect to the incident angle of the pixel 61G, the first phase difference detection pixel 61FA, and the second phase difference detection pixel 61FB. The horizontal axis in the diagram represents the incident angle θ of light to the pixel 61, and the vertical axis represents the level of the pixel signal output from the pixel 61 (hereinafter referred to as pixel value). The pixel value represents a value normalized using the pixel value of the pixel 61G when the incident angle θ is 0 degrees as the reference (=1). A characteristic Cfa in the diagram represents the output characteristic of the first phase difference detection pixel 61FA. A characteristic Cfb in the diagram represents the output characteristic of the second phase difference detection pixel 61FB. A characteristic Cg in the diagram represents the output characteristic of the pixel 61G.

[0049] 7 , the output characteristics of the pixel 61G are such that the pixel value is maximum when the incident angle θ is 0 degrees, and the pixel value gradually decreases as the absolute value of the incident angle θ increases. The output characteristics of the first phase difference detection pixel 61FA are such that the pixel value is greater in the positive range of the incident angle θ than in the negative range of the incident angle θ, and the pixel value is maximum when the incident angle θ is around 23 degrees. Furthermore, the output characteristics of the first phase difference detection pixel 61FA are greater than the maximum pixel value of the pixel 61G in the positive range of the incident angle θ.

[0050] The characteristic Cfb is obtained by inverting the characteristic Cfa across the vertical axis. The output characteristic of the second phase difference detection pixel 61FB is such that the pixel value is larger in the negative range of the incident angle θ than in the positive range of the incident angle θ, and the pixel value is maximum when the incident angle θ is around −23 degrees. Furthermore, the output characteristic of the second phase difference detection pixel 61FB is larger than the maximum pixel value of the pixel 61G in the negative range of the incident angle θ.

[0051] Although not shown, the output characteristics of the pixel 61R and the output characteristics of the pixel 61B are symmetrical with respect to the vertical axis, similar to the characteristic Cg, and the pixel value is maximum when the incident angle θ is 0 degrees, and the pixel value gradually decreases as the absolute value of the incident angle θ increases. Hereinafter, when the pixels 61R, 61G, and 61B are not distinguished from each other, they will be referred to as normal pixels 61, and when the first phase difference detection pixel 61FA and the second phase difference detection pixel 61FB are not distinguished from each other, they will be referred to as phase difference pixels 61.

[0052] A range HC shown in Fig. 7 indicates the range of possible incident angles θ (hereinafter also referred to as the incident angle range) of light incident on a pixel 61 located at the center in the row direction X of the light-receiving surface 60. A range HR shown in Fig. 7 indicates the range of possible incident angles of light incident on a pixel 61 located at the right end of the light-receiving surface 60. A range HL shown in Fig. 7 indicates the range of possible incident angles of light incident on a pixel 61 located at the left end of the light-receiving surface 60.

[0053] In this specification, a pixel 61 located in the center of the light-receiving surface 60 is referred to as a pixel 61 at a first position. Furthermore, pixels 61 located at the right and left ends of the light-receiving surface 60 are referred to as pixels 61 at a second position different from the first position. The first position is a position closer to the center of the light-receiving surface 60 of the image sensor 5 than the second position. The second position is a position closer to the edge of the light-receiving surface 60 than the first position.

[0054] The widths of the ranges HC, HR, and HL vary depending on the combination of the type, F-number, focal length, and focus lens position (focal position) of the lens device 40. For example, as the F-number decreases (the aperture size increases), the widths of the ranges HC, HR, and HL increase, whereas as the F-number increases (the aperture size decreases), the widths of the ranges HC, HR, and HL decrease. The center positions of the widths of the ranges HC, HR, and HL can also vary depending on the combination of the type, F-number, focal length, and focus lens position of the lens device 40. The type, F-number, focal length, and focus lens position of the lens device 40 are conditions that determine the range of incidence angles of light incident on the image sensor 5 and are conditions related to the imaging optical system. This condition related to the imaging optical system is one of the conditions related to the incidence angle of light on the pixels 61.

[0055] Fig. 8 is a diagram showing an example of the range HC when the F-number is smaller than that in Fig. 7. As can be seen from comparing Fig. 7 and Fig. 8, as the F-number becomes smaller, the difference between the pixel values ​​of the normal pixels 61 and the pixel values ​​of the phase difference pixels 61 becomes larger. For example, consider a case where the center of the light receiving surface 60 is set as the focus detection area, or where the phase difference pixels 61 are arranged only in this center.

[0056] 7 , the ratio of the pixel values ​​of the phase difference pixels 61 to the pixel values ​​of the normal pixels 61 is small, at a maximum of about 1.3, in the central part of the light receiving surface 60. Therefore, even if the exposure time of each pixel 61 of the image sensor 5 is set so that the normal pixels 61 are properly exposed, the pixel values ​​of the phase difference pixels 61 in the central part are unlikely to be saturated. Therefore, focus detection processing using the pixel values ​​of the phase difference pixels 61 in the central part can be performed with high accuracy.

[0057] On the other hand, when the F-number is set to fall within the range HC shown in FIG. 8 , the ratio of the pixel values ​​of the phase difference pixels 61 to the pixel values ​​of the normal pixels 61 becomes large, up to approximately 1.7 times. Therefore, if the exposure time of each pixel 61 of the image sensor 5 is set so that the normal pixels 61 are properly exposed, the pixel values ​​of the phase difference pixels 61 may become saturated. Therefore, in this embodiment, the system control unit 11 controls the ratio between the first exposure time, which is the exposure time of the phase difference pixels 61, and the second exposure time, which is the exposure time of the normal pixels 61 (hereinafter referred to as the exposure time ratio), based on the F-number, thereby preventing the pixel values ​​of the phase difference pixels 61 from becoming saturated. The exposure time ratio is defined as the value obtained by dividing the first exposure time by the second exposure time.

[0058] For example, the system control unit 11 controls the exposure time ratio to 1.0 when the F-number is set to a range HC shown in Fig. 7 (e.g., F8.0), and controls the exposure time ratio to 0.5 when the F-number is set to a range HC shown in Fig. 8 (e.g., F2.0). If the exposure time for proper exposure of the normal pixels 61 is 1 / 100 seconds regardless of the F-number, the exposure time of the phase difference pixels 61 is controlled to 1 / 100 seconds when the F-number is set as shown in Fig. 7, and the exposure time of the phase difference pixels 61 is controlled to 1 / 200 seconds when the F-number is set as shown in Fig. 8. Since the width of the range HC increases as the F-number decreases, the system control unit 11 controls the exposure time ratio to decrease as the F-number decreases. In other words, the system control unit 11 sets a first ratio, which is the exposure time ratio when the aperture stop amount is a first aperture stop amount (e.g., F2.0), to be smaller than a second ratio, which is the exposure time ratio when the aperture stop amount is a second aperture stop amount (e.g., F8.0) that is larger than the first aperture stop amount. In this way, it is possible to prevent the pixel values ​​of the phase difference pixels 61 used in focus detection processing from becoming saturated while maintaining appropriate exposure of the normal pixels 61.

[0059] As described above, the width and position of the range HC can vary depending on the combination of the F-number, the type of lens device 40, the focal length, and the focus lens position. Therefore, the system control unit 11 controls the exposure time ratio based on the conditions of this combination, thereby enabling high-precision focus detection processing to be performed regardless of the conditions related to the imaging optical system.

[0060] 7 , even if the conditions related to the imaging optical system are the same, the maximum value of the ratio of the pixel values ​​of the phase difference pixels 61 to the normal pixels 61 changes depending on the position of the pixel 61 on the light receiving surface 60. The position of the pixel 61 on the light receiving surface 60 is a position based on the image height of the imaging optical system, and can be expressed as the distance from the intersection point between the optical axis of the imaging optical system and the light receiving surface 60 to the pixel 61.

[0061] For example, at the right and left ends of the light receiving surface 60, which are far from the reference point, the maximum ratio of the pixel values ​​of the phase difference pixels 61 to the normal pixels 61 is approximately 2.3 times, whereas at the center of the light receiving surface 60, which is far from the reference point, the maximum ratio of the pixel values ​​of the phase difference pixels 61 to the normal pixels 61 is approximately 1.3 times.

[0062] Therefore, for example, the system control unit 11 sets the exposure time ratio to a first ratio for normal pixels 61 and phase difference pixels 61 located in a range (center) where the distance from the reference point is equal to or less than the threshold value TH, and sets the exposure time ratio to a second ratio smaller than the first ratio for normal pixels 61 and phase difference pixels 61 located in a range (right and left ends) where the distance from the reference point exceeds the threshold value TH. By doing so, even when the entire light receiving surface 60 is set as the focus detection area, it is possible to properly expose the normal pixels 61 while preventing the pixel values ​​of the phase difference pixels 61 used in the focus detection process from becoming saturated, thereby improving the accuracy of the focus detection process. In this way, the system control unit 11 may control the exposure time ratio based on conditions related to the positions of the pixels 61 on the light receiving surface 60.

[0063] In the above description, the exposure time ratio is set to be different between the normal pixels 61 and the phase difference pixels 61 whose distance from the reference point exceeds the threshold value TH and the normal pixels 61 and the phase difference pixels 61 whose distance from the reference point is equal to or less than the threshold value TH. However, this is not limiting. For example, an exposure time ratio determined based on the incident angle characteristics of the normal pixels 61 and the phase difference pixels 61 whose distance from the reference point is equal to or less than the threshold value TH may be applied to the entire light receiving surface 60.

[0064] The normal pixels 61 and phase difference pixels 61 at the right and left ends tend to receive a smaller amount of incident light than the normal pixels 61 and phase difference pixels 61 at the center. For this reason, even if an exposure time ratio is determined so that the pixel values ​​of the phase difference pixels 61 at the center do not saturate and this exposure time ratio is applied to the pixels 61 at the right and left ends, it is possible to prevent the pixel values ​​of the phase difference pixels 61 at the right and left ends from saturating, depending on the pixel structure of the image sensor 5. In this way, unifying the exposure time ratio over the entire light receiving surface 60 makes it easier to control the image sensor 5.

[0065] Furthermore, contrary to the above explanation, an exposure time ratio determined based on the incident angle characteristics of the normal pixels 61 and the phase difference pixels 61 whose distance from the reference point exceeds the threshold value TH may be applied to the entire light receiving surface 60.

[0066] Fig. 9 is a diagram showing a first modified example of the image sensor 5 and corresponds to Fig. 4. The image sensor 5 shown in Fig. 9 has a configuration in which the position of the second phase difference detection pixel 61FB constituting the phase difference detection pair in the image sensor 5 shown in Fig. 4 is changed to be adjacent to the first phase difference detection pixel 61FA constituting the phase difference detection pair in the downward direction YD, and a pixel 61G is disposed in the position originally occupied by the second phase difference detection pixel 61FB. In the example shown in Fig. 9, the major axis direction of the microlens ML included in the phase difference detection pair coincides with the column direction Y.

[0067] Even with the configuration shown in FIG. 9 , if the row direction X in FIG. 3 is replaced with the column direction Y to define the incident angle θ, the relationship between the phase difference detection pair and the output characteristics of the pixel 61G will be the same as that shown in FIG. 7 .

[0068] Fig. 10 is a diagram illustrating a second modified example of the image sensor 5, and corresponds to Fig. 4. The image sensor 5 illustrated in Fig. 10 has a configuration in which the pixel 61G and the pixel 61R adjacent to the phase difference detection pair in the downward direction YD in the image sensor 5 illustrated in Fig. 4 are replaced with a third phase difference detection pixel 61FC and a fourth phase difference detection pixel 61FD.

[0069] 10 , a microlens ML is shared by four pixels 61, namely, a first phase difference detection pixel 61FA, a second phase difference detection pixel 61FB, a third phase difference detection pixel 61FC, and a fourth phase difference detection pixel 61FD, and a phase difference detection group is configured by these four pixels 61. On the light receiving surface 60, a plurality of group rows, each configured with a plurality of phase difference detection groups lined up in the row direction X, are arranged in the column direction Y. It can also be said that on the light receiving surface 60, a plurality of group columns, each configured with a plurality of phase difference detection groups lined up in the column direction Y, are arranged in the row direction X.

[0070] The system control unit 11 derives a first average value of the pixel values ​​of the first phase difference detection pixel 61FA and the third phase difference detection pixel 61FC in the phase difference detection group, and derives a second average value of the pixel values ​​of the second phase difference detection pixel 61FB and the fourth phase difference detection pixel 61FD in the phase difference detection group. The system control unit 11 detects the phase difference in the row direction X by performing a correlation calculation between the group of first average values ​​and the group of second average values ​​derived for each phase difference detection group in the same group row.

[0071] The system control unit 11 derives a third average value of the pixel values ​​of the first phase difference detection pixel 61FA and the second phase difference detection pixel 61FB in the phase difference detection group, and derives a fourth average value of the pixel values ​​of the third phase difference detection pixel 61FC and the fourth phase difference detection pixel 61FD in the phase difference detection group. The system control unit 11 detects the phase difference in the column direction Y by performing a correlation calculation between the group of third average values ​​and the group of fourth average values ​​derived for each phase difference detection group in the same group column.

[0072] The change in the first average value with respect to the incident angle θ is equivalent to the characteristic Cfa in Fig. 7. The change in the second average value with respect to the incident angle θ is equivalent to the characteristic Cfb in Fig. 7.

[0073] Next, the configuration of a smartphone, which is another embodiment of the imaging device of the present invention, will be described.

[0074] Fig. 11 shows the external appearance of a smartphone 200. The smartphone 200 shown in Fig. 11 has a flat housing 201, and is provided with a display input unit 204 on one side of the housing 201, which is an integrated unit of a display panel 202 as a display unit and an operation panel 203 as an input unit.

[0075] Such a housing 201 also includes a speaker 205, a microphone 206, an operation unit 207, and a camera unit 208. Note that the configuration of the housing 201 is not limited to this, and for example, it is also possible to adopt a configuration in which the display unit and the input unit are independent, or a configuration having a folding structure or a sliding mechanism.

[0076] FIG. 12 is a block diagram showing the configuration of the smartphone 200 shown in FIG.

[0077] As shown in FIG. 12 , the main components of the smartphone include a wireless communication unit 210, a display input unit 204, a call unit 211, an operation unit 207, a camera unit 208, a memory unit 212, an external input / output unit 213, a GNSS (Global Navigation Satellite System) receiving unit 214, a motion sensor unit 215, a power supply unit 216, and a main control unit 220.

[0078] The smartphone 200 also has, as its main function, a wireless communication function for performing mobile wireless communication via a base station device BS (not shown) and a mobile communication network NW (not shown).

[0079] The wireless communication unit 210 performs wireless communication with a base station device BS accommodated in the mobile communication network NW in accordance with instructions from the main control unit 220. Using this wireless communication, various file data such as audio data and image data, e-mail data, etc. are sent and received, and web data, streaming data, etc. are received.

[0080] The display input unit 204 is a so-called touch panel that, under the control of the main control unit 220, displays images (still images and moving images) or text information, etc. to visually convey information to the user and detects user operations on the displayed information, and is equipped with a display panel 202 and an operation panel 203.

[0081] The display panel 202 uses a liquid crystal display (LCD), an organic electroluminescence display (OELD), or the like as a display device.

[0082] The operation panel 203 is a device placed so that an image displayed on the display surface of the display panel 202 can be seen, and detects one or more coordinates operated by a user's finger or a stylus. When this device is operated by the user's finger or a stylus, a detection signal generated by the operation is output to the main control unit 220. Next, the main control unit 220 detects the operation position (coordinates) on the display panel 202 based on the received detection signal.

[0083] As shown in Figure 12, the display panel 202 and operation panel 203 of a smartphone 200, which is an example of one embodiment of the imaging device of the present invention, are integrated to form a display input unit 204, and the operation panel 203 is positioned so that it completely covers the display panel 202.

[0084] When such an arrangement is adopted, operation panel 203 may also have a function for detecting user operations in areas outside display panel 202. In other words, operation panel 203 may have a detection area for the overlapping portion that overlaps display panel 202 (hereinafter referred to as a display area), and a detection area for the remaining outer edge portion that does not overlap display panel 202 (hereinafter referred to as a non-display area).

[0085] The size of the display area and the size of the display panel 202 may be exactly the same, but they do not necessarily have to be the same. The operation panel 203 may also have two sensitive areas: an outer edge portion and an inner portion. The width of the outer edge portion is designed appropriately depending on the size of the housing 201, etc.

[0086] Furthermore, the position detection method used in the operation panel 203 may be a matrix switch method, a resistive film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, a capacitance method, or the like, and any method may be used.

[0087] The call unit 211 is equipped with a speaker 205 or a microphone 206, and converts the user's voice input through the microphone 206 into voice data that can be processed by the main control unit 220 and outputs it to the main control unit 220, or decodes voice data received by the wireless communication unit 210 or the external input / output unit 213 and outputs it from the speaker 205.

[0088] Also, as shown in FIG. 11, for example, the speaker 205 can be mounted on the same surface as the display input unit 204, and the microphone 206 can be mounted on the side of the housing 201.

[0089] The operation unit 207 is a hardware key using a key switch or the like, and receives instructions from a user. For example, as shown in Fig. 11 , the operation unit 207 is a push-button switch mounted on the side surface of the housing 201 of the smartphone 200, which turns on when pressed with a finger or the like, and turns off when the finger is released due to the restoring force of a spring or the like.

[0090] The storage unit 212 stores the control program and control data of the main control unit 220, application software, address data associated with names or telephone numbers of communication partners, data of sent and received e-mails, web data downloaded by web browsing, downloaded content data, and also temporarily stores streaming data, etc. The storage unit 212 is composed of an internal storage unit 217 built into the smartphone and an external storage unit 218 having a removable external memory slot.

[0091] The internal memory unit 217 and the external memory unit 218 constituting the memory unit 212 are realized using storage media such as a flash memory type, a hard disk type, a multimedia card micro type, a card-type memory (e.g., MicroSD (registered trademark) memory, etc.), a RAM (Random Access Memory), a ROM (Read Only Memory), etc.

[0092] The external input / output unit 213 serves as an interface with all external devices connected to the smartphone 200, and is used to directly or indirectly connect to other external devices via communication (e.g., Universal Serial Bus (USB), IEEE 1394, Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) (registered trademark), UWB (Ultra Wideband) (registered trademark), ZigBee (registered trademark), etc.) or a network (e.g., Ethernet (registered trademark), wireless LAN (Local Area Network), etc.).

[0093] Examples of external devices connected to the smartphone 200 include a wired / wireless headset, a wired / wireless external charger, a wired / wireless data port, a memory card connected via a card socket, a SIM (Subscriber Identity Module Card) / UIM (User Identity Module Card) card, an external audio / video device connected via an audio / video I / O (Input / Output) terminal, a wirelessly connected external audio / video device, a wired / wirelessly connected smartphone, a wired / wirelessly connected personal computer, a wired / wirelessly connected personal computer, and earphones.

[0094] The external input / output unit 213 can transmit data received from such external devices to each component inside the smartphone 200, or transmit data inside the smartphone 200 to external devices.

[0095] The GNSS receiving unit 214 receives GNSS signals transmitted from the GNSS satellites ST1 to STn in accordance with instructions from the main control unit 220, executes positioning calculation processing based on the received multiple GNSS signals, and detects a position consisting of the latitude, longitude, and altitude of the smartphone 200. When the GNSS receiving unit 214 can acquire position information from the wireless communication unit 210 or the external input / output unit 213 (for example, a wireless LAN), it can also detect the position using the position information.

[0096] The motion sensor unit 215 includes, for example, a three-axis acceleration sensor, and detects the physical movement of the smartphone 200 in accordance with instructions from the main control unit 220. By detecting the physical movement of the smartphone 200, the direction of movement or acceleration of the smartphone 200 is detected. The detection result is output to the main control unit 220.

[0097] The power supply unit 216 supplies power stored in a battery (not shown) to each unit of the smartphone 200 in accordance with instructions from the main control unit 220 .

[0098] The main control unit 220 includes a microprocessor, operates according to the control program and control data stored in the storage unit 212, and controls all the units of the smartphone 200. The microprocessor of the main control unit 220 has the same functions as the system control unit 11. The main control unit 220 also includes a mobile communication control function that controls all the units of the communication system to perform voice communication or data communication via the wireless communication unit 210, and an application processing function.

[0099] The application processing function is realized by the main control unit 220 operating in accordance with the application software stored in the storage unit 212. Examples of the application processing function include an infrared communication function that controls the external input / output unit 213 to perform data communication with a connected device, an email function that sends and receives emails, and a web browsing function that views web pages.

[0100] The main control unit 220 also has an image processing function for displaying video on the display input unit 204 based on image data (still image or moving image data) such as received data or downloaded streaming data.

[0101] The image processing function refers to a function in which the main control unit 220 decodes the image data, performs image processing on the decoded result, and displays the image on the display input unit 204 .

[0102] Furthermore, the main control unit 220 executes display control for the display panel 202 and operation detection control for detecting user operations via the operation unit 207 and the operation panel 203 .

[0103] By executing display control, the main control unit 220 displays software keys such as icons or scroll bars for starting application software, or displays a window for creating an e-mail.

[0104] The scroll bar refers to a software key for receiving an instruction to move the displayed portion of an image, such as a large image that cannot fit in the display area of ​​the display panel 202 .

[0105] In addition, by executing operation detection control, the main control unit 220 detects user operations through the operation unit 207, accepts operations on the above icons and input of character strings into the input field of the above window through the operation panel 203, or accepts requests to scroll the displayed image through the scroll bar.

[0106] Furthermore, by executing operation detection control, the main control unit 220 determines whether the operation position on the operation panel 203 is an overlapping portion (display area) that overlaps the display panel 202 or an outer edge portion (non-display area) that does not overlap the display panel 202, and is equipped with a touch panel control function that controls the sensitive area of ​​the operation panel 203 or the display position of the software key.

[0107] The main control unit 220 can also detect a gesture operation on the operation panel 203 and execute a preset function in response to the detected gesture operation.

[0108] Gesture operation is not a simple touch operation as in the past, but rather refers to an operation of drawing a path with a finger or the like, specifying multiple positions simultaneously, or combining these to draw a path for at least one of multiple positions.

[0109] The camera unit 208 includes the lens device 40, the image sensor 5, and the digital signal processing unit 17 shown in FIG.

[0110] The captured image data generated by the camera unit 208 can be stored in the storage unit 212 or output via the external input / output unit 213 or the wireless communication unit 210 .

[0111] In the smartphone 200 shown in FIG. 12 , the camera unit 208 is mounted on the same surface as the display input unit 204, but the mounting position of the camera unit 208 is not limited to this, and it may be mounted on the back surface of the display input unit 204.

[0112] The camera unit 208 can also be used for various functions of the smartphone 200. For example, an image acquired by the camera unit 208 can be displayed on the display panel 202, or an image from the camera unit 208 can be used as one of the operation inputs for the operation panel 203.

[0113] Furthermore, when the GNSS receiving unit 214 detects a position, it can also detect the position by referring to an image from the camera unit 208. Furthermore, it can also refer to an image from the camera unit 208 to determine the optical axis direction of the camera unit 208 of the smartphone 200 or determine the current usage environment without using a triaxial acceleration sensor or by using the image in combination with a triaxial acceleration sensor. Of course, the image from the camera unit 208 can also be used in application software.

[0114] In addition, image data of still images or videos can be added with location information acquired by the GNSS receiving unit 214, audio information acquired by the microphone 206 (which may be converted to text information by the main control unit, etc.), posture information acquired by the motion sensor unit 215, etc., and stored in the memory unit 212 or output via the external input / output unit 213 or wireless communication unit 210.

[0115] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0116] This application is based on a Japanese patent application (Patent Application No. 2024-051058) filed on March 27, 2024, the contents of which are incorporated herein by reference.

[0117] REFERENCE SIGNS LIST 1 Imaging lens A1, A2 Range 2 Aperture 4 Lens control unit 5 Imaging element 8 Lens driving unit 9 Aperture driving unit 11 System control unit 14, 207 Operation unit 15 Memory control unit 16 Memory 17 Digital signal processing unit 20 External memory control unit 21 Storage medium 22 Display device 22a Display controller 22b Display surface 24 Control bus 25 Data bus 40 Lens device 60 Light receiving surface 61, 61B, 61G, 61R Pixel 61FA First phase difference detection pixel 61FB Second phase difference detection pixel 61FC Third phase difference detection pixel 61FD Fourth phase difference detection pixel 62 Pixel row 63 Drive circuit 64 Signal processing circuit 100 Digital camera 100A Main body 200 Smartphone 201 Housing 202 Display panel 203 Operation panel 204 Display input unit 205 Speaker 206 Microphone 208 Camera unit 210 Wireless communication unit 211 Call unit 212 Memory unit 213 External input / output unit 214 GNSS receiving unit 215 Motion sensor unit 216 Power supply unit 217 Internal memory unit 218 External memory unit 220 Main control unit

Claims

1. An imaging control device comprising a processor that controls an imaging element having a plurality of pixels, each including a photoelectric conversion unit that converts light that enters an imaging optical system and is focused by a microlens into an electric charge, wherein the plurality of pixels include a first pixel that shares the microlens with an adjacent pixel, and a second pixel that is different from the first pixel, and the processor controls the ratio between a first exposure time that is the exposure time of the first pixel and a second exposure time that is the exposure time of the second pixel, based on conditions related to the angle of incidence of light on the pixels.

2. An imaging control device according to claim 1, wherein the condition relating to the incident angle includes a condition relating to the imaging optical system.

3. An imaging control device according to claim 2, wherein the condition relating to the imaging optical system is a condition that determines the range of angles of incidence of light incident on the imaging element.

4. An imaging control device according to claim 3, wherein the conditions for determining the range of incident angles include at least one of the aperture value of the imaging optical system, the focal length of the imaging optical system, the focal position of the imaging optical system, and the type of the imaging optical system.

5. An imaging control device according to claim 4, wherein the conditions for determining the range of incident angles include the aperture value of the imaging optical system, and the processor makes the first ratio, which is the ratio when the aperture value is a first aperture value, smaller than the second ratio, which is the ratio when the aperture value is a second aperture value that is larger than the first aperture value.

6. An imaging control device according to claim 1, wherein the condition relating to the angle of incidence includes a condition relating to the position of the pixel on the light receiving surface of the imaging element.

7. An imaging control device according to claim 6, wherein the pixel positions are positions based on the image height of the imaging optical system.

8. An imaging control device according to claim 6, wherein the processor differentiates a third ratio between the first exposure time and the second exposure time for the first pixel and the second pixel at a first position from a fourth ratio between the first exposure time and the second exposure time for the first pixel and the second pixel at a second position different from the first position.

9. An imaging control device according to claim 8, wherein the first position is closer to the center of the light receiving surface of the imaging element than the second position, and the second ratio is smaller than the first ratio.

10. An imaging control device as described in claim 2, wherein the processor controls a first ratio between the first exposure time and the second exposure time at the first pixel and the second pixel located at a second position to a value based on the incident angle characteristics of the first pixel and the second pixel determined by the conditions, and controls a second ratio between the first exposure time and the second exposure time at the first pixel and the second pixel located at a first position closer to the center of the light receiving surface of the imaging element than the second position to the first ratio.

11. An imaging control device as described in claim 2, wherein the processor controls a fifth ratio between the first exposure time and the second exposure time at the first pixel and the second pixel at a first position to a value based on the incident angle characteristics of the first pixel and the second pixel determined by the condition, and controls a sixth ratio between the first exposure time and the second exposure time at the first pixel and the second pixel at a second position closer to the edge of the light receiving surface of the imaging element than the first position to the fifth ratio.

12. An imaging control device according to claim 1, wherein the second pixel is a pixel having the microlens alone.

13. An imaging device comprising: an imaging control device according to any one of claims 1 to 12; and the imaging element.

14. An imaging control method for controlling an imaging element having a plurality of pixels, each including a photoelectric conversion unit that converts light that has entered an imaging optical system and been collected by a microlens into an electric charge, wherein the plurality of pixels include a first pixel that shares the microlens with an adjacent pixel, and a second pixel that is different from the first pixel, and the imaging control method includes a step of controlling the ratio between a first exposure time that is the exposure time of the first pixel and a second exposure time that is the exposure time of the second pixel, based on conditions related to the angle of incidence of light on the pixels.

15. An imaging control program for controlling an imaging element having a plurality of pixels, each including a photoelectric conversion unit that converts light incident on an imaging optical system and collected by a microlens into an electric charge, wherein the plurality of pixels include a first pixel that shares the microlens with an adjacent pixel, and a second pixel that is different from the first pixel, and the imaging control program causes a processor to execute a step of controlling the ratio between a first exposure time that is the exposure time of the first pixel and a second exposure time that is the exposure time of the second pixel, based on conditions related to the angle of incidence of light on the pixels.

Images