Information processing device, imaging device, information processing method, and information processing program

Abstract

The present invention provides an information processing device, an imaging device, an information processing method, and an information processing program which make it possible to determine the presence or absence of extraordinary light.　Provided is an information processing method for processing output of an imaging element (5) which has a plurality of pixels (61) each including a photoelectric conversion unit PD that converts, into a charge, light focused by a microlens ML. The plurality of pixels (61) include: first pixels (a first phase difference detection pixel (61FA) and a second phase difference detection pixel (61FB)) which each share a microlens ML with an adjacent pixel; and second pixels (a pixel (61R), a pixel (61G), and a pixel (61B)) which differ from the first pixels. The information processing method includes a step for determining, on the basis of first pixel values of the first pixels and second pixel values of the second pixels, the presence or absence of extraordinary light which enters the first pixels.

Description

Information processing device, imaging device, information processing method, and information processing program

[0001] The technology disclosed herein relates to an information processing device, an imaging device, an information processing method, and an information processing program.

[0002] Patent Document 1 describes an imaging element that includes a pixel having one photoelectric conversion unit and one microlens, and a pixel having two photoelectric conversion units and one microlens.

[0003] Patent document 2 describes an image processing device that includes a light receiving unit in which a plurality of unit pixels, each of which is a set of a plurality of photoelectric conversion elements, are arranged in a two-dimensional matrix, and a microlens provided for each unit pixel and stacked on the light receiving surface of the unit pixel, and that compares the output values ​​of each of the plurality of photoelectric conversion elements for each unit pixel based on image data generated by an imaging element that includes the light receiving unit, in which a plurality of unit pixels, each of which is a set of a plurality of photoelectric conversion elements, are arranged in a two-dimensional matrix, and a microlens provided for each unit pixel and stacked on the light receiving surface of the unit pixel, and that compares the output values ​​of each of the plurality of photoelectric conversion elements for each unit pixel to detect abnormal outputs, and an estimation unit that estimates an abnormality using the output values ​​of each of the plurality of photoelectric conversion elements in the unit pixel in which the comparison unit has detected an abnormal output.

[0004] Patent Document 3 describes an imaging device that includes an image sensor configured by a two-dimensional array of pixels, each pixel having an on-chip microlens that is divided into a plurality of regions and that focuses a pupil image of an imaging optical system onto the plurality of regions, and a readout unit that reads out a photoelectrically converted signal for each of the plurality of divided regions; a signal intensity detection unit that detects the signal intensity of at least the peripheral signal out of the central signal and the peripheral signal in the plurality of divided regions for each pixel that are read out by the readout unit; and a ghost detection unit that detects ghosts based on the signal intensity detected by the signal intensity detection unit.

[0005] Japanese Patent Publication No. 2014-165787 Japanese Patent Publication No. 2018-152829 International Publication No. 2015 / 045829

[0006] An information processing device, an imaging device, an information processing method, and an information processing program according to one embodiment of the technology of the present disclosure are shown below. Note that, while the parentheses indicate corresponding components in the following embodiments, the present disclosure is not limited to these.

[0007] (1) An information processing device that processes output from an image sensor (image sensor 5) having a plurality of pixels (pixels 61) including a photoelectric conversion unit (photoelectric conversion unit PD) that converts light collected by a microlens (microlens ML) into an electric charge, wherein the plurality of pixels include a first pixel (first phase difference detection pixel 61FA, second phase difference detection pixel 61FB) that shares the microlens with an adjacent pixel, and a second pixel (pixel 61R, pixel 61G, pixel 61B) that is different from the first pixel, and the information processing device includes a processor (system control unit 11) that determines whether or not abnormal light is incident on the first pixel based on a first pixel value of the first pixel and a second pixel value of the second pixel.

[0008] (2) The information processing device according to (1), wherein the second pixel is a pixel having the microlens alone.

[0009] (3) The information processing device according to (1) or (2), wherein the processor determines the presence or absence of abnormal light based on a comparison result between the first pixel value and the second pixel value.

[0010] (4) The information processing device according to (3), wherein the processor determines the presence or absence of abnormal light based on the comparison result and thresholds (determination thresholds THC, THR, and THL).

[0011] (5) The information processing device according to (4), wherein the comparison result is a ratio between the first pixel value and the second pixel value, or a difference between the first pixel value and the second pixel value.

[0012] (6) The information processing device according to (5), wherein the processor determines that the abnormal light is present when the comparison result is equal to or greater than the threshold value.

[0013] (7) The information processing device according to any one of (4) to (6), wherein the processor controls the threshold value.

[0014] (8) The information processing device according to (7), wherein the processor changes the threshold value based on the position of the first pixel that is the target of the determination of the presence or absence of abnormal light.

[0015] (9) The information processing device according to (7) or (8), in which the processor changes the threshold value based on a condition of an imaging optical system disposed between the imaging element and the subject.

[0016] (10) The information processing device according to (9), wherein the condition includes at least one of an aperture value of a diaphragm included in the imaging optical system and a focal length of the imaging optical system.

[0017] (11) An information processing device according to any one of (1) to (10), wherein the first pixel includes a luminance filter (luminance filter LF), the second pixel includes a color filter (color filter CF), the plurality of pixels include the second pixel of a plurality of types having different types of color filters, and the processor determines whether or not abnormal light is incident on the first pixel based on a first pixel value of the first pixel and the second pixel value of any of the plurality of types of second pixels.

[0018] (12) An information processing device according to any one of (1) to (10), wherein the first pixel and the second pixel include a color filter (color filter CF), the plurality of pixels include a plurality of types of the first pixel having different types of the color filter and a plurality of types of the second pixel having different types of the color filter, and the processor determines whether or not abnormal light is incident on the first pixel based on a first pixel value of the first pixel including the first type of color filter and a second pixel value of the second pixel including the first type of color filter.

[0019] (13) The information processing device according to any one of (1) to (12), wherein the processor performs focus detection processing based on the pixel value of the first pixel.

[0020] (14) The information processing device according to (13), wherein, when the processor determines that there is a first pixel into which the abnormal light is incident, the processor performs focus detection processing based on the pixel values ​​of the first pixels excluding at least the first pixel.

[0021] (15) An information processing device according to (14), wherein, when the processor determines that there is a first pixel into which the abnormal light is incident, the processor performs focus detection processing based on the pixel value of the first pixel excluding the first pixel and the first pixels surrounding the first pixel.

[0022] (16) An information processing device according to any one of (1) to (15), wherein the processor further uses a comparison result of pixel values ​​of a plurality of the first pixels including a plurality of the photoelectric conversion units that share the microlens to determine whether the abnormal light exists.

[0023] (17) The information processing device according to any one of (1) to (16), wherein the first pixel is one in which the microlens is shared by two adjacent first pixels.

[0024] (18) An imaging device including: the information processing device according to any one of (1) to (17); and the imaging element.

[0025] (19) An information processing method for processing the output of an imaging element having a plurality of pixels including a photoelectric conversion unit that converts light collected by a microlens into an electric charge, wherein the plurality of pixels include a first pixel that shares the microlens with an adjacent pixel, and a second pixel that is different from the first pixel, and the information processing method includes a step of determining whether or not abnormal light is incident on the first pixel based on a first pixel value of the first pixel and a second pixel value of the second pixel.

[0026] (20) An information processing program for processing the output of an imaging element having a plurality of pixels including a photoelectric conversion unit that converts light collected by a microlens into an electric charge, wherein the plurality of pixels include a first pixel that shares the microlens with an adjacent pixel, and a second pixel that is different from the first pixel, and the information processing program causes a processor to execute a step of determining whether or not abnormal light is incident on the first pixel based on a first pixel value of the first pixel and a second pixel value of the second pixel.

[0027] FIG. 1 is a diagram showing a schematic configuration of a digital camera 100 that is an embodiment of an imaging device relating to the technique of the present disclosure. FIG. 2 is a plan view schematic diagram showing a schematic configuration of the image sensor 5 shown in FIG. 1. FIG. 3 is a schematic diagram for explaining the angle of light incident on a pixel 61 when viewed in the column direction Y. FIG. 4 is a schematic diagram showing a partially enlarged image capturing surface 60 of the image sensor 5 shown in FIG. 2. FIG. 5 is a cross-sectional view schematic of a range A1 shown in FIG. 4. FIG. 6 is a cross-sectional view schematic of a range A2 shown in FIG. 4. FIG. 7 is a diagram showing output characteristics of a pixel 61G, a first phase difference detection pixel 61FA, and a second phase difference detection pixel 61FB. FIG. 8 is a cross-sectional view schematic of a range A2 shown in FIG. 4 for explaining a first modified example of the image sensor 5. FIG. 9 is a diagram showing output characteristics of a pixel 61G, a first phase difference detection pixel 61FA, and a second phase difference detection pixel 61FB in the image sensor 5 of the modified example shown in FIG. 8. FIG. 10 is a diagram for explaining a second modified example of the imaging element 5, and corresponds to FIG. 4 . FIG. 11 is a schematic cross-sectional view of range A3 in FIG. 10 . FIG. 12 is a schematic cross-sectional view of range A4 in FIG. 10 . FIG. 13 is a diagram showing a third modified example of the imaging element 5, and corresponds to FIG. 4 . FIG. 14 is a diagram showing a fourth modified example of the imaging element 5, and corresponds to FIG. 4 . FIG. 15 shows the external appearance of the smartphone 200. FIG. 16 is a block diagram showing the configuration of the smartphone 200 shown in FIG. 15 .

[0028] Fig. 1 is a diagram showing a schematic configuration of a digital camera 100, which is an embodiment of an imaging device relating to the technique of the present disclosure. The digital camera 100 shown in Fig. 1 includes a lens device 40 having an imaging lens 1, an aperture 2, a lens driver 8 that drives the imaging lens 1, an aperture driver 9 that drives the aperture 2, and a lens controller 4 that controls the lens driver 8 and the aperture driver 9, and a main body 100A.

[0029] The main body 100A includes an image sensor 5, a system control unit 11 that controls the overall electrical control system of the digital camera 100, an operation unit 14, a display device 22, a memory 16 that includes RAM (Random Access Memory) and ROM (Read Only Memory), a memory control unit 15 that controls the storage of data in the memory 16 and the reading of data from the memory 16, a digital signal processing unit 17, and an external memory control unit 20 that controls the storage of data in a storage medium 21 and the reading of data from the storage medium 21.

[0030] The lens device 40 may be detachable from the main body 100 A, or may be integrated with the main body 100 A. The imaging lens 1 includes a focus lens.

[0031] The focus lens is a lens for adjusting the focus of an imaging optical system including the imaging lens 1 and the aperture 2, and is composed of a single lens or multiple lenses. When the focus lens moves in the optical axis direction, the position of the principal point of the focus lens (hereinafter also referred to as the focus lens position) changes along the optical axis, thereby changing the focal position on the subject side. Note that a liquid lens whose principal point position in the optical axis direction can be changed by electrical control may also be used as the focus lens.

[0032] The lens control unit 4 of the lens device 40 controls the lens drive unit 8 based on the lens drive signal transmitted from the system control unit 11 to change the focus lens position. The lens control unit 4 of the lens device 40 controls the diaphragm drive unit 9 based on the drive control signal transmitted from the system control unit 11 to change the opening amount (aperture value (F-number)) of the diaphragm 2.

[0033] The image sensor 5 captures an image of the subject through the imaging optical system provided between the image sensor 5 and the subject. The image sensor 5 has an imaging surface 60 (see FIG. 2 ) on which a plurality of pixels are arranged two-dimensionally, and converts an image of the subject formed on the imaging surface 60 by the imaging optical system into an image signal using the plurality of pixels and outputs the image signal. The output of the pixels included in the image sensor 5 is referred to as a pixel value or pixel signal, and a collection of pixel values ​​or pixel signals is referred to as an image signal.

[0034] For example, a complementary metal-oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor is used as the imaging element 5. In the following, an example in which the imaging element 5 is a CMOS image sensor will be described.

[0035] The system control unit 11 controls the entire digital camera 100, and its hardware structure consists of various processors that execute programs and perform processing. The programs (including information processing programs) executed by the system control unit 11 are stored in ROM (non-transitory storage medium) of the memory 16. The memory 16 and the system control unit 11 constitute an information processing device that processes the output of the image sensor 5.

[0036] The various types of processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs to perform various processes, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), or a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically to perform specific processes. More specifically, the structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0037] The system control unit 11 may be configured with one of various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0038] The system control unit 11 drives the image sensor 5 and the lens device 40 to output, as an image signal, an image of a subject captured through the imaging optical system of the lens device 40. The image signal output from the image sensor 5 is processed by the digital signal processing unit 17 to generate captured image data that is data suitable for display on the display device 22 or data suitable for storage in the storage medium 21.

[0039] An instruction signal from the user is input to the system control unit 11 through the operation unit 14. The operation unit 14 includes a touch panel integrated with the display surface 22b, various buttons, and the like.

[0040] The display device 22 includes a display surface 22b configured by an organic EL (electroluminescence) panel, a liquid crystal panel, or the like, and a display controller 22a that controls the display on the display surface 22b.

[0041] The memory control unit 15 , digital signal processing unit 17 , external memory control unit 20 , and display controller 22 a are interconnected by a control bus 24 and a data bus 25 , and are controlled by commands from the system control unit 11 .

[0042] 2 is a plan view schematically illustrating the configuration of the image sensor 5 shown in FIG. 1 . The image sensor 5 includes an imaging surface 60 on which pixel rows 62, each consisting of a plurality of pixels 61 arranged in a row direction X, are arranged in a column direction Y intersecting the row direction X, a drive circuit 63 that drives the pixels 61 arranged on the imaging surface 60, and a signal processing circuit 64 that processes pixel signals read out to signal lines from each pixel 61 in the pixel rows 62 arranged on the imaging surface 60. In the example of FIG. 2 , the row direction X and the column direction Y are orthogonal to each other. One side of the row direction X is referred to as the right direction XR, and the other side of the row direction X is referred to as the left direction XL. One side of the column direction Y is referred to as the upward direction YU, and the other side of the column direction Y is referred to as the downward direction YD.

[0043] FIG. 3 is a schematic diagram illustrating the angle of light incident on a pixel 61 when viewed in the column direction Y. A line L in the figure represents a line parallel to the optical axis of the imaging optical system or a line perpendicular to the imaging surface 60. Hereinafter, the angle between a light ray incident on a pixel 61 (indicated by a dashed arrow in FIG. 3 ) and the line L is defined as the incident angle θ of the light on that pixel 61. The angle between the line L and light incident obliquely from the right direction XR with respect to the line L is defined as a positive value of the incident angle θ, and the angle between the line L and light incident obliquely from the left direction XL with respect to the line L is defined as a negative value of the incident angle θ. The absolute value of the incident angle θ is larger at the ends of the imaging surface 60 in the right direction XR and the ends of the imaging surface 60 in the left direction XL than at the center of the imaging surface 60 (near the point where the light intersects with the optical axis of the imaging optical system). In other words, if the position of pixel 61 at the intersection with the optical axis on imaging surface 60 is taken as the reference position, the absolute value of the incident angle θ of pixel 61 increases as the position of pixel 61 moves away from the reference position in the row direction X.

[0044] 4 is a schematic diagram showing a partially enlarged image pickup surface 60 of the image pickup element 5 shown in FIG. 2 . The plurality of pixels 61 arranged on the image pickup surface 60 includes pixels corresponding to each of a plurality of wavelength bands (three in this embodiment). Specifically, the image pickup surface 60 includes a pixel 61R (a block marked with "R" in the figure) corresponding to the wavelength band of red light, a pixel 61G (a block marked with "G" in the figure) corresponding to the wavelength band of green light, a pixel 61B (a block marked with "B" in the figure) corresponding to the wavelength band of blue light, a first phase difference detection pixel 61FA (a block marked with "FA" in the figure) corresponding to the wavelength band of green light, and a second phase difference detection pixel 61FB (a block marked with "FB" in the figure) corresponding to the wavelength band of green light. Each pixel 61 arranged on the image pickup surface 60 receives light of the corresponding wavelength band and outputs a pixel signal corresponding to the amount of the received light.

[0045] Pixels 61R, 61G, and 61B are arranged based on a Bayer pattern on the imaging surface 60. That is, on the imaging surface 60, RG pixel rows in which pixels 61R and pixels 61G are alternately arranged in the row direction X, and GB pixel rows in which pixels 61G and pixels 61B are alternately arranged in the row direction X are alternately arranged in the column direction Y. The arrangement pattern of the pixels 61 arranged on the imaging surface 60 is not limited to the Bayer pattern, and various patterns can be adopted.

[0046] In some GB pixel rows among these multiple pixel rows, some pixels 61B are replaced with first phase difference detection pixels 61FA, and some pixels 61G are replaced with second phase difference detection pixels 61FB, with the first phase difference detection pixels 61FA and the second phase difference detection pixels 61FB adjacent to each other. In these GB pixel rows, a plurality of phase difference detection pairs, each consisting of a first phase difference detection pixel 61FA and its adjacent second phase difference detection pixel 61FB, are arranged side by side at intervals in the row direction X. These GB pixel rows may be composed solely of phase difference detection pairs. The phase difference detection pairs may also be provided in the RG pixel rows. In this case, the adjacent pixels 61G and 61R may be replaced with the phase difference detection pair.

[0047] The system control unit 11 performs focus detection processing based on the pixel values ​​of the first phase difference detection pixels 61FA and the second phase difference detection pixels 61FB. Specifically, the system control unit 11 detects a phase difference by performing a correlation calculation between a group of pixel signals output from the first phase difference detection pixels 61FA and a group of pixel signals output from the second phase difference detection pixels 61FB included in the same pixel row, and performs focus detection processing to derive a focus lens position required to focus on a target subject based on the phase difference.

[0048] Fig. 5 is a schematic cross-sectional view of range A1 shown in Fig. 4. Fig. 6 is a schematic cross-sectional view of range A2 shown in Fig. 4. As shown in Fig. 5 and Fig. 6, each pixel 61 provided on the imaging surface 60 includes a microlens ML that collects light from a subject, a photoelectric conversion unit PD that converts the light collected by the microlens ML into an electric charge, and a color filter CF that is provided between the photoelectric conversion unit PD and the microlens ML and that transmits light in a specific wavelength band.

[0049] Although not shown, each pixel 61 is provided with a readout circuit that converts the charge generated in the photoelectric conversion unit PD into a pixel value and reads it out. Furthermore, a light-shielding film that defines the light-receiving area of ​​the photoelectric conversion unit PD and a light-shielding film that shields the readout circuit disposed close to the photoelectric conversion unit PD from light are provided between the photoelectric conversion unit PD and the color filter CF. The photoelectric conversion unit PD is a photodiode formed in a semiconductor substrate such as silicon, but may also be composed of an organic material film disposed above the semiconductor substrate.

[0050] The color filter CF (referred to as an R filter in FIG. 5) included in the pixel 61R transmits red light, the color filter CF (referred to as a G filter in FIGS. 5 and 6) included in the pixel 61G transmits green light, and the color filter CF (referred to as a B filter in FIG. 6) included in the pixel 61B transmits blue light. The color filters CF included in each of the first phase difference detection pixel 61FA and the second phase difference detection pixel 61FB are G filters that transmit green light.

[0051] As described above, the imaging surface 60 is provided with a plurality of types of pixels 61 (pixels 61R, 61G, and 61B) having different types of color filters CF (three types in this embodiment), and a phase difference detection pair of a first phase difference detection pixel 61FA and a second phase difference detection pixel 61FB including the same type of color filter CF as the pixel 61G. Note that the color filters CF can be omitted when the structure of the photoelectric conversion unit PD itself is capable of separating red, green, and blue light.

[0052] The first phase difference detection pixel 61FA shares a microlens ML with its adjacent second phase difference detection pixel 61FB. That is, the microlens ML included in the first phase difference detection pixel 61FA and the microlens ML included in the adjacent second phase difference detection pixel 61FB are the same. The planar shape of the microlens ML provided in the phase difference detection pair is an ellipse with its major axis extending in the row direction X, as partially shown in FIG. 4 . This ellipse-shaped microlens ML is provided above two photoelectric conversion units PD arranged in the row direction X in the phase difference detection pair, straddling the two photoelectric conversion units PD. On the other hand, the pixel 61R, the pixel 61G, and the pixel 61B each have their own microlens ML, and the microlens ML is not shared with other pixels 61. It can also be said that the phase difference detection pair shares a microlens ML. The first phase difference detection pixel 61FA and the second phase difference detection pixel 61FB each constitute a first pixel. The pixel 61R, the pixel 61G, and the pixel 61B each constitute a second pixel different from the first pixel. It can also be said that the second pixel has optical characteristics different from those of the first pixel.

[0053] 7 is a diagram showing output characteristics of the pixel 61G, the first phase difference detection pixel 61FA, and the second phase difference detection pixel 61FB. The horizontal axis in the diagram represents the incident angle θ of light to the pixel 61, and the vertical axis represents the pixel value output from the pixel 61. The pixel values ​​are normalized using the pixel value of the pixel 61G when the incident angle θ is 0 degrees as the reference (=1). A characteristic Cfa in the diagram represents the output characteristic of the first phase difference detection pixel 61FA. A characteristic Cfb in the diagram represents the output characteristic of the second phase difference detection pixel 61FB. A characteristic Cg in the diagram represents the output characteristic of the pixel 61G.

[0054] 7 , the output characteristics of the pixel 61G are such that the pixel value is maximum when the incident angle θ is 0 degrees, and the pixel value gradually decreases as the absolute value of the incident angle θ increases. The output characteristics of the first phase difference detection pixel 61FA are such that the pixel value is greater in the positive range of the incident angle θ than in the negative range of the incident angle θ, and the pixel value is maximum when the incident angle θ is around 23 degrees. Furthermore, the output characteristics of the first phase difference detection pixel 61FA are greater than the maximum pixel value of the pixel 61G in the positive range of the incident angle θ.

[0055] The characteristic Cfb is obtained by inverting the characteristic Cfa across the vertical axis. The output characteristic of the second phase difference detection pixel 61FB is such that the pixel value is larger in the negative range of the incident angle θ than in the positive range of the incident angle θ, and the pixel value is maximum when the incident angle θ is around −23 degrees. Furthermore, the output characteristic of the second phase difference detection pixel 61FB is larger than the maximum pixel value of the pixel 61G in the negative range of the incident angle θ.

[0056] Although the output characteristics of pixel 61R and pixel 61B are not shown in the figure, similar to characteristic Cg, they are symmetrical with respect to the vertical axis, with the pixel value being maximum when the incident angle θ is 0 degrees, and the pixel value gradually decreasing as the absolute value of the incident angle θ increases.

[0057] A range HC shown in Fig. 7 indicates a range of possible incident angles θ (hereinafter also referred to as the incident angle range) of light incident on a pixel 61 located at the center in the row direction X of the imaging surface 60. A range HR shown in Fig. 7 indicates a range of possible incident angles of light incident on a pixel 61 located at an end in the right direction XR of the row direction X of the imaging surface 60. A range HL shown in Fig. 7 indicates a range of possible incident angles of light incident on a pixel 61 located at an end in the left direction XL of the row direction X of the imaging surface 60.

[0058] The widths of the ranges HC, HR, and HL vary depending on the aperture size (f-stop value) of the aperture. For example, as the aperture size increases, the widths of the ranges HC, HR, and HL increase, and as the aperture size decreases, the widths of the ranges HC, HR, and HL decrease. The widths of the ranges HC, HR, and HL also vary depending on the type of lens device 40 and the focal length. The center positions of the widths of the ranges HR and HL can also change depending on the combination of the aperture value and the focal length. The aperture value and focal length of the lens device 40 are each one of the imaging conditions of the imaging optical system.

[0059] As shown in ranges HC, HR, and HL in FIG. 7 , the range of the angle of incidence of light incident on a pixel 61 varies depending on the imaging conditions, but is determined by the position of that pixel 61 on the imaging surface 60. In the digital camera 100, a phenomenon known as ghosting or flare may occur. This phenomenon may occur when light with an incident angle θ that is outside the range of incident angles θ of light expected to be incident on the imaging surface 60 (the range from the left end of range HR to the right end of range HL in FIG. 7 ) and has a large absolute value (for example, light with an incident angle θ of ±25 degrees or ±30 degrees) enters the pixel 61. Light that causes such ghosting or flare is referred to as abnormal light.

[0060] In the pixel 61 disposed at a position where light in the range HC is incident, the ratio of the pixel value of the first phase difference detection pixel 61FA to the pixel value of the pixel 61G and the ratio of the pixel value of the second phase difference detection pixel 61FB to the pixel value of the pixel 61G are each at most approximately 1.3 times (referred to as the maximum value MC). When extraordinary light having an incident angle θ of 30 degrees is incident on the first phase difference detection pixel 61FA disposed at a position where light in the range HC is incident, the ratio of the pixel value of the first phase difference detection pixel 61FA to the pixel value of the pixel 61G becomes sufficiently larger than the maximum value MC. Furthermore, when extraordinary light having an incident angle θ of −30 degrees is incident on the second phase difference detection pixel 61FB disposed at a position where light in the range HC is incident, the ratio of the pixel value of the second phase difference detection pixel 61FB to the pixel value of the pixel 61G becomes sufficiently larger than the maximum value MC. Therefore, by monitoring this ratio, it is possible to determine whether or not abnormal light is incident on the first phase difference detection pixel 61FA and the second phase difference detection pixel 61FB, which are arranged at positions where light in the range HC is incident.

[0061] For example, for the pixel 61 located at a position where light in the range HC is incident, the system control unit 11 sets a value greater than the maximum value MC (e.g., 1.5 times) as the abnormal light determination threshold THC. Then, for the first phase difference detection pixel 61FA and the pixel 61G located at a position where light in the range HC is incident and nearby, the system control unit 11 determines whether the ratio of the pixel value of the first phase difference detection pixel 61FA to the pixel value of the pixel 61G is greater than or equal to the determination threshold THC, and if the determination result is YES, determines that abnormal light is incident on the first phase difference detection pixel 61FA. Furthermore, for the second phase difference detection pixel 61FB and the pixel 61G located in the vicinity of and at the position where the light in the range HC is incident, the system control unit 11 determines whether the ratio of the pixel value of the second phase difference detection pixel 61FB to the pixel value of the pixel 61G is equal to or greater than the determination threshold value THC, and if the determination result is YES, determines that abnormal light is incident on the second phase difference detection pixel 61FB.

[0062] In the range HR where the angle of incidence of light is steeper, the ratio of the pixel value of the pixel 61G to the pixel value of the second phase difference detection pixel 61FB does not change significantly compared to the area outside the range HR on the negative side, but the ratio of the pixel value of the pixel 61G to the pixel value of the first phase difference detection pixel 61FA changes significantly. For example, when abnormal light with an incident angle θ of −30 degrees is incident on the first phase difference detection pixel 61FA arranged at a position where light in the range HR is incident, the ratio of the pixel value of the first phase difference detection pixel 61FA to the pixel value of the pixel 61G becomes sufficiently larger than this ratio that can be assumed in the range HR.

[0063] For example, when light in the range HR is incident, the maximum value MR of the ratio of the pixel value of the first phase difference detection pixel 61FA to the pixel value of the pixel 61G is set to 0.4. For the pixel 61 located at a position where light in the range HR is incident, the system control unit 11 sets a value (e.g., 0.6) greater than this maximum value MR as the abnormal light determination threshold THR. Then, for the first phase difference detection pixel 61FA and the pixel 61G located near the position where light in the range HR is incident, the system control unit 11 determines whether the ratio of the pixel value of the first phase difference detection pixel 61FA to the pixel value of the pixel 61G is equal to or greater than the determination threshold THR. If the determination result is YES, the system control unit 11 determines that abnormal light is incident on the first phase difference detection pixel 61FA.

[0064] Similarly, in the range HL, compared to the area outside the range HL on the positive side, the ratio of the pixel value of the pixel 61G to the pixel value of the first phase difference detection pixel 61FA does not change significantly, but the ratio of the pixel value of the pixel 61G to the pixel value of the second phase difference detection pixel 61FB changes significantly. For example, when abnormal light with an incident angle θ of 30 degrees is incident on the second phase difference detection pixel 61FB arranged at a position where light in the range HL is incident, the ratio of the pixel value of the second phase difference detection pixel 61FB to the pixel value of the pixel 61G becomes sufficiently larger than this ratio that can be assumed in the range HL.

[0065] For example, when light in the range HL is incident, the maximum value ML of the ratio of the pixel value of the second phase difference detection pixel 61FB to the pixel value of the pixel 61G is set to 0.4. For the pixel 61 located at a position where light in the range HL is incident, the system control unit 11 sets a value greater than this maximum value ML (e.g., 0.6) as the abnormal light determination threshold THL. Then, for the second phase difference detection pixel 61FB and the pixel 61G located near the position where light in the range HL is incident, the system control unit 11 determines whether the ratio of the pixel value of the second phase difference detection pixel 61FB to the pixel value of the pixel 61G is equal to or greater than the determination threshold THL. If the determination result is YES, the system control unit 11 determines that abnormal light is incident on the second phase difference detection pixel 61FB.

[0066] Note that for the pixel 61 located at a position where light is incident within the incident angle range between the range HC and the range HR and the pixel 61 located at a position where light is incident within the incident angle range between the range HC and the range HL, the ratio of the pixel values ​​of the first phase difference detection pixel 61FA, the second phase difference detection pixel 61FB, and the pixel 61G maintains substantially the same relationship as in the range HC. Therefore, the presence or absence of abnormal light can be determined using the same method as in the range HC. However, the determination threshold for the presence or absence of abnormal light needs to be greater than the determination threshold THC.

[0067] The maximum values ​​MC, MR, and ML each change depending on the combination of aperture value and focal length. Therefore, it is preferable that the determination thresholds THC, THR, and THL be changed depending on the combination of aperture value and focal length. That is, it is preferable that each time the imaging conditions change, the system control unit 11 sets the determination thresholds THC, THR, and THL based on the imaging conditions, and determines the presence or absence of abnormal light using the set determination thresholds.

[0068] Although the above has described a method for determining the presence or absence of anomalous light based on the ratio of two pixel values, a similar determination can also be made using the difference between the two pixel values. For example, the system control unit 11 sets a determination threshold that is greater than the maximum possible difference (absolute value) between the pixel values ​​of the first phase difference detection pixel 61FA and the pixel 61G when light within the range HC is incident. For a pixel 61 at a position where light within the range HC is incident, the system control unit 11 determines that anomalous light is incident on the first phase difference detection pixel 61FA when the value obtained by subtracting the pixel value of the pixel 61G from the pixel value of the first phase difference detection pixel 61FA is equal to or greater than the determination threshold. Furthermore, the system control unit 11 determines that anomalous light is incident on the second phase difference detection pixel 61FB when the value obtained by subtracting the pixel value of the pixel 61G from the pixel value of the second phase difference detection pixel 61FB is equal to or greater than the determination threshold.

[0069] Furthermore, the system control unit 11 sets a determination threshold that is smaller than the smallest possible value of the difference (absolute value) between the pixel values ​​of the first phase difference detection pixel 61FA and the pixel 61G when light in the range HR is incident. For the pixel 61 at the position where light in the range HR is incident, if the value obtained by subtracting the pixel value of the first phase difference detection pixel 61FA from the pixel value of the pixel 61G is equal to or smaller than the determination threshold, the system control unit 11 determines that abnormal light is incident on the first phase difference detection pixel 61FA.

[0070] Furthermore, the system control unit 11 sets a determination threshold that is smaller than the smallest possible value of the difference (absolute value) between the pixel values ​​of the second phase difference detection pixel 61FB and the pixel 61G when light in the range HL is incident. For a pixel 61 at a position where light in the range HL is incident, the system control unit 11 determines that abnormal light is incident on the second phase difference detection pixel 61FB when the value obtained by subtracting the pixel value of the second phase difference detection pixel 61FB from the pixel value of the pixel 61G is equal to or smaller than the determination threshold.

[0071] In this way, for a first phase difference detection pixel 61FA located in a region where the incident angle θ of light is small (a pixel position where the incident angle θ is between the range HR and the range HL), the system control unit 11 determines whether or not abnormal light is incident on the first phase difference detection pixel 61FA based on the pixel value of the first phase difference detection pixel 61FA and the pixel value of a pixel 61G near the first phase difference detection pixel 61FA (for example, any of the pixels 61G located closest to the first phase difference detection pixel 61FA). More specifically, the system control unit 11 determines whether or not abnormal light is incident on the first phase difference detection pixel 61FA based on the comparison result (the ratio or difference described above) between these two pixel values ​​and a determination threshold.

[0072] Furthermore, for a second phase difference detection pixel 61FB located in the region where the light incident angle θ is small, the system control unit 11 determines whether or not abnormal light is incident on the second phase difference detection pixel 61FB based on the pixel value of the second phase difference detection pixel 61FB and the pixel value of a pixel 61G near the second phase difference detection pixel 61FB (for example, any of the pixels 61G located closest to the second phase difference detection pixel 61FB). More specifically, the system control unit 11 determines whether or not abnormal light is incident on the second phase difference detection pixel 61FB based on the comparison result (the ratio or difference described above) between these two pixel values ​​and a determination threshold.

[0073] Furthermore, for a first phase difference detection pixel 61FA located in a region where the light incident angle θ is large (a pixel position where light in the range HR is incident), the system control unit 11 determines whether or not abnormal light is incident on the first phase difference detection pixel 61FA based on the pixel value of the first phase difference detection pixel 61FA and the pixel value of a pixel 61G near the first phase difference detection pixel 61FA (for example, one of the pixels 61G located closest to the first phase difference detection pixel 61FA). More specifically, the system control unit 11 determines whether or not abnormal light is incident on the first phase difference detection pixel 61FA based on the comparison result (the ratio or difference described above) between these two pixel values ​​and a determination threshold.

[0074] Furthermore, for a second phase difference detection pixel 61FB located in a region where the light incident angle θ is large (a pixel position where light in the range HL is incident), the system control unit 11 determines whether or not abnormal light is incident on the second phase difference detection pixel 61FB based on the pixel value of the second phase difference detection pixel 61FB and the pixel value of a pixel 61G near the second phase difference detection pixel 61FB (for example, any of the pixels 61G located closest to the second phase difference detection pixel 61FB). More specifically, the system control unit 11 determines whether or not abnormal light is incident on the second phase difference detection pixel 61FB based on the comparison result (the ratio or difference described above) between these two pixel values ​​and a determination threshold.

[0075] As described above, the digital camera 100 can determine whether or not abnormal light is incident on the first phase difference detection pixel 61FA and the second phase difference detection pixel 61FB, thereby improving the accuracy of phase difference detection and enabling highly accurate focus control.

[0076] For example, when the system control unit 11 determines that there is a first phase difference detection pixel 61FA or a second phase difference detection pixel 61FB into which abnormal light is incident, the system control unit 11 detects the phase difference based on the pixel values ​​of the other phase difference detection pairs excluding at least the phase difference detection pair including the first phase difference detection pixel 61FA or the second phase difference detection pixel 61FB. In this way, the phase difference can be detected with high accuracy.

[0077] Furthermore, according to the digital camera 100, the system control unit 11 controls the threshold value for determining whether or not anomalous light is present to one of a plurality of values. Specifically, the system control unit 11 changes the threshold value based on the position on the imaging surface 60 of the first phase difference detection pixel 61FA or the second phase difference detection pixel 61FB that is being used to determine whether or not anomalous light is present. This improves the accuracy of determining whether or not anomalous light is present, compared to a configuration in which the threshold value is fixed across the entire imaging surface 60. Furthermore, the system control unit 11 changes the threshold value for determining whether or not anomalous light is present based on the conditions of the imaging optical system. This improves the accuracy of determining whether or not anomalous light is present.

[0078] Fig. 8 is a diagram for explaining a first modified example of the image sensor 5, and is a schematic cross-sectional view showing a range A2 in Fig. 4. The modified image sensor 5 shown in Fig. 8 has the same configuration as the one described above, except that the color filters CF included in each of the first phase difference detection pixel 61FA and the second phase difference detection pixel 61FB are changed to luminance filters LF.

[0079] The luminance filter LF has spectral characteristics that correlate with the luminance component of light, and may be an ND (Neutral Density) filter, a transparent filter, a white filter, or a gray filter. A configuration in which no member that blocks light transmission is provided between the microlens ML and the photoelectric conversion unit PD and light is directly incident on the photoelectric conversion unit can also be said to have a luminance filter LF. The luminance filter LF can transmit light of more wavelength components than the color filter CF.

[0080] Fig. 9 is a diagram showing output characteristics of the pixel 61G, the first phase difference detection pixel 61FA, and the second phase difference detection pixel 61FB in the image sensor 5 of the modified example shown in Fig. 8. As shown in Fig. 9, the first phase difference detection pixel 61FA and the second phase difference detection pixel 61FB including the luminance filter LF have a larger difference in sensitivity from the pixel 61G compared to the case of Fig. 7. Therefore, when the image sensor 5 having the output characteristics shown in Fig. 9 is used, the above-described determination thresholds THC, THR, and THL are set to be larger than those in the case of the output characteristics shown in Fig. 7.

[0081] In the image sensor 5 having the configuration shown in FIGS. 4 to 6 , the first phase difference detection pixel 61FA and the second phase difference detection pixel 61FB each include a G filter. Therefore, the presence or absence of abnormal light incident on these pixels is determined using the pixel value of the pixel 61G that includes the same type of G filter, thereby improving the accuracy of the determination.

[0082] 8 , the first phase difference detection pixel 61FA and the second phase difference detection pixel 61FB each include a luminance filter LF. Therefore, the presence or absence of abnormal light incident on these pixels can be determined using any one of the pixel value of the pixel 61R including the R filter, the pixel value of the pixel 61G including the G filter, and the pixel value of the pixel 61B including the B filter, thereby improving the accuracy of the determination.

[0083] For example, when determining whether or not there is abnormal light that contains a large amount of red components, the accuracy of the determination can be improved by determining the presence or absence of abnormal light using the pixel value of the pixel 61R that includes an R filter. Furthermore, when determining whether or not there is abnormal light that contains a large amount of green components, the accuracy of the determination can be improved by determining the presence or absence of abnormal light using the pixel value of the pixel 61G that includes a G filter. When determining whether or not there is abnormal light that contains a large amount of blue components, the accuracy of the determination can be improved by determining the presence or absence of abnormal light using the pixel value of the pixel 61B that includes a B filter.

[0084] When the pixel value of the pixel 61R is the largest among the pixels 61R, 61G, and 61B near the first phase difference detection pixel 61FA that is the target of abnormal light detection, the system control unit 11 determines whether or not abnormal light is incident on the first phase difference detection pixel 61FA based on the pixel values ​​of the first phase difference detection pixel 61FA and the pixel 61R near it.

[0085] When the pixel value of the pixel 61G is the largest among the pixels 61R, 61G, and 61B near the first phase difference detection pixel 61FA that is the target of abnormal light detection, the system control unit 11 determines whether or not abnormal light is incident on the first phase difference detection pixel 61FA based on the pixel values ​​of the first phase difference detection pixel 61FA and the pixel 61G near it.

[0086] When the pixel value of the pixel 61B is the largest among the pixels 61R, 61G, and 61B near the first phase difference detection pixel 61FA that is the target of abnormal light detection, the system control unit 11 determines whether or not abnormal light is incident on the first phase difference detection pixel 61FA based on the pixel values ​​of the first phase difference detection pixel 61FA and the pixel 61B near the first phase difference detection pixel 61FA.

[0087] In this way, when the presence or absence of abnormal light is determined by comparing pixel values ​​of the first phase difference detection pixel 61FA (second phase difference detection pixel 61FB) with those of the neighboring pixel 61 that detects a specific color, it is preferable to determine the above-described determination thresholds THC, THR, and THL for each detection color of the pixel 61. This is because the relationships between the output characteristics of the pixel 61R, the output characteristics of the pixel 61G, and the output characteristics of the pixel 61B with respect to the characteristics Cfb and Cfa shown in FIG. 9 are not the same and may differ from one another.

[0088] FIG. 10 is a diagram illustrating a second modified example of the image sensor 5, and corresponds to FIG. 4 . FIG. 11 is a schematic cross-sectional view of a range A3 in FIG. 10 . FIG. 12 is a schematic cross-sectional view of a range A4 in FIG. 10 . The image sensor 5 of the modified example shown in FIG. 10 differs from FIG. 4 in that it includes not only a phase difference detection pair including a G filter (denoted as (G) in FIG. 4 ), but also a phase difference detection pair including an R filter (denoted as (R) in FIG. 4 ) and a phase difference detection pair including a B filter (denoted as (B) in FIG. 4 ). In other words, the image sensor 5 of the modified example shown in FIG. 10 is configured to include multiple types of phase difference detection pairs with different color filters CF (three types in the example of FIG. 10 ) and pixels 61 other than the phase difference detection pairs with different types of color filters CF (three types in the example of FIG. 10 ).

[0089] In the digital camera 100 equipped with the image sensor 5 shown in FIG. 10 , the system control unit 11 determines the presence or absence of abnormal light for a first phase difference detection pixel 61FA (second phase difference detection pixel 61FB) that includes an R filter using the pixel value of a pixel 61R located nearby. Furthermore, the system control unit 11 determines the presence or absence of abnormal light for a first phase difference detection pixel 61FA (second phase difference detection pixel 61FB) that includes a G filter using the pixel value of a pixel 61G located nearby. Furthermore, the system control unit 11 determines the presence or absence of abnormal light for a first phase difference detection pixel 61FA (second phase difference detection pixel 61FB) that includes a B filter using the pixel value of a pixel 61B located nearby. In this case as well, it is preferable to use the above-described determination thresholds THC, THR, and THL that correspond to the detection color of the pixel 61 used for the determination.

[0090] According to the modification shown in FIG. 10, it is possible to determine with high accuracy whether or not there is abnormal light containing a large amount of red components, abnormal light containing a large amount of green components, or abnormal light containing a large amount of blue components.

[0091] For example, when the system control unit 11 determines that abnormal light is incident on a first phase difference detection pair including a color filter CF of a specific color, it is preferable to detect the phase difference based on the pixel values ​​of the remaining phase difference detection pairs, excluding the first phase difference detection pair and a second phase difference detection pair surrounding the first phase difference detection pair, and perform focus detection processing based on the phase difference. Even when it is determined that no abnormal light is present in a second phase difference detection pair surrounding the first phase difference detection pair, there is a possibility that some of the abnormal light that entered the first phase difference detection pair may have been incident thereon. Therefore, as described above, by excluding the pixel values ​​of the second phase difference detection pair from the phase difference calculation, the phase difference can be detected with high accuracy.

[0092] In the above description, the system control unit 11 determines whether or not abnormal light is incident on the first phase difference detection pixel 61FA based on a comparison result between the pixel value of the first phase difference detection pixel 61FA and the pixel value of a pixel 61 nearby that is not part of the phase difference detection pair. Also, the system control unit 11 determines whether or not abnormal light is incident on the second phase difference detection pixel 61FB based on a comparison result between the pixel value of the second phase difference detection pixel 61FB and the pixel value of a pixel 61 nearby that is not part of the phase difference detection pair.

[0093] As a modified example of this, the system control unit 11 may determine the presence or absence of abnormal light incident on the first phase difference detection pixel 61FA by further using a comparison result between the pixel value of the first phase difference detection pixel 61FA and the pixel value of a nearby pixel 61 other than the phase difference detection pair, and a comparison result between the pixel value of the second phase difference detection pixel 61FB constituting the phase difference detection pair with the first phase difference detection pixel 61FA. Furthermore, the system control unit 11 may determine the presence or absence of abnormal light incident on the second phase difference detection pixel 61FB by further using a comparison result between the pixel value of the second phase difference detection pixel 61FB and the pixel value of a nearby pixel 61 other than the phase difference detection pair, and a comparison result between the pixel value of the first phase difference detection pixel 61FA constituting the phase difference detection pair with the second phase difference detection pixel 61FB.

[0094] For example, for extraordinary light with a large absolute value of the incident angle θ, a large difference occurs between the pixel value of the first phase difference detection pixel 61FA and the pixel value of the second phase difference detection pixel 61FB, as shown in Fig. 7. Therefore, in a phase difference detection pair located at a position where light is incident in the range HC in Fig. 7, it is determined that extraordinary light is incident on the first phase difference detection pixel 61FA when the ratio of the pixel value of the first phase difference detection pixel 61FA to the pixel value of the pixel 61G is equal to or greater than the determination threshold THC and when the absolute value of the difference between the pixel value of the first phase difference detection pixel 61FA and the pixel value of the second phase difference detection pixel 61FB is equal to or greater than the threshold. Similarly, when the ratio of the pixel value of the second phase difference detection pixel 61FB to the pixel value of the pixel 61G is equal to or greater than the determination threshold THC and the absolute value of the difference between the pixel value of the first phase difference detection pixel 61FA and the pixel value of the second phase difference detection pixel 61FB is equal to or greater than the threshold, it is determined that abnormal light is incident on the second phase difference detection pixel 61FB. In this way, the presence or absence of abnormal light can be determined with higher accuracy.

[0095] In the explanation so far, it has been assumed that a phase difference detection pair for detecting a phase difference in the row direction X is arranged on the imaging surface 60. The above-described technology for determining the presence or absence of abnormal light is similarly applicable to an imaging element 5 in which a phase difference detection pair for detecting a phase difference in the column direction Y is arranged on the imaging surface 60. It is also similarly applicable to an imaging element 5 in which a phase difference detection pair for detecting a phase difference in the row direction X and a phase difference detection pair for detecting a phase difference in the column direction Y are arranged on the imaging surface 60.

[0096] Fig. 13 is a diagram showing a third modified example of the image sensor 5 and corresponds to Fig. 4. The image sensor 5 shown in Fig. 13 has a configuration in which the position of the second phase difference detection pixel 61FB constituting the phase difference detection pair in the image sensor 5 shown in Fig. 4 is changed to be adjacent to the first phase difference detection pixel 61FA constituting the phase difference detection pair in the downward direction YD, and a pixel 61G is disposed in the position originally occupied by the second phase difference detection pixel 61FB. In the example shown in Fig. 13, the major axis direction of the microlens ML included in the phase difference detection pair coincides with the column direction Y.

[0097] 13 , if the row direction X in FIG. 3 is replaced with the column direction Y to define the incident angle θ, the relationship between the phase difference detection pair and the output characteristics of the pixel 61G will be the same as that shown in FIG. 7 . Therefore, based on the pixel value of the first phase difference detection pixel 61FA and the pixel values ​​of the neighboring pixels 61G, it is possible to determine whether or not abnormal light is incident on the first phase difference detection pixel 61FA. Also, based on the pixel value of the second phase difference detection pixel 61FB and the pixel values ​​of the neighboring pixels 61G, it is possible to determine whether or not abnormal light is incident on the second phase difference detection pixel 61FB.

[0098] Fig. 14 is a diagram showing a fourth modified example of the image sensor 5, and corresponds to Fig. 4. The image sensor 5 shown in Fig. 14 has a configuration in which the pixel 61G and the pixel 61R that are adjacent to the phase difference detection pair in the downward direction YD in the image sensor 5 shown in Fig. 4 are replaced with a third phase difference detection pixel 61FC and a fourth phase difference detection pixel 61FD.

[0099] 14 , a microlens ML is shared by four pixels 61, namely, a first phase difference detection pixel 61FA, a second phase difference detection pixel 61FB, a third phase difference detection pixel 61FC, and a fourth phase difference detection pixel 61FD, and a phase difference detection group is configured by these four pixels 61. On the imaging surface 60, a plurality of group rows, each configured with a plurality of phase difference detection groups lined up in the row direction X, are arranged in the column direction Y. It can also be said that on the imaging surface 60, a plurality of group columns, each configured with a plurality of phase difference detection groups lined up in the column direction Y, are arranged in the row direction X.

[0100] The system control unit 11 derives a first average value of the pixel values ​​of the first phase difference detection pixel 61FA and the third phase difference detection pixel 61FC in the phase difference detection group, and derives a second average value of the pixel values ​​of the second phase difference detection pixel 61FB and the fourth phase difference detection pixel 61FD in the phase difference detection group. The system control unit 11 detects the phase difference in the row direction X by performing a correlation calculation between the group of first average values ​​and the group of second average values ​​derived for each phase difference detection group in the same group row.

[0101] The system control unit 11 derives a third average value of the pixel values ​​of the first phase difference detection pixel 61FA and the second phase difference detection pixel 61FB in the phase difference detection group, and derives a fourth average value of the pixel values ​​of the third phase difference detection pixel 61FC and the fourth phase difference detection pixel 61FD in the phase difference detection group. The system control unit 11 detects the phase difference in the column direction Y by performing a correlation calculation between the group of third average values ​​and the group of fourth average values ​​derived for each phase difference detection group in the same group column.

[0102] The change in the first average value with respect to the incident angle θ is equivalent to the characteristic Cfa in FIG. 7 . The change in the second average value with respect to the incident angle θ is equivalent to the characteristic Cfb in FIG. 7 . Therefore, based on the first average value and the pixel value of the pixel 61G, it is possible to determine whether or not abnormal light is incident on the first phase difference detection pixel 61FA and the third phase difference detection pixel 61FC. Furthermore, based on the second average value and the pixel value of the pixel 61G, it is possible to determine whether or not abnormal light is incident on the second phase difference detection pixel 61FB and the fourth phase difference detection pixel 61FD.

[0103] Furthermore, based on the third average value and the pixel value of the pixel 61G, it is possible to determine whether or not abnormal light is incident on the first phase difference detection pixel 61FA and the second phase difference detection pixel 61FB. Furthermore, based on the fourth average value and the pixel value of the pixel 61G, it is possible to determine whether or not abnormal light is incident on the third phase difference detection pixel 61FC and the fourth phase difference detection pixel 61FD.

[0104] Next, the configuration of a smartphone, which is another embodiment of the imaging device of the present invention, will be described.

[0105] Fig. 15 shows the external appearance of a smartphone 200. The smartphone 200 shown in Fig. 15 has a flat housing 201, and is provided on one surface of the housing 201 with a display panel 202 as a display unit and a display input unit 204 that is an integrated unit of an operation panel 203 as an input unit.

[0106] Such a housing 201 also includes a speaker 205, a microphone 206, an operation unit 207, and a camera unit 208. Note that the configuration of the housing 201 is not limited to this, and for example, it is also possible to adopt a configuration in which the display unit and the input unit are independent, or a configuration having a folding structure or a sliding mechanism.

[0107] FIG. 16 is a block diagram showing the configuration of the smartphone 200 shown in FIG.

[0108] As shown in FIG. 16, the main components of the smartphone include a wireless communication unit 210, a display input unit 204, a call unit 211, an operation unit 207, a camera unit 208, a memory unit 212, an external input / output unit 213, a GNSS (Global Navigation Satellite System) receiving unit 214, a motion sensor unit 215, a power supply unit 216, and a main control unit 220.

[0109] The smartphone 200 also has, as its main function, a wireless communication function for performing mobile wireless communication via a base station device BS (not shown) and a mobile communication network NW (not shown).

[0110] The wireless communication unit 210 performs wireless communication with a base station device BS accommodated in the mobile communication network NW in accordance with instructions from the main control unit 220. Using this wireless communication, various file data such as audio data and image data, e-mail data, etc. are sent and received, and web data, streaming data, etc. are received.

[0111] The display input unit 204 is a so-called touch panel that, under the control of the main control unit 220, displays images (still images and moving images) or text information, etc. to visually convey information to the user and detects user operations on the displayed information, and is equipped with a display panel 202 and an operation panel 203.

[0112] The display panel 202 uses a liquid crystal display (LCD), an organic electroluminescence display (OELD), or the like as a display device.

[0113] The operation panel 203 is a device placed so that an image displayed on the display surface of the display panel 202 can be seen, and detects one or more coordinates operated by a user's finger or a stylus. When this device is operated by the user's finger or a stylus, a detection signal generated by the operation is output to the main control unit 220. Next, the main control unit 220 detects the operation position (coordinates) on the display panel 202 based on the received detection signal.

[0114] As shown in Figure 16, the display panel 202 and operation panel 203 of a smartphone 200, which is an example of one embodiment of the imaging device of the present invention, are integrated to form a display input unit 204, and the operation panel 203 is positioned so that it completely covers the display panel 202.

[0115] When such an arrangement is adopted, operation panel 203 may also have a function for detecting user operations in areas outside display panel 202. In other words, operation panel 203 may have a detection area for the overlapping portion that overlaps display panel 202 (hereinafter referred to as a display area), and a detection area for the remaining outer edge portion that does not overlap display panel 202 (hereinafter referred to as a non-display area).

[0116] The size of the display area and the size of the display panel 202 may be exactly the same, but they do not necessarily have to be the same. The operation panel 203 may also have two sensitive areas: an outer edge portion and an inner portion. The width of the outer edge portion is designed appropriately depending on the size of the housing 201, etc.

[0117] Furthermore, the position detection method used in the operation panel 203 may be a matrix switch method, a resistive film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, a capacitance method, or the like, and any method may be used.

[0118] The call unit 211 is equipped with a speaker 205 or a microphone 206, and converts the user's voice input through the microphone 206 into voice data that can be processed by the main control unit 220 and outputs it to the main control unit 220, or decodes voice data received by the wireless communication unit 210 or the external input / output unit 213 and outputs it from the speaker 205.

[0119] Also, as shown in FIG. 15, for example, the speaker 205 can be mounted on the same surface as the display input unit 204, and the microphone 206 can be mounted on the side of the housing 201.

[0120] The operation unit 207 is a hardware key using a key switch or the like, and receives instructions from the user. For example, as shown in Fig. 15 , the operation unit 207 is a push-button switch mounted on the side surface of the housing 201 of the smartphone 200, which turns on when pressed with a finger or the like, and turns off when the finger is released due to the restoring force of a spring or the like.

[0121] The storage unit 212 stores the control program and control data of the main control unit 220, application software, address data associated with names or telephone numbers of communication partners, data of sent and received e-mails, web data downloaded by web browsing, downloaded content data, and also temporarily stores streaming data, etc. The storage unit 212 is composed of an internal storage unit 217 built into the smartphone and an external storage unit 218 having a removable external memory slot.

[0122] The internal memory unit 217 and the external memory unit 218 constituting the memory unit 212 are realized using storage media such as a flash memory type, a hard disk type, a multimedia card micro type, a card-type memory (e.g., MicroSD (registered trademark) memory, etc.), a RAM (Random Access Memory), a ROM (Read Only Memory), etc.

[0123] The external input / output unit 213 serves as an interface with all external devices connected to the smartphone 200, and is used to directly or indirectly connect to other external devices via communication (e.g., Universal Serial Bus (USB), IEEE 1394, Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) (registered trademark), UWB (Ultra Wideband) (registered trademark), ZigBee (registered trademark), etc.) or a network (e.g., Ethernet (registered trademark), wireless LAN (Local Area Network), etc.).

[0124] Examples of external devices that can be connected to the smartphone 200 include wired / wireless headsets, wired / wireless external chargers, wired / wireless data ports, memory cards connected via card sockets, SIM (Subscriber Identity Module) / UIM (User Identity Module) cards, external audio / video devices connected via audio / video I / O (Input / Output) terminals, wirelessly connected external audio / video devices, wired / wirelessly connected smartphones, wired / wirelessly connected personal computers, wired / wirelessly connected personal computers, earphones, etc.

[0125] The external input / output unit 213 can transmit data received from such external devices to each component inside the smartphone 200, or transmit data inside the smartphone 200 to external devices.

[0126] The GNSS receiving unit 214 receives GNSS signals transmitted from the GNSS satellites ST1 to STn in accordance with instructions from the main control unit 220, executes positioning calculation processing based on the received multiple GNSS signals, and detects a position consisting of the latitude, longitude, and altitude of the smartphone 200. When the GNSS receiving unit 214 can acquire position information from the wireless communication unit 210 or the external input / output unit 213 (for example, a wireless LAN), it can also detect the position using the position information.

[0127] The motion sensor unit 215 includes, for example, a three-axis acceleration sensor, and detects the physical movement of the smartphone 200 in accordance with instructions from the main control unit 220. By detecting the physical movement of the smartphone 200, the direction of movement or acceleration of the smartphone 200 is detected. The detection result is output to the main control unit 220.

[0128] The power supply unit 216 supplies power stored in a battery (not shown) to each unit of the smartphone 200 in accordance with instructions from the main control unit 220 .

[0129] The main control unit 220 includes a microprocessor, operates according to the control program and control data stored in the storage unit 212, and controls all the units of the smartphone 200. The microprocessor of the main control unit 220 has the same functions as the system control unit 11. The main control unit 220 also includes a mobile communication control function that controls all the units of the communication system to perform voice communication or data communication via the wireless communication unit 210, and an application processing function.

[0130] The application processing function is realized by the main control unit 220 operating in accordance with the application software stored in the storage unit 212. Examples of the application processing function include an infrared communication function that controls the external input / output unit 213 to perform data communication with a connected device, an email function that sends and receives emails, and a web browsing function that views web pages.

[0131] The main control unit 220 also has an image processing function for displaying video on the display input unit 204 based on image data (still image or moving image data) such as received data or downloaded streaming data.

[0132] The image processing function refers to a function in which the main control unit 220 decodes the image data, performs image processing on the decoded result, and displays the image on the display input unit 204 .

[0133] Furthermore, the main control unit 220 executes display control for the display panel 202 and operation detection control for detecting user operations via the operation unit 207 and the operation panel 203 .

[0134] By executing display control, the main control unit 220 displays software keys such as icons or scroll bars for starting application software, or displays a window for creating an e-mail.

[0135] The scroll bar refers to a software key for receiving an instruction to move the displayed portion of an image, such as a large image that cannot fit in the display area of ​​the display panel 202 .

[0136] In addition, by executing operation detection control, the main control unit 220 detects user operations through the operation unit 207, accepts operations on the above icons and input of character strings into the input field of the above window through the operation panel 203, or accepts requests to scroll the displayed image through the scroll bar.

[0137] Furthermore, by executing operation detection control, the main control unit 220 determines whether the operation position on the operation panel 203 is an overlapping portion (display area) that overlaps the display panel 202 or an outer edge portion (non-display area) that does not overlap the display panel 202, and is equipped with a touch panel control function that controls the sensitive area of ​​the operation panel 203 or the display position of the software key.

[0138] The main control unit 220 can also detect a gesture operation on the operation panel 203 and execute a preset function in response to the detected gesture operation.

[0139] Gesture operation is not a simple touch operation as in the past, but rather refers to an operation of drawing a path with a finger or the like, specifying multiple positions simultaneously, or combining these to draw a path for at least one of multiple positions.

[0140] The camera unit 208 includes the lens device 40, the image sensor 5, and the digital signal processing unit 17 shown in FIG.

[0141] The captured image data generated by the camera unit 208 can be stored in the storage unit 212 or output via the external input / output unit 213 or the wireless communication unit 210 .

[0142] In the smartphone 200 shown in FIG. 16 , the camera unit 208 is mounted on the same surface as the display input unit 204, but the mounting position of the camera unit 208 is not limited to this, and it may be mounted on the back surface of the display input unit 204.

[0143] The camera unit 208 can also be used for various functions of the smartphone 200. For example, an image acquired by the camera unit 208 can be displayed on the display panel 202, or an image from the camera unit 208 can be used as one of the operation inputs for the operation panel 203.

[0144] Furthermore, when the GNSS receiving unit 214 detects a position, it can also detect the position by referring to an image from the camera unit 208. Furthermore, it can also refer to an image from the camera unit 208 to determine the optical axis direction of the camera unit 208 of the smartphone 200 or determine the current usage environment without using a triaxial acceleration sensor or by using the image in combination with a triaxial acceleration sensor. Of course, the image from the camera unit 208 can also be used in application software.

[0145] In addition, image data of still images or videos can be added with location information acquired by the GNSS receiving unit 214, audio information acquired by the microphone 206 (which may be converted to text information by the main control unit, etc.), posture information acquired by the motion sensor unit 215, etc., and stored in the memory unit 212 or output via the external input / output unit 213 or wireless communication unit 210.

[0146] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0147] This application is based on a Japanese patent application (Patent Application No. 2023-168348) filed on September 28, 2023, the contents of which are incorporated herein by reference.

[0148] REFERENCE SIGNS LIST 1 Imaging lens A1, A2, A3, A4 Range 4 Lens control unit 5 Imaging element 8 Lens driving unit 9 Driving unit 11 System control unit 14, 207 Operation unit 15 Memory control unit 16 Memory 17 Digital signal processing unit 20 External memory control unit 21 Storage medium 22 Display device 22a Display controller 22b Display surface 24 Control bus 25 Data bus 40 Lens device 60 Imaging surface 61, 61B, 61G, 61R Pixel 61FA First phase difference detection pixel 61FB Second phase difference detection pixel 61FC Third phase difference detection pixel 61FD Fourth phase difference detection pixel 62 Pixel row 63 Driving circuit 64 Signal processing circuit 100 Digital camera 100A Main body 200 Smartphone 201 Housing 202 Display panel 203 Operation panel 204 Display input unit 205 Speaker 206 Microphone 208 Camera unit 210 Wireless communication unit 211 Call unit 212 Memory unit 213 External input / output unit 214 GNSS receiving unit 215 Motion sensor unit 216 Power supply unit 217 Internal memory unit 218 External memory unit 220 Main control unit

Claims

1. An information processing device that processes the output of an image sensor having a plurality of pixels including a photoelectric conversion unit that converts light collected by a microlens into an electric charge, wherein the plurality of pixels include a first pixel that shares the microlens with an adjacent pixel, and a second pixel that is different from the first pixel, and the information processing device is equipped with a processor that determines the presence or absence of abnormal light incident on the first pixel based on a first pixel value of the first pixel and a second pixel value of the second pixel.

2. An information processing device according to claim 1, wherein the second pixel is a pixel having the microlens alone.

3. An information processing device according to claim 2, wherein the processor determines the presence or absence of abnormal light based on a comparison result between the first pixel value and the second pixel value.

4. An information processing device according to claim 3, wherein the processor determines the presence or absence of abnormal light based on the comparison result and a threshold value.

5. An information processing device according to claim 4, wherein the comparison result is a ratio between the first pixel value and the second pixel value, or a difference between the first pixel value and the second pixel value.

6. An information processing device according to claim 5, wherein the processor determines that the abnormal light is present when the comparison result is equal to or greater than the threshold value.

7. An information processing device according to any one of claims 4 to 6, wherein the processor controls the threshold value.

8. An information processing device according to claim 7, wherein the processor changes the threshold value based on the position of the first pixel that is the subject of the determination of the presence or absence of abnormal light.

9. An information processing device according to claim 7, wherein the processor changes the threshold value based on the conditions of an imaging optical system disposed between the imaging element and a subject.

10. An information processing device according to claim 9, wherein the conditions include at least one of an aperture value of an aperture included in the imaging optical system and a focal length of the imaging optical system.

11. An information processing device as described in any one of claims 1 to 6, wherein the first pixel includes a brightness filter, the second pixel includes a color filter, the plurality of pixels include a plurality of types of second pixels having different types of color filters, and the processor determines the presence or absence of abnormal light incident on the first pixel based on a first pixel value of the first pixel and the second pixel value of any of the plurality of types of second pixels.

12. An information processing device as described in any one of claims 1 to 6, wherein the first pixel and the second pixel include color filters, the plurality of pixels include a plurality of types of first pixels having different types of color filters and a plurality of types of second pixels having different types of color filters, and the processor determines the presence or absence of abnormal light incident on the first pixel based on a first pixel value of the first pixel including the first type of color filter and the second pixel value of the second pixel including the first type of color filter.

13. An information processing device according to any one of claims 1 to 6, wherein the processor performs focus detection processing based on the pixel value of the first pixel.

14. An information processing device according to claim 13, wherein, when the processor determines that there is a first pixel into which the abnormal light is incident, it performs focus detection processing based on the pixel values ​​of the first pixels excluding at least the first pixel.

15. An information processing device according to claim 14, wherein, when the processor determines that there is a first pixel into which the abnormal light is incident, it performs focus detection processing based on the pixel value of the first pixel excluding the first pixel and the first pixels surrounding the first pixel.

16. An information processing device according to any one of claims 1 to 6, wherein the processor determines the abnormal light by further using a comparison result of pixel values ​​of a plurality of the first pixels including a plurality of the photoelectric conversion units in which the microlenses are shared.

17. An information processing device according to any one of claims 1 to 6, wherein the first pixels each have a microlens shared by two adjacent first pixels.

18. An imaging device comprising: an information processing device according to any one of claims 1 to 6; and the imaging element.

19. An information processing method for processing the output of an imaging element having a plurality of pixels including a photoelectric conversion unit that converts light collected by a microlens into an electric charge, wherein the plurality of pixels include a first pixel that shares the microlens with an adjacent pixel, and a second pixel that is different from the first pixel, and the information processing method includes a step of determining the presence or absence of abnormal light incident on the first pixel based on a first pixel value of the first pixel and a second pixel value of the second pixel.

20. An information processing program for processing the output of an imaging element having a plurality of pixels including a photoelectric conversion unit that converts light collected by a microlens into an electric charge, wherein the plurality of pixels include a first pixel that shares the microlens with an adjacent pixel, and a second pixel that is different from the first pixel, and the information processing program causes a processor to execute a step of determining whether or not abnormal light is incident on the first pixel based on a first pixel value of the first pixel and a second pixel value of the second pixel.

Images