Imaging element and imaging device

Abstract

This invention provides a camera element and a camera device. The camera element is formed by repeatedly arranging a first pixel array in a first direction and a second direction, wherein M is set to a natural number of (3) or more, and N is set to a natural number of (3) or more. The first pixel array is composed of M pixels arranged in the first direction and N pixels arranged in the second direction. The first pixel array includes a first pixel and a second pixel. The first pixel is included in a second pixel array composed of M-1 pixels arranged in the first direction and N-1 pixels arranged in the second direction. The second pixel is a pixel other than the first pixel. The functions of the first pixel and the second pixel are different.

Description

Technical Field

[0001] The present invention relates to a camera element and a camera device. Background Technology

[0002] Patent document 1 describes an imaging element having a pixel group for imaging and a pixel group for phase difference detection.

[0003] Patent document 2 describes an imaging element configured such that rows of pixels for imaging and rows of pixels for phase difference detection are arranged alternately, and a transparent filter is provided on the phase difference pixels.

[0004] Patent document 3 describes a camera element having a pixel arrangement pattern consisting of high-sensitivity pixels and low-sensitivity pixels.

[0005] Patent document 4 describes an imaging element that includes a phase difference detection pixel, a pixel including a polarizer, and a pixel including a white filter.

[0006] Previous technical documents Patent documents Patent Document 1: Japanese Patent Application Publication No. 2015-216519 Patent Document 2: Japanese Patent Application Publication No. 2021-158668 Patent Document 3: Japanese Patent Application Publication No. 2013-038504 Patent Document 4: Japanese Patent Application Publication No. 2012-212978 Summary of the Invention

[0007] means for solving technical problems The imaging element and imaging device of one embodiment of the technology of the present invention are shown below. (1) An imaging element is an imaging element formed by repeatedly arranging a first pixel array in a first direction and a second direction, wherein, Set M to a natural number greater than or equal to 3, and set N to a natural number greater than or equal to 3. The aforementioned first pixel array is composed of M pixels arranged in the aforementioned first direction and N pixels arranged in the aforementioned second direction. The aforementioned first pixel array includes a first pixel and a second pixel. The first pixel is contained within a second pixel array consisting of M-1 pixels arranged in the first direction and N-1 pixels arranged in the second direction. The second pixel is any pixel other than the first pixel. The functions of the first pixel and the second pixel mentioned above are different. (2) According to the imaging element described in (1), wherein, The second pixel is continuously arranged in at least one of the first direction and the second direction. (3) According to the imaging element described in (1) or (2), wherein, The second pixel mentioned above includes a first type of pixel whose pixel structure is different from that of the first pixel mentioned above. (4) According to the imaging element described in (3), wherein, The first type of pixel mentioned above includes pixels used for phase difference detection. (5) According to the imaging element described in (4), wherein, The aforementioned phase difference detection pixel is positioned outside the position of the pixel in the aforementioned second direction where the aforementioned second pixel array is arranged. (6) According to the imaging element described in (4) or (5), wherein, The aforementioned phase difference detection pixel is positioned at a location other than the position of the pixel in the aforementioned first direction where the aforementioned second pixel array is arranged. (7) The imaging element according to any one of (4) to (6), wherein, The shape of the microlens of the aforementioned phase difference detection pixel is different from that of the aforementioned first pixel. (8) According to the imaging element described in (7), wherein, The aforementioned phase difference detection pixels share a single microlens across multiple pixels. (9) According to the imaging element described in (3), wherein, The first type of pixel mentioned above includes polarization pixels used for polarization image acquisition. (10) According to the imaging element described in (3), wherein, The first type of pixel mentioned above includes pixels with an opening area different from the first type of pixel mentioned above. (11) According to the imaging element described in (10), wherein, The aperture area of ​​the pixel that is different from the first pixel is smaller than the aperture area of ​​the first pixel. (12) According to the imaging element described in (1) or (2), wherein, The second pixel mentioned above includes a second type of pixel whose spectral characteristics are different from those of the first pixel mentioned above. (13) According to the imaging element described in (12), wherein, The second type of pixel mentioned above includes pixels with or without a brightness filter. (14) According to the imaging element described in (1) or (2), wherein, The second pixel mentioned above includes a third type of pixel whose sensitivity characteristics are different from those of the first pixel mentioned above. (15) According to the imaging element described in (14), wherein, The first pixel mentioned above contains multiple pixels corresponding to multiple colors. The third type of pixel mentioned above includes multiple pixels corresponding to the aforementioned multiple colors. (16) According to the imaging element described in (14) or (15), wherein, The sensitivity of the third type of pixel is lower than that of the first type of pixel. (17) The imaging element according to any one of (1) to (16), wherein, The first pixel mentioned above contains multiple pixels corresponding to multiple colors. The second pixel mentioned above contains multiple pixels corresponding to the aforementioned multiple colors. The aforementioned imaging element includes: a first conversion circuit that converts the analog signal output from the first pixel into a digital signal; and The second conversion circuit converts the analog signal output from the aforementioned multiple pixels of the second pixel into a digital signal. The noise characteristics of the first conversion circuit and the second conversion circuit are different. (18) The imaging element according to any one of (1) to (16) comprises: The first conversion circuit converts the pixel signal output from the first pixel into a digital signal; and The second conversion circuit converts the pixel signal output from the second pixel into a digital signal. The signal conversion speeds of the first conversion circuit and the second conversion circuit described above are different. (19) According to the imaging element described in (3), wherein, The first type of pixel mentioned above includes at least one of a distance measurement pixel and a brightness change detection pixel. (20) The imaging element according to any one of (1) to (19), wherein, The arrangement pattern of the first pixel in the second pixel array becomes a pattern capable of capturing a color image. (twenty one) A camera device comprising a camera element and a processor as described in any one of (1) to (20). (twenty two) According to the camera device described in (21), wherein, The processor performs the following control: changes the exposure time in at least a portion of the second pixel and the first pixel. (twenty three) According to the camera device described in (22), wherein, The arrangement pattern of at least a portion of the second pixels mentioned above becomes a pattern capable of capturing a color image. (twenty four) According to the camera device described in (23), wherein, The arrangement pattern of the first pixel in the second pixel array becomes a pattern capable of capturing a color image. (25) A camera element having a first pixel array repeatedly arranged in a first direction and a second direction, wherein, Set L to an odd number greater than 3. The first pixel array described above is composed of L pixels arranged in the second direction arranged in the first direction. The aforementioned first pixel array includes: a first pixel, contained in a second pixel array formed by L-1 pixels arranged in the second direction and arranged in the first direction; and a second pixel other than the aforementioned first pixel. The functions of the first pixel and the second pixel mentioned above are different. Attached Figure Description

[0033] Figure 1 This is a diagram showing a schematic structure of a digital camera 100, which is an embodiment of the imaging device according to the technology of the present invention.

[0034] Figure 2 It means Figure 1 A schematic plan view of the general structure of the camera element 5 shown.

[0035] Figure 3 It is a local magnification representation Figure 2 A schematic diagram of the imaging surface 60 of the imaging element 5 shown.

[0036] Figure 4 This is a cross-sectional view of the first pixel 61R, the first pixel 61G, and the first pixel 61B.

[0037] Figure 5 It means to Figure 3The diagram shows an example of a first type of pixel, where a portion of the five second pixels 61F in the first pixel array P1 is set to a pixel structure that is different from that of the first pixel 61 contained in the second pixel array P2.

[0038] Figure 6 yes Figure 5 The diagram shows cross-sectional views of the first type of pixel 61FXr and the first type of pixel 61FXl.

[0039] Figure 7 It means to Figure 3 The diagram shows an example of a first type of pixel with a different pixel structure from the first pixel 61 contained in the second pixel array P2, where another portion of the five second pixels 61F in the first pixel array P1 is set as a first type of pixel.

[0040] Figure 8 It means to Figure 3 The diagram shows an example of a first type of pixel where all five second pixels 61F in the first pixel array P1 are set to have a different pixel structure than the first pixel 61 contained in the second pixel array P2.

[0041] Figure 9 This is a diagram showing a modified example of the pixel structure of a pixel used for phase difference detection.

[0042] Figure 10 It means Figure 8 A diagram showing a variation of the pixel configuration.

[0043] Figure 11 It means to Figure 3 The diagram shows an example of polarized pixels used for image acquisition, where all five second pixels 61F in the first pixel array P1 are set to have a different pixel structure than the first pixel 61 contained in the second pixel array P2.

[0044] Figure 12 yes Figure 11 The diagram shows a cross-sectional view of the polarization pixel 61FP.

[0045] Figure 13 It means to Figure 3 The diagram shows an example of a first type of pixel with a different pixel structure than the first pixel 61 contained in the second pixel array P2. The five second pixels 61F in the first pixel array P1 are shown as another type of first pixel.

[0046] Figure 14 yes Figure 13 The first type of pixel 61FR, the first type of pixel 61FG, and the first type of pixel 61FB shown are... Figure 13 The diagram shows cross-sectional views of the first pixel 61R, the first pixel 61G, and the first pixel 61B.

[0047] Figure 15 It means Figure 13 The first type of pixel 61FR, the first type of pixel 61FG, and the first type of pixel 61FB shown are... Figure 13 A schematic diagram of a modified example of the cross-section of the first pixel 61R, the first pixel 61G, and the first pixel 61B.

[0048] Figure 16 Observation from the ML side of the microlens Figure 15 The diagram shows the first type of pixel 61FR, the first type of pixel 61FG, and the first type of pixel 61FB.

[0049] Figure 17 It means to Figure 3 The diagram shows an example of five second pixels 61F in the first pixel array P1, which are shown as second pixels with different spectral characteristics from the first pixel 61 contained in the second pixel array P2.

[0050] Figure 18 It means Figure 17 A schematic diagram of the first example of the cross-sectional structure of the second type of pixel 61FL shown.

[0051] Figure 19 It means Figure 17 A schematic diagram of the second example of the cross-sectional structure of the second type of pixel 61FL shown.

[0052] Figure 20 It means Figure 17 A schematic diagram of the third example of the cross-sectional structure of the second type of pixel 61FL shown.

[0053] Figure 21 It means to Figure 3 The diagram shows an example of a third type of pixel with different sensitivity characteristics from the five second pixels 61F in the first pixel array P1, which are shown in the second pixel array P2.

[0054] Figure 22 It means to Figure 3 The diagram shows an example of a portion of the five second pixels 61F in the first pixel array P1 having the same structure as the first pixel 61 contained in the second pixel array P2.

[0055] Figure 23 It means to include Figure 3 The diagram shows an example of the structure of the analog-to-digital conversion circuit included in the signal processing circuit 64 of the imaging element 5 of the imaging surface 60 with the pixel configuration shown.

[0056] Figure 24 It means to Figure 3The diagram shows another example of a portion of the five second pixels 61F in the first pixel array P1 being configured with the same structure as the first pixel 61 contained in the second pixel array P2.

[0057] Figure 25 yes Figure 2 The modified example of the imaging surface 60 of the imaging element 5 shown is a schematic diagram of the imaging surface 60 partially enlarged.

[0058] Figure 26 This is a diagram showing the appearance of the smartphone 200.

[0059] Figure 27 It means Figure 26 The diagram shows the structure of the smartphone 200. Detailed Implementation

[0060] Figure 1 This is a diagram showing a schematic structure of a digital camera 100, which is an embodiment of the imaging device according to the technology of the present invention. Figure 1 The digital camera 100 shown includes: a lens assembly 40, which has a camera lens 1, an aperture 2, a lens drive unit 8 for driving the camera lens 1, an aperture drive unit 9 for driving the aperture 2, and a lens control unit 4 for controlling the lens drive unit 8 and the aperture drive unit 9; and a main body 100A.

[0061] The main body 100A includes: an image sensor 5; a system control unit 11, which centrally controls the entire electrical control system of the digital camera 100; an operation unit 14; a display device 22; a memory 16, including RAM (Random Access Memory) and ROM (Read Only Memory); a memory control unit 15, which controls the data storage to and from the memory 16; a digital signal processing unit 17; and an external memory control unit 20, which controls the data storage to and from the storage medium 21.

[0062] The lens assembly 40 can be a device that can be detached from the main body 100A, or it can be an integrated device with the main body 100A. The camera lens 1 includes a focusing lens.

[0063] A focusing lens is a lens used to adjust the focal point of a camera optical system, including a camera lens 1 and an aperture 2, and is composed of a single lens or multiple lenses. By moving the focusing lens along the optical axis, the position of the principal point of the focusing lens (hereinafter also referred to as the focusing lens position) changes along the optical axis, thereby changing the focal point position on the subject side. Alternatively, a liquid lens that allows the position of the principal point in the optical axis direction to be changed electrically can be used as the focusing lens.

[0064] The lens control unit 4 of the lens assembly 40 controls the lens drive unit 8 according to the lens drive signal sent from the system control unit 11, thereby changing the position of the focusing lens. The lens control unit 4 of the lens assembly 40 controls the aperture drive unit 9 according to the drive control signal sent from the system control unit 11, thereby changing the aperture 2 opening size.

[0065] The imaging element 5 captures an image of the subject using the aforementioned imaging optical system positioned between the imaging element 5 and the subject. The imaging element 5 has an imaging surface 60 (reference) with multiple pixels arranged in a two-dimensional configuration. Figure 2 The image of the subject imaged onto the imaging surface 60 by the imaging optical system is converted into an image signal by the multiple pixels and output. The output of the pixels included in the imaging element 5 is recorded as a pixel signal, and the set of pixel signals is recorded as an image signal.

[0066] The imaging element 5 uses a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. The following example illustrates an imaging element 5 using a CMOS image sensor.

[0067] The system control unit 11 centrally controls the entire digital camera 100, and its hardware structure consists of various processors that execute programs for processing. The programs executed by the system control unit 11 (including the control program for the imaging element 5) are stored in the ROM (non-temporary storage medium) of the memory 16.

[0068] Processors come in various forms, including general-purpose processors that execute programs for various processes, such as CPUs (Central Processing Units); programmable logic devices (PLDs) whose circuit structures can be modified after manufacturing FPGAs (Field Programmable Gate Arrays); and application-specific integrated circuits (ASICs) with circuit structures specifically designed to perform specific processes. More specifically, the structures of these various processors are circuits composed of combined semiconductor elements and other circuit components.

[0069] The system control unit 11 can be composed of one of various processors, or it can be composed of a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs or a combination of CPU and FPGA).

[0070] The system control unit 11 drives the imaging element 5 and the lens assembly 40, and outputs the image of the subject captured by the imaging optical system of the lens assembly 40 as an image signal. The image signal output from the imaging element 5 is processed by the digital signal processing unit 17 to generate data suitable for display on the display device 22 or data suitable for storage in the storage medium 21, i.e., imaging image data.

[0071] Instructions from the user are input to the system control unit 11 via the operation unit 14. The operation unit 14 includes a touch panel integrated with the display surface 22b and various buttons.

[0072] The display device 22 includes a display surface 22b composed of an organic EL (electroluminescence) panel or a liquid crystal panel, and a display controller 22a for controlling the display of the display surface 22b.

[0073] The memory control unit 15, digital signal processing unit 17, external memory control unit 20, and display controller 22a are connected to each other via control bus 24 and data bus 25, and are controlled by instructions from system control unit 11.

[0074] Figure 2 It means Figure 1 The diagram shows a schematic plan view of the imaging element 5. The imaging element 5 includes: an imaging surface 60, in which a plurality of pixel rows 62, each consisting of a plurality of pixels 61 arranged in the row direction X, are arranged in the column direction Y, which intersects the row direction X; a driving circuit 63, which drives the pixels 61 arranged in the imaging surface 60; and a signal processing circuit 64, which processes the pixel signals read from each pixel 61 of the pixel rows 62 arranged in the imaging surface 60 to a signal line.

[0075] exist Figure 2 In this example, the row direction X is orthogonal to the column direction Y. One direction of the row direction X is denoted as the right direction XR, and the other direction is denoted as the left direction XL. One direction of the column direction Y is denoted as the up direction YU, and the other direction is denoted as the down direction YD.

[0076] Alternatively, it can be said that the camera surface 60 has multiple pixel columns arranged along the row direction X, each consisting of multiple pixels 61 arranged along the column direction Y. The row direction X constitutes the first direction, and the column direction Y constitutes the second direction.

[0077] Figure 3 It is a local magnification representation Figure 2The diagram shows the imaging surface 60 of the imaging element 5. Hereinafter, M and N will be defined as natural numbers of 3 or higher. The upper limits for M and N are not limited to these, but are preferably set to 8.

[0078] On the camera surface 60, M ( ) are arranged along the row direction X. Figure 3 In this example, there are 3 pixels (61 pixels) arranged along the column direction Y, with N pixels in total. Figure 3 In the example, the first pixel array P1, consisting of 3 pixels 61, is repeatedly arranged along the row direction X and the column direction Y. The first pixel array P1 is configured as N pixel groups arranged in the column direction Y, each consisting of M pixels 61 arranged in the row direction X. On the imaging surface 60, there are multiple first pixel arrays P1 arranged along the row direction X and multiple arrangements along the column direction Y.

[0079] The first pixel array P1 consists of M-1 pixels arranged along the row direction. Figure 3 In this example, there are 2) pixels, 61 in total, and N-1 are arranged along the column direction Y. Figure 3 In the example, the second pixel array P2 is composed of 2 pixels 61. The second pixel array P2 is composed of N-1 pixel groups arranged in the column direction Y, each consisting of M-1 pixels 61 arranged in the row direction X. The pixels 61 contained in the second pixel array P2 are also referred to as the first pixel 61. The pixels 61 in the first pixel array P1 other than the first pixel 61 are also referred to as the second pixel 61F.

[0080] The second pixel array P2 includes multiple first pixels 61 corresponding to multiple wavelength bands. Figure 3 In the example, the second pixel array P2 includes a first pixel 61R (the block marked with the character "R" in the figure) corresponding to the wavelength band of red light, two first pixels 61G (the blocks marked with the character "G" in the figure) corresponding to the wavelength band of green light, and a first pixel 61B (the block marked with the character "B" in the figure) corresponding to the wavelength band of blue light. The first pixels 61R, 61G, and 61B respectively receive the corresponding wavelength band and output the pixel signal corresponding to its light intensity.

[0081] The arrangement pattern of the first pixel 61 in the second pixel array P2 becomes a pattern capable of capturing color images. Figure 3 In the example, in the second pixel array P2, the first pixel 61 is arranged according to a Bayer pattern. The arrangement pattern of the first pixel 61 in the second pixel array P2 is not limited to the Bayer pattern, and various patterns can be used.

[0082] The first pixel array P1 includes the first pixel 61 contained in the second pixel array P2 (in Figure 3In the example, there are four pixels: the first pixel 61R, two first pixels 61G, and the first pixel 61B, and the second pixel 61F (the five blocks marked with the character "F" in the diagram). The second pixel 61F is configured to surround the second pixel array P2. Figure 3 In the example, it becomes a structure with 3 consecutively arranged in the row direction X and 3 consecutively arranged in the column direction Y.

[0083] The functions of the first pixel 61R, the first pixel 61G, the first pixel 61B, and the second pixel 61F are different. The five second pixels 61F contained in the first pixel array P1 can all have the same function, or they can contain multiple pixels with different functions.

[0084] The functional difference between the two pixels is obtained by at least one of the following: the difference in pixel structure of the two pixels, the difference in spectral characteristics of the two pixels, the difference in sensitivity characteristics of the two pixels, and the difference in the purpose of using the pixel signals output from the two pixels.

[0085] For example, the pixel signal of the first pixel 61 included in the second pixel array P2 is used for the purpose of recording imaging image data. The pixel signal of the second pixel 61F included in the first pixel array P1 is used for the purpose of controlling the quality of the recorded imaging image data (control of noise, sharpness, dynamic range, or color vibrancy, etc.). Alternatively, the pixel signal of the second pixel 61F included in the first pixel array P1 may be used, for example, to generate a real-time preview image, or to determine the material, color, texture, etc. of an object, or to obtain imaging image data with exposure and noise characteristics different from the aforementioned imaging image data, or to obtain distance information of the subject, or to detect moving parts of the subject.

[0086] Figure 4 This is a cross-sectional schematic diagram of the first pixel 61R, the first pixel 61G, and the first pixel 61B. The first pixels 61R, 61G, and 61B each include: a microlens ML that focuses light from the subject; a photoelectric conversion unit PD that converts the light focused by the microlens ML into electrical charge; a beam splitter CF disposed between the photoelectric conversion unit PD and the microlens ML and transmitting light of a specific wavelength band; and a light-shielding film LS disposed between the beam splitter CF and the photoelectric conversion unit PD. The photoelectric conversion unit PD is a photodiode formed in a semiconductor substrate such as silicon, but it can also be composed of an organic material film disposed on top of the semiconductor substrate.

[0087] The beam-splitting filter CF contained in pixel 61R (in Figure 4 The R filter (recorded as R filter) transmits red light, and the CF filter (in the first pixel 61G) contains a beam splitter filter (in... Figure 4The filter (referred to as G filter) transmits green light, and the spectral filter CF contained in the first pixel 61B (in...) Figure 4 The filter (recorded as a B filter) transmits blue light.

[0088] The differences in the image structure mentioned above refer to the presence or absence of the beam splitter filter CF, the presence or absence of the functional element disposed between the microlens ML and the photoelectric conversion unit PD, the difference in the shape or size of the opening of the light-shielding film LS, and the difference in the shape of the microlens ML, etc.

[0089] according to Figure 3 The pixel configuration shown allows for the generation of color imaging image data based on the pixel signal of the first pixel 61 of the second pixel array P2, without the need for interpolation processing of the pixel signals at each position of the first pixel 61. Therefore, the quality of the imaging image data can be improved or the processing load reduced. Furthermore, since the second pixel 61F is arranged around the second pixel array P2, the aforementioned quality control of the imaging image data can be performed using the pixel signal of the second pixel 61F, thereby improving the quality of the imaging image data. Moreover, by using the pixel signal of the second pixel 61F, detailed information such as the movement, color, or texture of the photographed subject can be obtained, thus enabling multi-functionality of the digital camera 100.

[0090] The following is a specific example of the five second pixels 61F in the first pixel array P1.

[0091] Figure 5 It means to Figure 3 The diagram shows an example of a first type of pixel whose pixel structure is different from that of the first pixel 61 contained in the second pixel array P2, where a portion of the five second pixels 61F in the first pixel array P1 is shown.

[0092] exist Figure 5 In the example, in the first pixel array P1, a first-type pixel 61FXr (the block labeled "FXr" in the figure) and a first-type pixel 61FXl (the block labeled "FXl" in the figure) for phase difference detection are arranged at positions other than the position of the first pixel 61 of the second pixel array P2 arranged in the column direction Y. Figure 5 In the example, it is preferable to set the remaining two second pixels 61F in the first pixel array P1 as pixels with functions different from the first type of pixel 61FXr and the first type of pixel 61FXl.

[0093] Figure 6 yes Figure 5 The diagram shows cross-sectional views of the first type of pixel 61FXr and the first type of pixel 61FXl. Figure 6As shown, the first type of pixel 61FXr and the first type of pixel 61FXl are structures in which the size of the opening of the light-shielding film LS is changed relative to the structure of the first type of pixel 61G. The opening of the light-shielding film LS of the first type of pixel 61FXr is such that, for example, the right half of the opening of the light-shielding film LS of the first type of pixel 61G is closed. The opening of the light-shielding film LS of the first type of pixel 61FXl is such that, for example, the left half of the opening of the light-shielding film LS of the first type of pixel 61G is closed.

[0094] Through the structure of this light-shielding film LS, the first pixel 61FXr receives one of a pair of light beams from two different portions of the pupil region of the imaging optical system of the lens assembly 40 arranged in the row direction X. The first pixel 61FXl receives the other portion of the pair of light beams. Furthermore, the first pixel 61 included in the second pixel array P2 receives both of the pair of light beams. The system control unit 11 detects the phase difference based on the pixel signals of the first pixel 61FXr and the first pixel 61FXl, and controls the position of the focusing lens based on this phase difference.

[0095] The beam-splitting filter CF included in the first type of pixel 61FXr and the first type of pixel 61FXl can be a filter other than the G filter. Furthermore, a brightness filter may be provided instead of the beam-splitting filter CF.

[0096] A luminance filter possesses spectral characteristics related to the luminance component of light and is equivalent to an ND (Neutral Density) filter, a transparent filter, a white filter, or a gray filter. A luminance filter allows more wavelength components of light to pass through the beam splitter filter (CF). Therefore, by replacing the beam splitter filter (CF) with a luminance filter, the accuracy of phase difference detection using the pixel signals of the first type pixel 61FXr and the first type pixel 61FXl can be improved.

[0097] The first type of pixel 61FXr and the first type of pixel 61FXl can be structures without a beam splitter filter CF and a luminance filter. In this way, more wavelength components of light can be transmitted compared to the beam splitter filter CF. Therefore, the accuracy of phase difference detection of pixel signals using the first type of pixel 61FXr and the first type of pixel 61FXl can be improved. In the case of a structure with a luminance filter instead of a beam splitter filter CF, or in the case of a structure without a beam splitter filter CF and a luminance filter, the first type of pixel 61FXr and the first type of pixel 61FXl can also be referred to as a second type of pixel with different spectral characteristics than the first pixel 61.

[0098] exist Figure 5In the pixel configuration shown, a first type pixel 61FXr and a first type pixel 61FXl are provided in pixel rows 62 other than the pixel row 62 containing the first type pixel 61 included in the second pixel array P2. Therefore, it is easy to control the exposure time in the pixel row 62 containing the first type pixel 61FXr and the first type pixel 61FXl and the pixel row 62 containing the first type pixel 61 included in the second pixel array P2.

[0099] For example, by extending the exposure time of the pixel row 62 containing the first type of pixel 61FXr and the first type of pixel 61FXl, the detection accuracy of the phase difference of the pixel signal output from the pixel row 62 containing the first type of pixel 61FXr and the first type of pixel 61FXl can be improved. By capturing the subject through the pixel row 62 containing the first pixel 61 included in the second pixel array P2 while the position of the focusing lens is controlled according to this phase difference, the sharpness of the imaging image data obtained by processing the pixel signal output from the pixel row 62 can be improved.

[0100] Figure 7 It means to Figure 3 The diagram shows an example of a first type of pixel whose pixel structure is different from that of the first pixel 61 contained in the second pixel array P2, where a portion of the five second pixels 61 in the first pixel array P1 is shown.

[0101] exist Figure 7 In the example, in the first pixel array P1, a position other than the position of the first pixel 61 of the second pixel array P2 is arranged in the row direction X. Figure 6 The first type of pixel 61FXr and the first type of pixel 61FXl are used for phase difference detection in the structure shown.

[0102] exist Figure 7 In the example, a first type pixel 61FXr is arranged to the left of the first pixel 61G in the upper left of the second pixel array P2, and a first type pixel 61FXl is arranged to the left of the first pixel 61B in the second pixel array P2. Figure 7 In the example, it is preferable to set the remaining three second pixels 61F in the first pixel array P1 as pixels with functions different from the first type of pixel 61FXr and the first type of pixel 61FXl.

[0103] exist Figure 7In the pixel configuration shown, pixel row 62, which includes the first pixel 61 contained in the second pixel array P2, contains both a first type pixel 61FXr and a first type pixel 61FXl. Therefore, it is possible to simultaneously read out pixel signals from the first type pixel 61FXr and the first type pixel 61FXl, and also read out pixel signals from the first pixel 61 contained in the second pixel array P2. Thus, even in continuous shooting or similar situations, it is possible to focus on the subject and take a picture.

[0104] Figure 8 It means to Figure 3 The diagram shows an example where all five second pixels 61F in the first pixel array P1 are set to be of the first type (pixels for phase difference detection) with a pixel structure different from that of the first pixel 61 contained in the second pixel array P2. Figure 8 The pixel configuration shown is to make Figure 5 The pixel configuration shown and Figure 7 The structure is composed of pixel configurations as shown. Figure 8 The structure shown can obtain Figure 5 The advantages and advantages explained in the text Figure 7 The advantages of both are explained in the text.

[0105] exist Figure 6 In the example, the phase difference can be detected by the first type of pixel 61FXr and the first type of pixel 61FXl through the shape of the light-shielding film LS, but this can also be achieved through the shape of the microlens ML.

[0106] For example, such as Figure 9 As shown, the structure can be configured as follows: the opening size of the light-shielding film LS of the first type of pixel 61FXr and the first type of pixel 61FXl is the same as that of the first pixel 61 included in the second pixel array P2. Instead, in adjacent first type of pixels 61FXr and 61FXl, the microlens ML is shared in the first type of pixels 61FXr and 61FXl, and the shape of the microlens ML is different from that of the first pixel 61. Figure 9 As shown, it is also possible to receive one of a pair of light beams consisting of two different portions of the pupil region of the camera optical system of the lens device 40 arranged in the X-direction through which the first pixel 61FXr passes, and to receive the other of the pair of light beams through the first pixel 61FXl. Figure 9 The structure of the first type of pixel 61FXr and the first type of pixel 61FXl has a wider aperture area, which can improve the detection accuracy of phase difference.

[0107] Figure 10 It means Figure 8 A diagram showing a variation of the pixel configuration. Figure 10 The pixel configuration shown is as follows: Figure 8 In the second pixel array P2, the left-side adjacent first pixel 61FXr of the upper left first pixel 61G is replaced with the first pixel 61FYu for phase difference detection, and the left-side adjacent first pixel 61FXl of the first pixel 61B in the second pixel array P2 is replaced with the first pixel 61FYd for phase difference detection. The first pixels 61FYu and 61FYd form a pixel structure capable of detecting the phase difference in the column direction Y.

[0108] exist Figure 10 In the diagram, a portion of the microlens ML is represented by a dashed line when the microlens ML is shared in adjacent first-type pixels 61FXr and 61FXl, and also shared in adjacent first-type pixels 61FYu and 61FYd. According to... Figure 10 The pixel configuration shown is capable of detecting phase difference in two directions.

[0109] Figure 11 It means to Figure 3 The diagram shows an example of obtaining polarized pixels with a different pixel structure than the five second pixels 61F in the first pixel array P1 and the first pixels 61 contained in the second pixel array P2.

[0110] exist Figure 11 In the example, all pixels 61 except for the second pixel array P2 become polarized pixels 61FP. Figure 12 yes Figure 11 The diagram shows a cross-sectional view of the polarization pixel 61FP. Figure 12 As shown, polarization pixel 61FP has a structure in which the beam-splitting filter CF in the first pixel 61 included in the second pixel array P2 is replaced by a polarizer PF, and the pixel structure is different from that of each first pixel 61 included in the second pixel array P2. The first pixel array P1 can also contain multiple polarization pixels 61FP, which contain polarizers PF with different vibration directions of the transmitted light. Polarization pixels FP constitute the first type of pixel.

[0111] like Figure 11 As shown, by setting the second pixel 61F as a polarizing pixel 61FP, it is possible to generate color imaging image data based on the pixel signal output from the second pixel array P2, and to generate at least one polarized image based on the pixel signal output from the polarizing pixel 61FP. Using the polarized image, the material, color, and texture of objects contained within the subject can be determined. Therefore, using this determination result, image processing of the imaging image data and subject recognition can be performed, thereby improving the quality of the imaging image data or achieving high-precision focus control.

[0112] Additionally, an example is shown here where all second pixels 61F are set to polarized pixels 61FP, but this is not a limitation. For example, one could... Figure 5 At least one of the two second pixels 61F of the first pixel array P1 in the array is set as a polarization pixel 61FP. Alternatively, it can be... Figure 7 At least one of the three second pixels 61F in the first pixel array P1 is set as a polarized pixel 61FP.

[0113] exist Figure 11 In the pixel configuration shown, at least one of the polarization pixels FP can be replaced with at least one of a distance measurement pixel and a brightness change detection pixel. The distance measurement pixel consists of pixels that acquire the distance to the subject using TOF (Time of Flight). The brightness change detection pixel is, for example, composed of pixels used in event-driven vision sensors. The pixel structures of both the distance measurement pixel and the brightness change detection pixel are different from the first pixel 61 included in the second pixel array P2, constituting a first type of pixel.

[0114] Figure 13 It shows that Figure 3 The diagram shows five second pixels 61F in the first pixel array P1, which are examples of first type pixels 61FR (squares marked "FR" in the figure), first type pixels 61FG (squares marked "FG" in the figure), and first type pixels 61FB (squares marked "FB" in the figure), which have different pixel structures from the first pixel 61 contained in the second pixel array P2.

[0115] Figure 14 yes Figure 13 The first type of pixel 61FR, the first type of pixel 61FG, and the first type of pixel 61FB shown are... Figure 13 The diagram shows cross-sectional views of the first pixel 61R, the first pixel 61G, and the first pixel 61B. The first type of pixels 61FR, 61FG, and 61FB have the same pixel structure except for the difference in the transmission wavelength range of the beam-splitting filter CF. Therefore, in Figure 14 Only one pixel is shown in the diagram. Similarly, only one pixel is shown for the first pixel 61R, the first pixel 61G, and the first pixel 61B.

[0116] The beam-splitter CF of pixel 61FR has the same spectral characteristics as the beam-splitter CF included in pixel 61R. The beam-splitter CF of pixel 61FG has the same spectral characteristics as the beam-splitter CF included in pixel 61G. The beam-splitter CF of pixel 61FB has the same spectral characteristics as the beam-splitter CF included in pixel 61B.

[0117] like Figure 14 As shown, the first type of pixel 61FR, the first type of pixel 61FG and the first type of pixel 61FB have the same pixel structure as the first type of pixel 61R, the first type of pixel 61G and the first type of pixel 61B, except that the opening area of ​​the light-shielding film LS of the first type of pixel 61FR, the first type of pixel 61FG and the first type of pixel 61FB is smaller than that of the first type of pixel 61R, the first type of pixel 61G and the first type of pixel 61B.

[0118] Furthermore, the positional relationship between the opening center of the light-shielding film LS and the center of the microlens ML in the first type of pixel 61FR, the first type of pixel 61FG, and the first type of pixel 61FB is the same as the positional relationship between the opening center of the light-shielding film LS and the center of the microlens ML in the first type of pixel 61R, the first type of pixel 61G, and the first type of pixel 61B.

[0119] according to Figure 13 The pixel configuration shown allows for two different exposures of the same subject: a first shot based on the first pixel 61R, the first pixel 61G, and the first pixel 61B, and a second shot based on the first pixel 61FR, the first pixel 61FG, and the first pixel 61FB. Thus, for the same subject, it is possible to take two shots with different exposures, thereby enabling the digital camera 100 to function as a multi-functional device.

[0120] Furthermore, in the first pixel array P1, as long as at least one first-type pixel 61FR, one first-type pixel 61FG, and one first-type pixel 61FB are provided, color photography can be performed using the first-type pixels 61FR, 61FG, and 61FB. Therefore, for example, in... Figure 13 In this configuration, the first type pixel 61FG located in the upper right corner and the first type pixel 61FR located to its left in the first pixel array P1 can be replaced with a second pixel 61F that performs other functions. For example, the second pixel 61F that performs other functions can be a pixel for phase difference detection, a polarization pixel, a distance measurement pixel, or a brightness change detection pixel, as described so far.

[0121] Figure 15 It means Figure 13 The first type of pixel 61FR, the first type of pixel 61FG, and the first type of pixel 61FB shown are... Figure 13 The diagram shows a modified cross-section of the first pixel 61R, the first pixel 61G, and the first pixel 61B. The first type of pixels 61FR, 61FG, and 61FB have the same pixel structure as the first pixel 61R, 61G, and 61B, except that a grid-shaped filter MF is added between the beam splitter filter CF and the light-shielding film LS.

[0122] Figure 16 Observation from the ML side of the microlens Figure 15 The image shows the first type of pixel 61FR, the first type of pixel 61FG, and the first type of pixel 61FB. (See image.) Figure 16 As shown, the grid filter MF is a filter with multiple openings Mfa, and the portion outside the openings Mfa becomes a light-blocking area. The total area of ​​these multiple openings Mfa is smaller than the opening area of ​​the light-blocking film LS of each of the first pixel 61R, the first pixel 61G, and the first pixel 61B. Even with the above structure, it is possible to capture two patterns of the same subject under different exposure conditions, thus enabling the digital camera 100 to achieve multi-functionality.

[0123] Figure 17 It means to Figure 3 The diagram shows an example of five second pixels 61F in the first pixel array P1, which are second pixels 61FL (blocks marked "FL" in the figure) with different spectral characteristics from the first pixel 61 contained in the second pixel array P2. Figure 18 It means Figure 17 A schematic diagram of the first example of the cross-sectional structure of the second type of pixel 61FL shown.

[0124] like Figure 18 As shown, the second type of pixel 61FL is a pixel structure in which the spectral filter CF in the first pixel 61 contained in the second pixel array P2 is replaced with a luminance filter LF. As mentioned above, the luminance filter LF has spectral characteristics related to the luminance component of light, and ND filters, transparent filters, white filters, or gray filters belong to this category. The luminance filter LF enables the transmission of light with more wavelength components than the spectral filter CF.

[0125] according to Figure 17 and Figure 18 The structure shown allows for the acquisition of more luminance components through the second pixel 61FL compared to the first pixel 61 contained in the second pixel array P2. Therefore, by utilizing the pixel signal of the second pixel 61FL, high-quality processing such as noise reduction can be performed on the imaging image data generated based on the pixel signal of the first pixel 61 contained in the second pixel array P2.

[0126] Figure 19 It means Figure 17 This is a schematic diagram of the second example of the cross-sectional structure of the second type of pixel 61FL. (See diagram below.) Figure 19 As shown, the second type of pixel 61FL is a pixel structure in which the beam-splitting filter CF of the first pixel 61 contained in the second pixel array P2 has been removed. The second type of pixel 61FL does not have a beam-splitting filter CF, so it can transmit more wavelength components of light compared to the first pixel 61 contained in the second pixel array P2.

[0127] according to Figure 19 The pixel structure shown can be more efficient than... Figure 18 The pixel structure acquires more luminance components. Therefore, by utilizing the pixel signal of the second pixel 61FL, further noise reduction and other quality enhancement processing can be performed on the imaging image data generated based on the pixel signal of the first pixel 61 contained in the second pixel array P2.

[0128] Figure 20 It means Figure 17 A schematic diagram of the third example of the cross-sectional structure of the second type of pixel 61FL shown. (See diagram below.) Figure 20 As shown, the second type of pixel 61FL is a pixel structure in which the spectral filter CF in the first pixel 61 contained in the second pixel array P2 is replaced with a specific filter CFa. The specific filter CFa has different spectral characteristics than the spectral filter CF of the first pixel 61 contained in the second pixel array P2.

[0129] A specific filter CFa is, for example, any one of the following: a filter that transmits infrared light, a filter that transmits ultraviolet light, a filter that transmits cyan light, a filter that transmits yellow light, a filter that transmits magenta light, a filter that has a narrower or wider spectral characteristic than the spectral filter CF of the first pixel 61R, a filter that has a narrower or wider spectral characteristic than the spectral filter CF of the first pixel 61G, or a filter that has a narrower or wider spectral characteristic than the spectral filter CF of the first pixel 61B.

[0130] exist Figure 17 In the example, there are 5 second-type pixels 61FL. Therefore, as these 5 second-type pixels 61FL, it is possible to illustrate a structure that includes a second-type pixel 61FL that transmits infrared light, a second-type pixel 61FL that transmits ultraviolet light, a second-type pixel 61FL that transmits cyan light, a second-type pixel 61FL that transmits yellow light, and a second-type pixel 61FL that transmits magenta light.

[0131] according to Figure 20 The structure shown allows for the determination of colors of subjects that cannot be determined in the first pixel 61 of the second pixel array P2, using the second pixel 61FL. Therefore, by utilizing the pixel signal of the second pixel 61FL, colors of specific wavelengths can be emphasized (made more vibrant) in the image data generated based on the pixel signal of the first pixel 61 contained in the second pixel array P2. Furthermore, by utilizing the pixel signal of the second pixel 61FL, it is possible to identify objects that cannot be recognized in the second pixel array P2.

[0132] Figure 21 It means to Figure 3 The diagram shown illustrates an example of a third type of pixel whose sensitivity characteristics differ from those of the first pixel 61 contained in the second pixel array P2. (The diagram is incomplete and requires further context.) Figure 21 In the example, the first pixel array P1 contains two third-type pixels 61Fr (blocks marked "Fr" in the figure), two third-type pixels 61Fg (blocks marked "Fg" in the figure), and one third-type pixel 61Fb (blocks marked "Fb" in the figure).

[0133] The third type of pixel 61Fr has the same spectral characteristics as the first type of pixel 61R, but with lower sensitivity. The third type of pixel 61Fg has the same spectral characteristics as the first type of pixel 61G, but with lower sensitivity. The third type of pixel 61Fb has the same spectral characteristics as the first type of pixel 61B, but with lower sensitivity. The sensitivity difference between the two pixels can be achieved through improvements to the light-shielding film LS, microlens ML, or beam-splitting filter CF.

[0134] according to Figure 21 The pixel configuration shown enables a first image capture of the same subject, achieved by first pixels 61R, 61G, and 61B, and a third image capture, achieved by third pixels 61Fr, 61Fg, and 61Fb, with a sensitivity different from the first image capture. Therefore, for example, by using the image signal obtained in the first image capture and the image signal obtained in the third image capture, it is possible to perform image capture with extended dynamic range.

[0135] Furthermore, as long as at least one third-type pixel 61Fr, one third-type pixel 61Fg, and one third-type pixel 61Fb are provided in the first pixel array P1, the dynamic range of the image captured by the second pixel array P2 can be extended. Therefore, for example, in Figure 21 In this configuration, the third type pixel 61Fg located in the upper right corner and the third type pixel 61Fr located to its left in the first pixel array P1 can be replaced with a second pixel 61F that performs other functions. As a second pixel 61F that performs other functions, for example, the first type pixel (pixel for phase difference detection, polarization pixel, distance measurement pixel, or brightness change detection pixel) or the second type pixel described so far can be used.

[0136] Figure 22 It means to Figure 3 The diagram shows an example where a portion of the five second pixels 61F in the first pixel array P1 is configured to have the same structure as the first pixels 61 contained in the second pixel array P2. Figure 22In the example, the configuration is such that in the first pixel array P1, a pixel 61fg with the same spectral characteristics, sensitivity characteristics and pixel structure as the first pixel 61G is arranged next to the first pixel 61G in the upper left corner; a pixel 61fb with the same spectral characteristics, sensitivity characteristics and pixel structure as the first pixel 61B is arranged next to the first pixel 61B in the upper left corner; and a pixel 61fr with the same spectral characteristics, sensitivity characteristics and pixel structure as the first pixel 61R is arranged next to the pixel 61fg in the upper left corner.

[0137] exist Figure 22 The diagram shows a structural example of the analog-to-digital conversion circuit included in the signal processing circuit 64 of the imaging element 5. The signal processing circuit 64 includes: a second conversion circuit 64A, which is configured corresponding to a specific pixel column consisting only of pixels 61fr, 61fg, and 61fb, and converts analog signals into digital signals; and a first conversion circuit 64B, which is configured corresponding to pixel columns other than the specific pixel column, and converts analog signals into digital signals.

[0138] The second conversion circuit 64A and the first conversion circuit 64B have different noise characteristics. The first conversion circuit 64B has the following noise characteristics: the noise level contained in the digital signal at ISO (International Organization for Standardization) sensitivity settings above the threshold becomes a first level proportional to the ISO sensitivity. The second conversion circuit 64A has the following noise characteristics: the noise level contained in the digital signal at ISO sensitivity settings above the aforementioned threshold becomes a second level lower than the first level. Therefore, when using the second conversion circuit 64A to convert analog signals to digital signals, compared to using the first conversion circuit 64B to convert analog signals to digital signals, noise contained in the pixel signal can be reduced when shooting at high ISO sensitivity settings.

[0139] according to Figure 22 The pixel configuration and signal processing circuit 64 shown can, even in dark shooting environments and with high ISO sensitivity, obtain a first pixel signal obtained by shooting based on the first pixel 61R, the first pixel 61G, and the first pixel 61B, and a second pixel signal with low noise obtained by shooting based on the first pixel 61fr, the first pixel 61fg, and the first pixel 61fb for the same subject. Therefore, by using these first pixel signals and second pixel signals, low-noise imaging image data can be generated.

[0140] exist Figure 22 In this array, the arrangement pattern of pixels 61fr, 61fg, and 61fb can be the same as that of the second pixel array P2, which is a Bayer pattern.

[0141] Figure 23 It means including Figure 3 This diagram illustrates an example of the structure of the analog-to-digital conversion circuit included in the signal processing circuit 64 of the imaging element 5 of the imaging surface 60 with the shown pixel configuration. Figure 23 In this context, each second pixel 61F, which is the first pixel array P1, can adopt the various structures described above.

[0142] Figure 23 The signal processing circuit 64 shown includes: a second conversion circuit 64C, which is configured corresponding to a second specific pixel column consisting only of the second pixel 61F, and converts analog signals into digital signals; and a first conversion circuit 64D, which is configured corresponding to pixel columns other than the second specific pixel column, and converts analog signals into digital signals.

[0143] The signal conversion speeds of the second conversion circuit 64C and the first conversion circuit 64D are different. The signal conversion speed of the second conversion circuit 64C is faster than that of the first conversion circuit 64D. According to... Figure 23 The structure of the signal processing circuit 64 shown allows for faster reading of a portion of the pixel signal of the second pixel 61F than the first pixel 61 of the second pixel array P2. Therefore, it enables high-speed implementation of functions utilizing a portion of the pixel signal of the second pixel 61F. For example, if in Figure 7 Applicable in the pixel configuration shown Figure 23 The structure of the signal processing circuit 64 allows it to read out pixel signals from the first type of pixel 61FXr and the first type of pixel 61FXl faster than the second pixel array P2. As a result, it enables high-speed control of the focus position.

[0144] In addition, regarding Figure 23 The pixel column corresponding to the first conversion circuit 64D shown can be configured with two circuits: conversion circuit AD1, which converts the pixel signal of the first pixel 61 in the pixel column to a digital signal, and conversion circuit AD2, which converts the pixel signal of the second pixel 61F in the pixel column to a digital signal. In this case, conversion circuit AD1 is configured with the same structure as the first conversion circuit 64D. Conversion circuit AD2 is configured with the same structure as the second conversion circuit 64C. In this way, the pixel signals of all second pixels 61F can be read out faster than those of the second pixel array P2.

[0145] Figure 24 It means to Figure 3 The diagram shows another example of a portion of the five second pixels 61F in the first pixel array P1 being configured with the same structure as the first pixels 61 contained in the second pixel array P2. Figure 24In the example, the first pixel array P1 can be configured with two types: first pixel array P1a and first pixel array P1b, forming a structure in which first pixel array P1a and first pixel array P1b are arranged alternately in the row direction X and column direction Y.

[0146] The structure is as follows: in the upper left of the first pixel 61G in the first pixel array P1a, a pixel 61fg with the same spectral characteristics, sensitivity characteristics and pixel structure as the first pixel 61G is arranged above it, and a pixel 61fr with the same spectral characteristics, sensitivity characteristics and pixel structure as the first pixel 61R is arranged on the left and right sides of the pixel 61fg.

[0147] The structure is as follows: a pixel 61fg is arranged above the first pixel 61G in the upper left of the first pixel array P1b, and a pixel 61fb with the same spectral characteristics, sensitivity characteristics and pixel structure as the first pixel 61B is arranged on the left and right sides of the pixel 61fg.

[0148] As Figure 24 The second pixel 61F in the image can adopt the various structures described above.

[0149] exist Figure 24 In the structure, the RG pixel row, in which pixels 61fr and 61fg alternate in the row direction X, and the GB pixel row, in which pixels 61fg and 61fb alternate in the row direction X, are alternated in the column direction Y, separated by two pixel rows containing the first pixel 61 in the second pixel array P2. The arrangement pattern of the pixels contained in the RG pixel row and the GB pixel row becomes a Bayer pattern-based arrangement pattern, which becomes a pattern capable of capturing color images.

[0150] The system control unit 11 controls the first exposure time of each pixel 61 in the RG pixel row and the BG pixel row, and the second exposure time of each pixel 61 in other pixel rows. Furthermore, the system control unit 11 controls the generation of a live view image based on the pixel signals read from each pixel 61 in the RG pixel row and the BG pixel row, and controls the generation of image data for recording based on the pixel signals read from each pixel 61 in other pixel rows.

[0151] The system control unit 11 can update the live view image at a high frequency, for example, by making the second exposure time longer than the first exposure time, even during recording shooting based on long exposure.

[0152] Furthermore, when using a structure that allows exposure time to be controlled on a pixel-by-pixel basis rather than on a pixel-by-pixel line basis as the imaging element 5, in Figure 24In this system, it is possible to control the exposure time of all pixels 61 except those in the second pixel array P2, and the exposure time of pixels 61 in the second pixel array P2. Thus, for example, during the capture of a subject using the second pixel array P2, a live view image can be generated and displayed based on pixel signals read from each pixel 61 in the RG and BG pixel rows. Furthermore, other functions such as focus position control can be performed based on pixel signals read from the second pixel 61F.

[0153] This control of exposure time applies even if the second pixel 61F contained in the first pixel array P1 has any function. For example, in Figure 24 In the pixel configuration shown, pixel 61fr can be changed to the first type of pixel 61FR or the third type of pixel 61Fr, pixel 61fg can be changed to the first type of pixel 61FG or the third type of pixel 61Fg, and pixel 61fb can be changed to the first type of pixel 61FB or the third type of pixel 61Fb.

[0154] Figure 25 yes Figure 2 The modified example of the imaging surface 60 of the imaging element 5 shown is a schematic diagram with a partial enlargement of the imaging surface 60. Hereinafter, L will be described as an odd number of 3 or more. The upper limit of L is not limited to this, but it is preferred to be 7.

[0155] exist Figure 25 In the example, on the camera surface 60, a first pixel array PX is repeatedly arranged along the row direction X and the column direction Y. This first pixel array PX is configured to be L (in) arranged along the column direction Y. Figure 25 In this example, let L=3) pixels 61 are arranged along the row direction X. On the imaging surface 60, multiple arrays PX of the first pixel arranged along the row direction X are arranged along the column direction Y. Figure 25 In the example, the first pixel array PX is configured such that a pixel group consisting of L pixels 61 arranged along the column direction Y is arranged along the row direction X in L-1 rows.

[0156] The first pixel array PX contains a second pixel array PY, which is configured as L-1 (in) elements arranged along the column direction Y. Figure 25 In this example, 2 pixels (61 pixels) are arranged along the X-axis. Figure 25 In the example, the second pixel array PY is configured such that L-1 pixel groups 61, which are arranged along the column direction Y, are arranged along the row direction X. The second pixel array PY has the same structure as the second pixel array P2. The pixels 61 in the first pixel array PX other than those in the second pixel array PY become pixels with different functions from the pixels 61 contained in the second pixel array PY (same as the second pixel 61F mentioned above).

[0157] according to Figure 25 The pixel configuration shown allows for the generation of color imaging image data based on the pixel signals of pixel 61 in the second pixel array PY, without the need for interpolation processing of the pixel signals at the location of pixel 61. Therefore, the quality of the imaging image data can be improved or the processing load reduced. Furthermore, since the second pixel 61F is provided, the aforementioned quality control of the imaging image data can be performed using the pixel signals of the second pixel 61F, thereby improving the quality of the imaging image data. Moreover, by using the pixel signals of the second pixel 61F, detailed information such as the movement, color, or texture of the photographed subject can be obtained, thus enabling multi-functionality of the digital camera 100.

[0158] Next, the structure of a smartphone, which is another embodiment of the camera device according to the technology of the present invention, will be described.

[0159] Figure 26 This is a diagram showing the appearance of the smartphone 200. Figure 26 The smartphone 200 shown has a flat frame 201, and a display input unit 204 is provided on one side of the frame 201, which integrates a display panel 202 as a display unit and an operation panel 203 as an input unit.

[0160] Furthermore, this frame 201 includes a speaker 205, a microphone 206, an operation unit 207, and a camera unit 208. In addition, the structure of the frame 201 is not limited to this; for example, it can also adopt a structure in which the display unit and the input unit are independent, or a structure with a folding structure or a sliding mechanism.

[0161] Figure 27 It means Figure 26 A block diagram of the structure of the smartphone 200 shown.

[0162] like Figure 27 As shown, the main components of a smartphone include a wireless communication unit 210, a display input unit 204, a call unit 211, an operation unit 207, a camera unit 208, a storage unit 212, an external input / output unit 213, a GNSS (Global Navigation Satellite System) receiver unit 214, a motion sensor unit 215, a power supply unit 216, and a main control unit 220.

[0163] Furthermore, as a primary function of the smartphone 200, it possesses wireless communication capabilities, enabling mobile wireless communication via a base station device BS (not shown) and a mobile communication network NW (not shown).

[0164] The wireless communication unit 210 performs wireless communication with the base station device BS housed in the mobile communication network NW according to the instructions of the main control unit 220. It uses this wireless communication to send and receive various types of data, such as voice data, image data, and email data, as well as to receive network data or streaming data.

[0165] The display input unit 204 is a touch panel that, under the control of the main control unit 220, displays images (still images and moving images) or text information to convey information to the user's vision and detects user operations on the displayed information. It also includes a display panel 202 and an operation panel 203.

[0166] Display panel 202 uses LCD (Liquid Crystal Display), OELD (Organic Electro-Luminescence Display), etc. as display devices.

[0167] The operation panel 203 is a device configured to visually recognize images displayed on the display surface of the display panel 202 and detect one or more coordinates operated by a user's finger or stylus. If the device is operated by a user's finger or stylus, a detection signal generated by the operation is output to the main control unit 220. Then, the main control unit 220 detects the operation position (coordinates) on the display panel 202 based on the received detection signal.

[0168] like Figure 27 As shown, in a smartphone 200 exemplified as one embodiment of the camera device of the present invention, the display panel 202 and the operation panel 203 are integrated to form a display input unit 204, but the operation panel 203 is configured to completely cover the display panel 202.

[0169] When this configuration is adopted, the operation panel 203 can also have the function of detecting user operations in areas other than the display panel 202. In other words, the operation panel 203 can also have a detection area (hereinafter referred to as the display area) for the overlapping part that coincides with the display panel 202 and a detection area (hereinafter referred to as the non-display area) for the outer edge part that does not coincide with the display panel 202.

[0170] Alternatively, the size of the display area can be exactly the same as the size of the display panel 202, but it is not necessary for them to be identical. Furthermore, the operation panel 203 can have two sensing areas: an outer edge and an inner side. The width of the outer edge is appropriately designed based on the size of the frame 201, etc.

[0171] Furthermore, the position detection method used in the operation panel 203 can include matrix switch method, resistive film method, surface acoustic wave method, infrared method, electromagnetic induction method, electrostatic capacitance method, etc., and any one of these methods can be used.

[0172] The call unit 211 includes a speaker 205 or a microphone 206, which converts the user's voice input through the microphone 206 into voice data that can be processed by the main control unit 220 and outputs it to the main control unit 220, or decodes the voice data received by the wireless communication unit 210 or the external input / output unit 213 and outputs it from the speaker 205.

[0173] And, as Figure 26 As shown, for example, the speaker 205 can be mounted on the same surface as the surface where the display input section 204 is provided, and the microphone 206 can be mounted on the side of the frame 201.

[0174] The operation unit 207 uses a hardware key such as a key switch, and receives instructions from the user. For example, such as Figure 26 As shown, the operation unit 207 is mounted on the side of the frame 201 of the smartphone 200, and is a button-type switch that is turned on when pressed by a finger or the like, and turned off by the restoring force of a spring or the like when the finger is released.

[0175] Storage unit 212 stores the control program and control data of main control unit 220, application software, address data that establishes correspondences between the names or phone numbers of communication partners, data of sent and received emails, web data downloaded through web browsing, downloaded content data, and temporarily stores streaming data, etc. Furthermore, storage unit 212 consists of an internal storage unit 217 built into the smartphone and an external storage unit 218 with a removable external memory slot.

[0176] In addition, the internal storage units 217 and external storage units 218 constituting the storage unit 212 are implemented using storage media such as flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., MicroSD memory), RAM (Random Access Memory), ROM (Read Only Memory).

[0177] The external input / output unit 213 serves as an interface to all external devices connected to the smartphone 200, and is used to connect directly or indirectly to other external devices via communication (e.g., Universal Serial Bus (USB), IEEE 1394, Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, etc.) or networks (e.g., Ethernet, Wireless LAN, etc.).

[0178] External devices connected to the smartphone 200 include, for example, wired / wireless headphones, wired / wireless external chargers, wired / wireless data ports, memory cards connected via card slots, SIM (Subscriber Identity Module Card) / UIM (User Identity Module Card) cards, external audio / video devices connected via audio / video I / O (Input / Output) terminals, wirelessly connected external audio / video devices, wired / wireless connected smartphones, wired / wireless connected personal computers, and headphones.

[0179] The external input / output unit 213 can transmit data received from such an external device to the internal components of the smartphone 200, or transmit data from the internal components of the smartphone 200 to an external device.

[0180] The GNSS receiver 214 receives GNSS signals transmitted from GNSS satellites ST1 to STn according to the instructions of the main control unit 220, performs positioning calculations based on the received multiple GNSS signals, and detects the location, including the latitude, longitude, and altitude of the smartphone 200. When the GNSS receiver 214 can obtain location information from the wireless communication unit 210 or the external input / output unit 213 (e.g., a wireless LAN), it can also use that location information to detect its location.

[0181] The motion sensor unit 215, for example, includes a 3-axis accelerometer, and detects the physical motion of the smartphone 200 according to the instructions of the main control unit 220. By detecting the physical motion of the smartphone 200, the direction of movement or acceleration of the smartphone 200 is detected. The detection result is output to the main control unit 220.

[0182] The power supply unit 216 supplies power stored in the battery (not shown) to each part of the smartphone 200 according to the instructions of the main control unit 220.

[0183] The main control unit 220 is equipped with a microprocessor, which operates according to the control program and control data stored in the storage unit 212, and centrally controls all parts of the smartphone 200. The microprocessor of the main control unit 220 has the same functions as the system control unit 11. Furthermore, the main control unit 220 has mobile communication control functions and application processing functions for controlling all parts of the communication system in order to conduct voice or data communication via the wireless communication unit 210.

[0184] The application processing functions are implemented by the main control unit 220 according to the application software stored in the storage unit 212. Examples of application processing functions include infrared communication functions that control the external input / output unit 213 and communicate with the counterpart device, email functions that send and receive emails, and web browsing functions that view web pages.

[0185] Furthermore, the main control unit 220 has image processing functions such as displaying images on the display input unit 204 based on image data (data of still images or moving images) such as received data or downloaded streaming data.

[0186] The image processing function refers to the function of the main control unit 220 decoding the image data, performing image processing on the decoding result, and displaying the image on the display input unit 204.

[0187] In addition, the main control unit 220 performs display control on the display panel 202 and operation detection control for user operations via the operation unit 207 and the operation panel 203.

[0188] By executing display control, the main control unit 220 displays software keys such as icons or scroll bars for launching application software, or displays a window for composing emails.

[0189] Additionally, the scroll bar refers to a software key used to indicate the display portion of a large image that cannot be fully contained within the display area of ​​the display panel 202.

[0190] Furthermore, by performing operation detection control, the main control unit 220 detects user operations through the operation unit 207, or receives operations on the aforementioned icons and input of strings into the input fields of the aforementioned windows through the operation panel 203, or receives scrolling requests for the displayed images through the scroll bar.

[0191] In addition, by performing operation detection control, the main control unit 220 has the following touch panel control function: determining whether the operation position of the operation panel 203 is an overlapping part (display area) that coincides with the display panel 202, or an outer edge part (non-display area) that does not coincide with the display panel 202, and controlling the display position of the sensing area or software key of the operation panel 203.

[0192] Furthermore, the main control unit 220 can also detect gesture operations on the operation panel 203 and execute preset functions according to the detected gesture operations.

[0193] Gesture operation is not the simple touch operation of the past, but refers to the operation of drawing a trajectory using fingers or other means, or specifying multiple positions at the same time, or combining them to draw a trajectory for at least one of the multiple positions.

[0194] Camera Department 208 includes Figure 1 The lens assembly 40, the camera element 5, and the digital signal processing unit 17 are shown.

[0195] The imaging image data generated by the camera unit 208 can be stored in the storage unit 212, or output through the external input / output unit 213 or the wireless communication unit 210.

[0196] exist Figure 27 In the smartphone 200 shown, the camera unit 208 is mounted on the same side as the display input unit 204, but the mounting position of the camera unit 208 is not limited to this, and it can also be mounted on the back of the display input unit 204.

[0197] Furthermore, the camera unit 208 can be used for various functions of the smartphone 200. For example, images captured by the camera unit 208 can be displayed on the display panel 202, or images from the camera unit 208 can be used as one of the operation inputs for the operation panel 203.

[0198] Furthermore, when the GNSS receiver 214 detects its position, it can also refer to the image from the camera unit 208 to detect the position. Moreover, it can refer to the image from the camera unit 208 to determine the optical axis direction of the smartphone 200's camera unit 208, or to determine the current operating environment, without using a 3-axis accelerometer or in conjunction with a 3-axis accelerometer. Of course, the image from the camera unit 208 can also be used within the application software.

[0199] In addition, image data such as location information acquired by the GNSS receiver 214, voice information acquired by the microphone 206 (which can be converted into text information by the main control unit, etc.), and attitude information acquired by the motion sensor 215, which are attached to still images or moving images, can be stored in the storage unit 212, or output through the external input / output unit 213 or the wireless communication unit 210.

[0200] The various embodiments have been described above, but the present invention is not limited to this example. Obviously, those skilled in the art will be able to conceive of various modifications or alterations within the scope described in the technical solution, and it should be understood that these modifications or alterations naturally fall within the technical scope of the present invention. Furthermore, the constituent elements of the above embodiments can be combined arbitrarily without departing from the spirit of the invention.

[0201] Furthermore, this application is based on Japanese Patent Application No. 2024-046434, filed on March 22, 2024, the contents of which are incorporated herein by reference.

[0202] Symbol Explanation 1-Imaging lens, 2-Aperture, 4-Lens control unit, 5-Image sensor, 8-Lens drive unit, 9-Aperture drive unit, 11-System control unit, 14, 207-Operation unit, 15-Memory control unit, 16-Memory, 17-Digital signal processing unit, 20-External memory control unit, 21-Storage medium, 22-Display device, 22a-Display controller, 22b-Display surface, 24-Control bus, 25-Data bus, 40-Lens assembly, 60-Image sensor, 61, 61fb, 61fg, 61fr-Pixel, 61, 61B, 61G, 61R-First pixel, 61FB, 61FG, 61FR, 61FYd, 61FXl, 61FXr, 61FYu-First type pixel, 61F-Second pixel, 61FL-Second type pixel, 61FP-Polarized pixel, 61Fb, 61Fg, 61Fr - Third type of pixel, 62 - Pixel row, 63 - Driving circuit, 64 - Signal processing circuit, 64A, 64C - Second conversion circuit, 64B, 64D - First conversion circuit, 100 - Digital camera, 100A - Main body, 200 - Smartphone, 201 - Frame, 202 - Display panel, 203 - Operation panel, 204 - Display input unit, 205 - Speaker, 206 - Microphone, 208 - Camera unit, 210 - Wireless communication unit, 211 - Talking unit, 212 - Storage unit, 213 - External input / output unit, 214 - GNSS receiver unit, 215 - Motion sensor unit, 216 - Power supply unit, 217 - Internal storage unit, 218 - External storage unit, 220 - Main control unit, P1, P1a, P1b, PX - First pixel array, P2, PY - Second pixel array.

Claims

1. A camera element, which is a camera element formed by repeating a first pixel array in a first direction and a second direction, wherein, Set M to a natural number greater than or equal to 3, and set N to a natural number greater than or equal to 3. The first pixel array is composed of M pixels arranged in the first direction and N pixels arranged in the second direction. The first pixel array includes a first pixel and a second pixel. The first pixel is contained in a second pixel array consisting of M-1 pixels arranged in the first direction and N-1 pixels arranged in the second direction. The second pixel is any pixel other than the first pixel. The first pixel and the second pixel have different functions.

2. The imaging element according to claim 1, wherein, The second pixel is continuously arranged in at least one of the first direction and the second direction.

3. The imaging element according to claim 2, wherein, The second pixel includes a first type of pixel whose pixel structure is different from that of the first pixel.

4. The imaging element according to claim 3, wherein, The first type of pixel includes pixels used for phase difference detection.

5. The imaging element according to claim 4, wherein, The phase difference detection pixel is positioned outside the position of the pixel in the second pixel array arranged in the second direction.

6. The imaging element according to claim 4, wherein, The phase difference detection pixel is positioned outside the position of the pixel in the second pixel array arranged in the first direction.

7. The imaging element according to claim 4, wherein, The shape of the microlens of the pixel used for phase difference detection is different from that of the first pixel.

8. The imaging element according to claim 7, wherein, The phase difference detection pixels share a single microlens among multiple pixels.

9. The imaging element according to claim 3, wherein, The first type of pixel includes polarization pixels used for polarization image acquisition.

10. The imaging element according to claim 3, wherein, The first type of pixel includes pixels with an opening area different from that of the first pixel.

11. The imaging element according to claim 10, wherein, The pixel whose opening area is different from that of the first pixel has an opening area smaller than that of the first pixel.

12. The imaging element according to claim 2, wherein, The second pixel includes a second type of pixel whose spectral characteristics are different from those of the first pixel.

13. The imaging element according to claim 12, wherein, The second type of pixel includes pixels with or without a brightness filter.

14. The imaging element according to claim 2, wherein, The second pixel includes a third pixel whose sensitivity characteristics differ from those of the first pixel.

15. The imaging element according to claim 14, wherein, The first pixel contains multiple pixels corresponding to multiple colors. The third type of pixel includes multiple pixels corresponding to the multiple colors.

16. The imaging element according to claim 15, wherein, The sensitivity of the third type of pixel is lower than that of the first type of pixel.

17. The imaging element according to claim 1, wherein, The first pixel contains multiple pixels corresponding to multiple colors. The second pixel includes multiple pixels corresponding to the multiple colors. The imaging element includes: a first conversion circuit that converts an analog signal output from the first pixel into a digital signal; and The second conversion circuit converts the analog signal output from the multiple pixels of the second pixel into a digital signal. The first conversion circuit and the second conversion circuit have different noise characteristics.

18. The imaging element according to claim 1, comprising: The first conversion circuit converts the pixel signal output from the first pixel into a digital signal; and The second conversion circuit converts the pixel signal output from the second pixel into a digital signal. The first conversion circuit and the second conversion circuit have different signal conversion speeds.

19. The imaging element according to claim 3, wherein, The first type of pixel includes at least one of a distance measurement pixel and a brightness change detection pixel.

20. The imaging element according to claim 1, wherein, The arrangement pattern of the first pixel in the second pixel array becomes a pattern capable of capturing a color image.

21. A camera device comprising a camera element and a processor as described in any one of claims 1 to 20.

22. The camera device according to claim 21, wherein, The processor performs the following control: changing the exposure time in at least a portion of the second pixel and the first pixel.

23. The camera device according to claim 22, wherein, The arrangement pattern of at least a portion of the second pixels becomes a pattern capable of capturing a color image.

24. The camera device according to claim 23, wherein, The arrangement pattern of the first pixel in the second pixel array becomes a pattern capable of capturing a color image.

Images