Imaging device and endoscope system

Abstract

The present invention relates to an imaging device (10) comprising: a polarization image sensor (20) in which 2 * 2 pixels for detecting four polarization components are two-dimensionally arranged; an interpolation processing unit (40) that calculates RGB values for each of the four polarization components for each pixel from the pixel values output by the polarization image sensor (20); a sorting processing unit (44) that sorts the polarization components; a reference generation unit (46) that calculates a reference brightness value obtained by combining the brightness values of the plurality of polarization components; and a synthesis processing unit (48) that: a) outputs a reference RGB value obtained by synthesizing the RGB values of the plurality of polarization components when the luminance value coincides with a predetermined threshold value; b) when the reference luminance value is less than a threshold value, outputting a synthesized RGB value obtained by synthesizing an upper RGB value obtained by synthesizing RGB values of the first and second bits of polarization components and the reference RGB value; and c) when the reference luminance value is less than the threshold value, outputting a synthesized RGB value obtained by synthesizing a lower RGB value obtained by synthesizing the RGB values of the third and fourth bits of polarization components and the reference RGB value.

Description

Technical Field

[0001] The present invention relates to a photographing device and an endoscope system. Background Art

[0002] Cameras for observing the polarization state of light from a subject are known. For example, a technique has been proposed that acquires multiple color images with different polarization states in a time-division manner, generates a composite image using polarization information about the subject based on the multiple color images, and performs color correction based on a reference image selected from the multiple color images. Furthermore, a technique has been proposed that suppresses whitening and blackening by using an HDR (High Dynamic Range) image, a composite of multiple color images with different exposure conditions, as a reference image (see, for example, Patent Document 1).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2017-191986. Summary of the Invention

[0006] In the above-mentioned technology, since a variable phase difference plate is used to obtain a color image having different polarization states, coloration caused by the variable phase difference plate may occur.

[0007] The present invention has been made in view of the above-mentioned circumstances, and an object of the present invention is to provide a technology for generating a clear image using a polarization image sensor.

[0008] A photographing device according to one embodiment of the present invention comprises: a polarization image sensor having pixel groups arranged two-dimensionally, and an RGB color filter arranged in a Bayer arrangement according to each pixel group, wherein the pixel group includes 2×2 pixels for detecting four polarization components that are different for each pixel; an interpolation processing unit, which separates the pixel values ​​output from the polarization image sensor according to each polarization component to generate four Bayer array images corresponding to the four polarization components, and calculates the RGB values ​​of the four polarization components for each pixel by performing debayering and up-conversion on the four Bayer array images; a brightness calculation unit, which calculates the brightness values ​​of the four polarization components for each pixel based on the RGB values ​​of the four polarization components; and a sorting processing unit, which sorts the four polarization components for each pixel in the order of the brightness values ​​of the four polarization components. Sequence; a reference generating unit, which calculates, for each pixel, a reference brightness value obtained by synthesizing the brightness values ​​of a plurality of polarization components including at least the second and third digits; and a synthesizing processing unit, which a) when the brightness value is consistent with a predetermined threshold value, outputs a reference RGB value obtained by synthesizing the RGB values ​​of the plurality of polarization components; b) when the reference brightness value is less than the threshold value, outputs a synthesized RGB value obtained by synthesizing an upper RGB value and the reference RGB value, the upper RGB value being obtained by synthesizing the RGB values ​​of the first and second digits of the polarization components; and c) when the reference brightness value is less than the threshold value, outputs a synthesized RGB value obtained by synthesizing a lower RGB value and the reference RGB value, the lower RGB value being obtained by synthesizing the RGB values ​​of the third and fourth digits of the polarization components.

[0009] Another embodiment of the present invention is an endoscope system including the imaging device of the aforementioned embodiment. The endoscope system comprises: an endoscope including an insertion portion having a distal end facing a subject, a polarization image sensor disposed within the distal end, and a transmission cable disposed within the insertion portion for transmitting an output signal of the polarization image sensor; and an image processing device including an interpolation processing unit, a brightness calculation unit, a sorting processing unit, a reference generation unit, and a synthesis processing unit, and acquiring the output signal via the transmission cable.

[0010] Furthermore, arbitrary combinations of the above-described constituent elements, and configurations and descriptions of the present invention in which methods, apparatuses, systems, and the like are interchanged with each other may also be practiced as additional modes of the present invention.

[0011] According to the present invention, a clear image can be generated using a polarization image sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a diagram schematically showing the configuration of the imaging device according to the first embodiment.

[0013] Figure 2is a diagram schematically showing the structure of a light detection layer of a polarization image sensor.

[0014] Figure 3 FIG. 1 is a plan view schematically showing the structure of a polarizing plate layer of a polarization image sensor.

[0015] Figure 4 is a plan view schematically showing the structure of a color filter layer of a polarization image sensor.

[0016] Figure 5 is a plan view schematically showing the structure of a microlens layer of a polarization image sensor.

[0017] Figure 6 FIG. 1 is a diagram schematically showing the flow of image processing by the interpolation processing unit.

[0018] Figure 7 It is a diagram schematically showing the flow of image processing by the synthesis processing unit.

[0019] Figure 8 It is a diagram schematically showing the configuration of an imaging device according to the second embodiment.

[0020] Figure 9 is a plan view schematically showing the structure of a color filter layer of a non-polarization image sensor.

[0021] Figure 10 This is a graph showing an example of the frequency characteristics of the low-pass filter used in the high-frequency extraction unit.

[0022] Figure 11 FIG. 1 is a diagram schematically showing the flow of image processing performed by the second processing unit.

[0023] Figure 12 It is a diagram schematically showing the configuration of an imaging device according to a third embodiment.

[0024] Figure 13 FIG. 1 is a diagram schematically showing the flow of image processing performed by the second processing unit.

[0025] Figure 14 It is a diagram schematically showing the configuration of an imaging device according to a fourth embodiment.

[0026] Figure 15 It is a diagram schematically showing the configuration of an imaging device according to the fifth embodiment.

[0027] Figure 16 It is a diagram schematically showing an imaging unit of a first configuration example according to the fifth embodiment.

[0028] Figure 17It is a diagram schematically showing an imaging unit of a second configuration example according to the fifth embodiment.

[0029] Figure 18 It is a diagram schematically showing an imaging unit according to a third configuration example of the fifth embodiment.

[0030] Figure 19 It is a diagram schematically showing the configuration of an endoscope system according to the sixth embodiment.

[0031] Figure 20 It is a diagram schematically showing the configuration of an endoscope system according to the seventh embodiment. DETAILED DESCRIPTION

[0032] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Specific numerical values ​​and the like shown in these embodiments are merely illustrative examples to facilitate understanding of the invention and, except where otherwise noted, do not limit the present invention. Furthermore, elements not directly related to the present invention are omitted from the drawings.

[0033] (First embodiment)

[0034] Figure 1 1 is a diagram schematically showing the configuration of an imaging device 10 according to Embodiment 1. The imaging device 10 includes an imaging unit 12 and an image processing device 14.

[0035] The photographing unit 12 includes a photographing lens 18 and a polarization image sensor 20 .

[0036] The photographing lens 18 is provided in front of the polarization image sensor 20. The photographing lens 18 is configured to form an image of the incident light 16 incident on the photographing unit 12 on the light receiving surface of the polarization image sensor 20. The photographing lens 18 can include any number of optical lenses.

[0037] The polarization image sensor 20 includes a plurality of pixels for capturing the incident light 16. The polarization image sensor 20 includes a light detection layer 22, a polarizer layer 24, a color filter layer 26, and a microlens layer 28. The light detection layer 22, the polarizer layer 24, the color filter layer 26, and the microlens layer 28 are arranged so as to overlap in the incident direction of the incident light 16. Figure 1 In the example, when viewed from the incident direction of the incident light 16, the microlens layer 28, the color filter layer 26, the polarizer layer 24, and the light detection layer 22 are arranged in this order. It should be noted that there is no limitation on the stacking order of the polarizer layer 24 and the color filter layer 26. For example, the microlens layer 28, the polarizer layer 24, the color filter layer 26, and the light detection layer 22 can be stacked in this order.

[0038] Figure 21 is a top view schematically showing the structure of the light detection layer 22 of the polarization image sensor 20. The light detection layer 22 is configured similarly to a two-dimensional image sensor such as a CCD (Charge Coupled Devices) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The light detection layer 22 includes a photodiode 22a, which is used to detect incident light 16 and convert it into an electrical signal. The light detection layer 22 includes a plurality of photodiodes 22a arranged in a two-dimensional manner. The light detection layer 22 includes, for example, one photodiode 22a for each pixel 30 of the polarization image sensor 20.

[0039] Figure 3 This is a top view schematically illustrating the structure of the polarizing plate layer 24 of the polarization image sensor 20. The polarizing plate layer 24 includes a first polarizer 24a, a second polarizer 24b, a third polarizer 24c, and a fourth polarizer 24d for detecting different polarization components for each pixel 30. That is, one of the four polarizers 24a to 24d is provided in each pixel 30. The first polarizer 24a selectively transmits a first polarization component, which is linearly polarized light in a first direction (e.g., horizontal or 0 degrees). The second polarizer 24b selectively transmits a second polarization component, which is linearly polarized light in a second direction (e.g., diagonally right or 45 degrees). The third polarizer 24c selectively transmits a third polarization component, which is linearly polarized light in a third direction (e.g., vertical or 90 degrees). The fourth polarizer 24d selectively transmits a fourth polarization component, which is linearly polarized light in a fourth direction (e.g., diagonally left or 135 degrees). The polarizers 24a to 24d are, for example, wire-grid polarizers.

[0040] The polarizer layer 24 has a structure in which pixel groups 32, each consisting of four pixels arranged in a 2×2 vertical and horizontal pattern, are arranged two-dimensionally as a repeating unit. Each pixel group 32 includes a first pixel equipped with a first polarizer 24a, a second pixel equipped with a second polarizer 24b, a third pixel equipped with a third polarizer 24c, and a fourth pixel equipped with a fourth polarizer 24d. The first and third polarizers 24a and 24c are positioned in diagonally opposite pixels within a pixel group 32. The second and fourth polarizers 24b and 24d are positioned in diagonally opposite pixels within a pixel group 32. The four polarizers 24a to 24d are arranged two-dimensionally, vertically and horizontally, for every other pixel.

[0041] Figure 4: This is a plan view schematically showing the structure of the color filter layer 26 of the polarization image sensor 20. The color filter layer 26 includes a red (R) filter 26a, a green (Gr) filter 26b, a blue (B) filter 26c, and a green (Gb) filter 26d arranged in a Bayer arrangement for each pixel group 32. That is, any one of the four color filters 26a to 26d is provided in one pixel group 32. The four color filters 26a to 26d are arranged so as to occupy four pixels of 2×2 in the vertical and horizontal directions, respectively. The color filter layer 26 has a structure in which a pixel group 34 having four pixel groups 32 adjacent to each other in the vertical and horizontal directions is arranged two-dimensionally as a repeating unit. The pixel group 34 has 16 pixels of 4×4 in the vertical and horizontal directions.

[0042] Figure 5 2 is a plan view schematically showing the structure of the microlens layer 28 of the polarization image sensor 20. The microlens layer 28 includes a plurality of microlenses 28a arranged two-dimensionally. For example, the microlens layer 28 includes one microlens 28a for each pixel 30 of the polarization image sensor 20.

[0043] return Figure 1 Next, the image processing device 14 will be described. The image processing device 14 generates an image using the output signal of the polarization image sensor 20. The image processing device 14 includes a signal acquisition unit 38, an interpolation processing unit 40, a brightness calculation unit 42, a sorting processing unit 44, a reference generation unit 46, and a synthesis processing unit 48.

[0044] The image processing device 14 can be composed of, for example, an electronic circuit such as a DSP (Digital Signal Processor) or an ISP (Image Signal Processor) for performing hardware-based signal processing or image processing. Each functional block constituting the image processing device 14 can be composed of one or more electronic circuits. The image processing device 14 can also be implemented by a combination of hardware and software. The hardware of the image processing device 14 can also be implemented by components such as processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), memories such as ROM (Read Only Memory) and RAM (Random Access Memory), or mechanical devices. The software of the image processing device 14 can also be implemented by a computer program, etc. In this case, the image processing device 14 is depicted as functional blocks implemented by the collaboration of hardware and software. Those skilled in the art should understand that the functional blocks of the image processing device 14 can be implemented in various ways by a combination of hardware and software.

[0045] The signal acquisition unit 38 acquires the image signal 36 output from the polarization image sensor 20. The image signal 36 corresponds to the raw data output from the polarization image sensor 20 and is, for example, serial data of the pixel values ​​of the pixels 30 read in the order of their addresses. The number of bits of the pixel values ​​in the image signal 36 is not particularly limited; for example, it is 1 second.

[0046] The interpolation processing unit 40 generates color images for each of the four polarization components based on the image signal 36 acquired by the signal acquisition unit 38. The interpolation processing unit 40 separates the pixel values ​​included in the image signal 36 for each polarization component, generating four Bayer array images corresponding to the four polarization components. The interpolation processing unit 40 performs debayering (Bayer transformation) and upconversion (horizontal and vertical interpolation) on each of the four Bayer array images to generate four upconverted images in which the RGB values ​​of the four polarization components are set for each pixel.

[0047] Figure 6 Schematically shows the flow of image processing by the interpolation processing unit 40. Figure 6 In the figure, for simplicity of explanation, only the 16 pixels of 4×4 in both vertical and horizontal dimensions that make up one pixel group 34 are cut out and shown. The input image 50 corresponds to RAW data based on the image signal 36, and each pixel 30 stores only one pixel value. One pixel group 34 includes 16 pixels corresponding to a combination of four colors (R, Gr, Gb, B) and four polarization components (1, 2, 3, 4). For example, pixel value R1 represents the pixel value of a pixel of the first polarization component (1) of red (R) that has passed through the R filter 26a and the first polarizer 24a.

[0048] The interpolation processing unit 40 separates the input image 50 into four Bayer array images 52a, 52b, 52c, and 52d for each polarization component. The first Bayer array image 52a is composed of pixel values ​​R1, Gr1, Gb1, and B1 for the first polarization component of the four colors (R, Gr, Gb, and B). The positions (phases) of the pixel values ​​R1, Gr1, Gb1, and B1 are the same as those of the input image 50. The second Bayer array image 52b is composed of pixel values ​​R2, Gr2, Gb2, and B2 for the second polarization component of the four colors (R, Gr, Gb, and B). The positions (phases) of the pixel values ​​R2, Gr2, Gb2, and B2 are the same as those of the input image 50. The third Bayer array image 52c is composed of pixel values ​​R3, Gr3, Gb3, and B3 for the third polarization component of the four colors (R, Gr, Gb, and B). The positions (phases) of the pixel values ​​R3, Gr3, Gb3, and B3 are the same as those in the input image 50. The fourth Bayer array image 52d includes pixel values ​​R4, Gr4, Gb4, and B4 of the fourth polarization component of the four colors (R, Gr, Gb, and B). The positions (phases) of the pixel values ​​R4, Gr4, Gb4, and B4 are the same as those in the input image 50.

[0049] The interpolation processing unit 40 debayers each of the four Bayer array images 52a to 52d to generate debayer images 54a, 54b, 54c, and 54d, each storing a single RGB value for each 2×2 pixel (i.e., pixel group 32). The first debayer image 54a is generated by debayering the first Bayer array image 52a. Any known technique can be used for the debayering process. The first debayer image 54a is composed of RGB values ​​corresponding to the position (phase) of the pixel in the pixel group 32 where the first polarizer 24a is provided, namely, RGB11, RGB12, RGB13, and RGB14. Here, RGB values ​​are array data containing pixel values ​​(R, G, B) for each of R, G, and B values.

[0050] Unless explicitly mentioned, the term "RGB" in the following mathematical formulas generally refers to calculations based on an array (R, G, B) of R, G, and B values, and refers to mathematical formulas that calculate the R, G, and B values ​​separately. Specifically, the term "RGB" is used to simplify the description of the calculations of the R, G, and B values.

[0051] A second debayer image 54b is generated by debayering the second Bayer array image 52b. The second debayer image 54b is composed of RGB values ​​corresponding to the positions (phases) of the pixels in the pixel group 32 at which the second polarizer 24b is provided, namely, RGB21, RGB22, RGB23, and RGB24. A third debayer image 54c is generated by debayering the third Bayer array image 52c. The third debayer image 54c is composed of RGB values ​​corresponding to the positions (phases) of the pixels in the pixel group 32 at which the third polarizer 24c is provided, namely, RGB31, RGB32, RGB33, and RGB34. A fourth debayer image 54d is generated by debayering the fourth Bayer array image 52d. The fourth debayer image 54d is composed of RGB values ​​corresponding to the positions (phases) of the pixels in the pixel group 32 at which the fourth polarizer 24d is provided, namely, RGB41, RGB42, RGB43, and RGB44.

[0052] The interpolation processing unit 40 generates four up-converted images 56a, 56b, 56c, and 56d by up-converting the four debayer images 54a to 54d. The interpolation processing unit 40 generates four up-converted images 56a to 56d by interpolating the pixel values ​​of the four debayer images 54a to 54d in the horizontal and vertical directions. Any known technology can be used for the up-conversion process. Figure 6 In the example, up-conversion is performed by a factor of 2 in both the vertical and horizontal directions. However, the up-conversion ratio is not particularly limited and can be set to any ratio.

[0053] The first up-converted image 56a is generated based on the first debayer image 54a. The first up-converted image 56a also includes RGB values ​​corresponding to the positions (phases) of pixels different from the pixels at which the first polarizer 24a is provided. The first up-converted image 56a also includes, for example, RGB values ​​corresponding to the positions (phases) of pixels at which the second polarizer 24b is provided, namely, RGB11b, RGB12b, RGB13b, RGB14b. The first up-converted image 56a also includes, for example, RGB values ​​corresponding to the positions (phases) of pixels at which the third polarizer 24c is provided, namely, RGB11c, RGB12c, RGB13c, RGB14c. The first up-converted image 56a also includes, for example, RGB values ​​corresponding to the positions (phases) of pixels at which the fourth polarizer 24d is provided, namely, RGB11d, RGB12d, RGB13d, RGB14d. Therefore, Figure 6 The first up-converted image 56 a in φ includes RGB values ​​of the first polarization component set for the same number of pixels as the input image 50 .

[0054] The second up-converted image 56b is generated based on the second debayer image 54b. The second up-converted image 56b also includes RGB values ​​corresponding to the position (phase) of pixels different from the pixels provided with the second polarizer 24b. The second up-converted image 56b also includes, for example, RGB values ​​corresponding to the position (phase) of pixels provided with the first polarizer 24a, namely, RGB21a, RGB22a, RGB23a, RGB24a. The second up-converted image 56b also includes, for example, RGB values ​​corresponding to the position (phase) of pixels provided with the third polarizer 24c, namely, RGB21c, RGB22c, RGB23c, RGB24c. The second up-converted image 56b also includes, for example, RGB values ​​corresponding to the position (phase) of pixels provided with the fourth polarizer 24d, namely, RGB21d, RGB22d, RGB23d, RGB24d. Therefore, Figure 6 The second up-converted image 56 b in φ includes RGB values ​​of the second polarization component set for the same number of pixels as the input image 50 .

[0055] The third up-converted image 56c is generated based on the third debayer image 54c. The third up-converted image 56c also includes RGB values ​​corresponding to the position (phase) of pixels different from the pixels provided with the third polarizer 24c. The third up-converted image 56c also includes, for example, RGB values ​​corresponding to the position (phase) of pixels provided with the first polarizer 24a, namely, RGB31a, RGB32a, RGB33a, RGB34a. The third up-converted image 56c also includes, for example, RGB values ​​corresponding to the position (phase) of pixels provided with the second polarizer 24b, namely, RGB31b, RGB32b, RGB33b, RGB34b. The third up-converted image 56c also includes, for example, RGB values ​​corresponding to the position (phase) of pixels provided with the fourth polarizer 24d, namely, RGB31d, RGB32d, RGB33d, RGB34d. Therefore, Figure 6 The third up-converted image 56 c in φ includes RGB values ​​of the third polarization component set for the same number of pixels as the input image 50 .

[0056] The fourth up-converted image 56d is generated based on the fourth debayer image 54d. The fourth up-converted image 56d also includes RGB values ​​corresponding to the position (phase) of pixels different from the pixels provided with the fourth polarizer 24d. The fourth up-converted image 56d also includes, for example, RGB values ​​corresponding to the position (phase) of pixels provided with the first polarizer 24a, namely, RGB41a, RGB42a, RGB43a, and RGB44a. The fourth up-converted image 56d also includes, for example, RGB values ​​corresponding to the position (phase) of pixels provided with the second polarizer 24b, namely, RGB41b, RGB42b, RGB43b, and RGB44b. The fourth up-converted image 56d also includes, for example, RGB values ​​corresponding to the position (phase) of pixels provided with the third polarizer 24c, namely, RGB41c, RGB42c, RGB43c, and RGB44c. Therefore, Figure 6 The fourth up-converted image 56 d in φ includes RGB values ​​of the fourth polarization component set for the same number of pixels as the input image 50 .

[0057] The interpolation processing section 40 generates four up-converted images 56a to 56d corresponding to the four polarization components in this manner, thereby calculating the RGB values ​​of the four polarization components for each pixel. Figure 6 The RGB values ​​of the four polarization components in the pixel corresponding to the pixel value R1 of the input image 50 are RGB11, RGB21a, RGB31a, and RGB41a.

[0058] Hereinafter, the RGB values ​​of the four polarization components of each pixel output from the interpolation processing unit 40 are expressed as RGBa, RGBb, RGBc, and RGBd. RGBa is the RGB value of the first polarization component, RGBb is the RGB value of the second polarization component, RGBc is the RGB value of the third polarization component, and RGBd is the RGB value of the fourth polarization component. Figure 6 The pixel corresponding to the pixel value R1 of the input image 50 is set as the focus pixel, then RGBa=RGB11, RGBb=RGB21a, RGBc=RGB31a, and RGBd=RGB41a.

[0059] return Figure 1 The brightness calculation unit 42 calculates the brightness value (Y value) of each of the four polarization components for each pixel based on the RGB values ​​(RGBa to RGBd) of each of the four polarization components. The brightness calculation unit 42 can calculate the Y value from the RGB values ​​using, for example, the following equation (1) defined in accordance with the international standard (ITU-R BT.709) for video signals in HDTV broadcasting.

[0060] Y=0.2126R+0.7152G+0.0722B…(1)

[0061] The luminance calculation section 42 calculates, for each pixel, a luminance value Ya of the first polarization component, a luminance value Yb of the second polarization component, a luminance value Yc of the third polarization component, and a luminance value Yd of the fourth polarization component.

[0062] The sorting unit 44 sorts the four polarization components for each pixel in the order of their brightness values ​​Ya to Yd. The sorting unit 44 outputs ranking signals D1 to D4 representing the first to fourth polarization components. For example, the sorting unit 44 outputs a first signal D1 representing the polarization component with the largest brightness value, a second signal D2 representing the polarization component with the second largest brightness value, a third signal D3 representing the polarization component with the second largest brightness value, and a fourth signal D4 representing the polarization component with the smallest brightness value. The ranking signals D1 to D4 output, for example, one of "0," "1," "2," and "3," as identification values ​​(numbers) for distinguishing the four polarization components. For example, the first polarization component is "0," the second polarization component is "1," the third polarization component is "2," and the fourth polarization component is "3." As an example, if the brightness values ​​in a specific pixel are Yc>Yb>Yd>Ya, the first signal D1 = 2, the second signal D2 = 1, the third signal D3 = 3, and the fourth signal D4 = 0. In addition, the order of the four polarization components may differ for each pixel, and thus the output values ​​of the four order signals D1 to D4 may differ for each pixel.

[0063] The reference generation unit 46 calculates a reference luminance value Ys for each pixel, which is a composite of luminance values ​​of a plurality of polarization components. The reference luminance value Ys can be calculated using the following equation (2).

[0064] Ys=k1·Y1+k2·Y2+k3·Y3+k4·Y4…(2)

[0065] Here, the brightness values ​​Y1 to Y4 are the brightness values ​​of the first to fourth polarization components sorted by the sorting processing unit 44. The coefficients k1 to k4 are weighting coefficients used to synthesize the brightness values ​​Y1 to Y4. The coefficients k1 to k4 are set so that at least the second coefficient k2 and the third coefficient k3 are not zero. In other words, the reference brightness value Ys is calculated by synthesizing the brightness values ​​Y2 and Y3 of at least the second and third polarization components. Thus, the reference brightness value Ys can represent the brightness value in the middle of the brightness values ​​Y1 to Y4 of the four polarization components. For example, the four coefficients k1 to k4 can be set as follows: k1 = 0, k2 = 0.5, k3 = 0.5, k4 = 0. In this case, the reference brightness value Ys is the average of the brightness values ​​Y2 and Y3 of the second and third polarization components. For example, the four coefficients k1 to k4 can be set as follows: k1 = 0, k2 = 0.25, k3 = 0.25, k4 = 0.25. In this case, the reference luminance value Ys is the average value of the four polarization components Y1 to Y4.

[0066] The synthesis processing unit 48 calculates a composite RGB value for each pixel by synthesizing the RGB values ​​of multiple polarization components. The synthesis processing unit 48 changes the method for calculating the composite RGB value according to the magnitude of the reference luminance value Ys. When the reference luminance value Ys matches the predetermined threshold value Yth (i.e., Ys = Yth), the synthesis processing unit 48 sets the reference RGB value (RGBs) as the composite RGB value (RGBm) (i.e., RGBm = RGBs). The reference RGB value (RGBs) is a value obtained by weighted averaging the RGB values ​​of the four polarization components using coefficients k1 to k4, and can be calculated using the following equation (3).

[0067] RGBs=k1·RGB1+k2·RGB2+k3·RGB3+k4·RGB4…(3)

[0068] Here, the values ​​of coefficients k1 to k4 are the same as those used when calculating the reference luminance value Ys by the reference generation unit 46. For example, k1 = 0, k2 = 0.5, k3 = 0.5, and k4 = 0. RGB1 to RGB4 are the RGB values ​​of the polarization components ranked first through fourth by the sorting unit 44. The threshold Yth is set to a luminance value that appears intermediate between black (e.g., minimum luminance value Ymin) and white (e.g., maximum luminance value Ymax). For example, it can be set to 18% of the maximum luminance value Ymax. For a luminance value of 1 in the second digit, Ymax = 4095 and Yth = 737.

[0069] If the reference luminance value Ys is less than a predetermined threshold value Yth (i.e., Ys < Yth), the synthesis processing unit 48 calculates a composite RGB value by synthesizing the upper RGB value (RGBt) and the reference RGB value (RGBs). The upper RGB value (RGBt) is the RGB value of the first and second polarization components. If the reference luminance value Ys is less than the threshold value Yth, the composite RGB value can be made larger than the reference RGB value by synthesizing the upper RGB value, which is greater than the reference RGB value. This can suppress the darkening of dark pixels.

[0070] The upper RGB value (RGBt) is a weighted average of the RGB value (RGB1) of the first polarization component and the RGB value (RGB2) of the second polarization component. The synthesis processing unit 48 can calculate the upper RGB value (RGBt) for each pixel using the following equation (4), for example.

[0071] RGBt=(RGB1·RGB1+RGB2·RGB2) / (RGB1+RGB2)…(4)

[0072] In the above-mentioned formula (4), since weighted averaging is performed based on the brightness of RGB1 and RGB2, the weight of the synthesis of the RGB value (RGB1) of the first polarization component becomes larger.

[0073] When Ys<Yth, the synthesis processing unit 48 can calculate the synthesized RGB value (RGBm) for each pixel using the following equation (5).

[0074] RGBm=t·RGBt+(1-t)·RGBs…(5)

[0075] Here, the upper weighting coefficient t can be calculated for each pixel using the following equation (6).

[0076] t=(Yth-Ys) / Yth…(6)

[0077] According to the above formula (6), the upper weighting coefficient t is set to increase as the reference luminance value Ys decreases. For example, when Ys = Yth, t = 0, and the above formula (5) becomes RGBm = RGBs. When Ys = 0, t = 1, and the above formula (5) becomes RGBm = RGBt. Therefore, when Ys < Yth, as the reference luminance value Ys decreases (i.e., becomes darker), the weight of the upper RGB value (RGBt) in the composite RGB value (RGBm) increases, and the weight of the reference RGB value (RGBs) decreases.

[0078] When the reference luminance value Ys is greater than a predetermined threshold value Yth (i.e., Ys>Yth), the synthesis processing unit 48 calculates a composite RGB value by synthesizing the lower RGB value (RGBu) obtained by synthesizing the RGB values ​​of the third and fourth polarization components with the reference RGB value (RGBs). When the reference luminance value Ys is greater than the threshold value Yth, the composite RGB value can be made smaller than the reference RGB value by synthesizing lower RGB values ​​that are smaller than the reference RGB value. This can suppress whitening in bright pixels.

[0079] The lower RGB value (RGBu) is a weighted average of the RGB value of the third polarization component (RGB3) and the RGB value of the fourth polarization component (RGB4). The synthesis processing unit 48 can calculate the upper RGB value (RGBu) for each pixel using the following equation (7), for example.

[0080] RGBu=[RGB4·(RGBmax-RGB4)+RGB3·(RGBmax-RGB3)] / [(RGBmax-RGB4)+(RGBmax-RGB3)]…(7)

[0081] Here, RGBmax is the maximum pixel value of the RGB value. When the pixel value is 1 in the second place, RGBmax = 4095. In the above formula (7), the value obtained by subtracting the RGB value from the maximum pixel value (RGBmax) is used as a weighting coefficient. The value obtained by subtracting the RGB value from the maximum pixel value (RGBmax) represents the size of the darkness of the RGB value. In the above formula (7), the weighted average is performed based on the size of the darkness of RGB4 and RGB3, so the weight of the synthesis of the RGB value (RGB4) of the fourth polarization component becomes larger.

[0082] When Ys>Yth, the synthesis processing unit 48 can calculate the synthesized RGB value (RGBm) for each pixel using the following equation (8).

[0083] RGBm=u·RGBu+(1-u)·RGBs…(8)

[0084] Here, the lower-order weighting coefficient u can be calculated for each pixel using the following equation (9).

[0085] u=(Ys-Yth) / (Ymax-Yth)…(9)

[0086] According to the above formula (9), the lower-order weighting coefficient u is set to increase as the reference luminance value Ys increases. For example, when Ys = Yth, u = 0, and the above formula (8) becomes RGBm = RGBs. When Ys = Ymax, u = 1, and the above formula (8) becomes RGBm = RGBu. Therefore, when Ys > Yth, as the reference luminance value Ys increases (i.e., becomes brighter), the weight of the lower-order RGB value (RGBu) in the composite RGB value (RGBm) increases, and the weight of the reference RGB value (RGBs) decreases.

[0087] Figure 7 Schematically shows the flow of image processing by the synthesis processing unit 48 . Figure 7 The flow of processing from inputting the RGB values ​​(RGBa to RGBd) of the four polarization components output from the interpolation processing unit 40 to outputting a composite RGB value (RGBm) is shown. Figure 7 Also shown are a brightness calculation section 42 , a sorting processing section 44 , and a reference generation section 46 .

[0088] The brightness calculation unit 42 uses the RGB values ​​(RGBa to RGBd) of the four polarization components as input and outputs the brightness values ​​(Ya to Yd) of the four polarization components. The brightness calculation unit 42 calculates the brightness values ​​(Ya to Yd) from the RGB values ​​(RGBa to RGBd) using the above equation (1).

[0089] The ranking processing unit 44 receives the luminance values ​​(Ya to Yd) output from the luminance calculation unit 42 as input, ranks the luminance values, and outputs ranking signals D1 to D4 indicating the rankings of the four polarization components.

[0090] Reference generation unit 46 receives the luminance values ​​(Ya to Yd) output from luminance calculation unit 42 and the ranking signals (D1 to D4) output from sorting unit 44 as input, and outputs reference luminance value Ys. Reference generation unit 46 calculates reference luminance value Ys using equation (2). Reference generation unit 46 can use, for example, the values ​​of coefficients k1 to k4 specified by registers (not shown).

[0091] The synthesis processing unit 48 includes a reference synthesis unit 60 , a higher-order synthesis unit 62 , a lower-order synthesis unit 64 , a coefficient calculation unit 66 , and an output synthesis unit 68 .

[0092] The reference synthesis unit 60 receives as input the RGB values ​​(RGBa to RGBd) of the four polarization components output from the interpolation processing unit 40 and the order signals (D1 to D4) output from the sorting processing unit 44, and calculates the reference RGB value (RGBs). The reference synthesis unit 60 calculates the reference RGB value (RGBs) using the above equation (3). The reference synthesis unit 60 sets the RGB value of the polarization component specified by the value of the first bit signal D1 to RGB1, the RGB value of the polarization component specified by the value of the second bit signal D2 to RGB2, the RGB value of the polarization component specified by the value of the third bit signal D3 to RGB3, and the RGB value of the polarization component specified by the value of the fourth bit signal D4 to RGB4. For example, the reference synthesis unit 60 can use the values ​​of coefficients k1 to k4 specified by a register (not shown) shared with the brightness calculation unit 42.

[0093] The upper-order synthesis unit 62 receives as input the RGB values ​​(RGBa to RGBd) of the four polarization components output from the interpolation processing unit 40 and the first bit signal D1 and the second bit signal D2 output from the sorting processing unit 44, and calculates the upper-order RGB value (RGBt). The upper-order synthesis unit 62 sets the RGB value of the polarization component specified by the value of the first bit signal D1 as RGB1 and the RGB value of the polarization component specified by the value of the second bit signal D2 as RGB2, and calculates the upper-order RGB value (RGBt) using the above formula (4).

[0094] The lower-order synthesis unit 64 receives as input the RGB values ​​(RGBa to RGBd) of the four polarization components output from the interpolation processing unit 40 and the third-bit signal D3 and the fourth-bit signal D4 output from the sorting processing unit 44, and calculates the lower-order RGB value (RGBu). The lower-order synthesis unit 64 sets the RGB value of the polarization component specified by the value of the third-bit signal D3 to RGB3 and the RGB value of the polarization component specified by the value of the fourth-bit signal D4 to RGB4, and calculates the lower-order RGB value (RGBu) using the above equation (7).

[0095] The coefficient calculation unit 66 receives the reference luminance value Ys output from the reference synthesis unit 60 as input and calculates the upper weighting coefficient t and the lower weighting coefficient u. The coefficient calculation unit 66 calculates the upper weighting coefficient t using the above equation (6). The coefficient calculation unit 66 calculates the lower weighting coefficient u using the above equation (9). The coefficient calculation unit 66 can use, for example, the value of Yth specified by a register (not shown).

[0096] The output synthesis unit 68 receives as input the reference luminance value Ys output from the reference generation unit 46, the reference RGB value (RGBs) output from the reference synthesis unit 60, the upper RGB value (RGBt) output from the upper synthesis unit 62, the lower RGB value (RGBu) output from the lower synthesis unit 64, and the upper weighting coefficient t and lower weighting coefficient u output from the coefficient calculation unit 66, and calculates a synthesized RGB value (RGBm). a) When Ys = Yth, the output synthesis unit 68 makes the synthesized RGB value (RGBm) equal to the reference RGB value (RGBs). b) When Ys < Yth, the output synthesis unit 68 synthesizes the upper RGB value (RGBt) and the reference RGB value (RGBs) using the above-mentioned equation (5) to calculate the synthesized RGB value (RGBm). c) When Ys > Yth, the output synthesis unit 68 synthesizes the lower RGB value (RGBu) and the reference RGB value (RGBs) using the above-mentioned equation (8) to calculate the synthesized RGB value (RGBm).

[0097] The output synthesis unit 68 sequentially calculates the synthesized RGB value (RGBm) for each pixel for all pixels to be processed, thereby generating a color image in which the RGB value of each pixel is the synthesized RGB value (RGBm).

[0098] According to the present embodiment, by synthesizing the RGB values ​​of the four polarization components obtained using the polarization image sensor, a color image with suppressed whitening and blackening can be generated. In the case of using a conventional (non-polarized) image sensor that does not use a polarizing plate, a color image with whitening and blackening is sometimes generated depending on the way the light in the subject is reflected. For example, in the case of wanting to shoot a subject across a glass, sometimes whitening occurs due to the strong reflection of light on the glass surface, making it impossible to properly capture the image of the subject on the opposite side of the glass. In addition, in the case of a subject illuminated by strong light, sometimes the image becomes blackened in the area that becomes the shadow of the subject. According to the present embodiment, by using a reference RGB value (RGBs) of the RGB values ​​of multiple polarization components that synthesizes at least the second and third polarization components as the basis of the pixel value, the influence of the highest polarization component (i.e., the first polarization component) that may cause whitening can be suppressed, and the influence of the lowest polarization component (i.e., the fourth polarization component) that may cause blackening can be suppressed.

[0099] According to the present embodiment, in pixels having a relatively large reference luminance value Ys, the composite RGB value (RGBm) is calculated by increasing the contribution of the lower RGB value (RGBu), and the pixel value can be reduced compared to the case where the reference RGB value is used. Thus, the occurrence of whitening can be appropriately suppressed. For example, as the reference luminance value Ys becomes larger, the occurrence of whitening can be more appropriately suppressed by increasing the weight of the synthesis of the lower RGB value (RGBu). In addition, in pixels having a relatively small reference luminance value Ys, the composite RGB value (RGBm) is calculated by increasing the contribution of the upper RGB value (RGBt), and the pixel value can be increased compared to the case where the reference RGB value is used. Thus, the occurrence of blackening can be appropriately suppressed. For example, as the reference luminance value Ys becomes smaller, the occurrence of blackening can be more appropriately suppressed.

[0100] (Second embodiment)

[0101] Figure 8 This figure schematically illustrates the structure of an imaging device 10A according to a second embodiment. The imaging device 10A according to the second embodiment includes an imaging unit 12A and an image processing device 14A. The following description of the second embodiment focuses on the differences from the first embodiment, with similarities omitted as appropriate. In the accompanying drawings, identical components to those in the first embodiment are denoted by the same reference numerals.

[0102] The imaging unit 12A includes an imaging lens 18, a polarization image sensor 20, a non-polarization image sensor 70, and a spectroscopic element 80. The imaging lens 18 and the polarization image sensor 20 have the same configuration as in the first embodiment. The imaging unit 12A differs from the first embodiment in that it also includes the non-polarization image sensor 70 and the spectroscopic element 80.

[0103] The spectroscopic element 80 is disposed after the imaging lens 18. The spectroscopic element 80 splits the incident light 16 that has passed through the imaging lens 18 into a first light 16a and a second light 16b. The first light 16a enters the polarization image sensor 20, and the second light 16b enters the non-polarization image sensor 70. The spectroscopic element 80 is, for example, a non-polarization beam splitter, and splits the incident light 16 into the first light 16a and the second light 16b at a 1:1 intensity ratio. The partially reflective surface of the spectroscopic element 80 is, for example, a half-mirror made of a metal thin film or the like.

[0104] The polarization image sensor 20 captures the first light 16a split by the spectroscopic element 80. The non-polarization image sensor 70 captures the second light 16b split by the spectroscopic element 80. The polarization image sensor 20 and the non-polarization image sensor 70 are arranged coaxially with respect to the optical axis of the incident light 16. The imaging lens 18 is arranged to form images of the incident light 16 on the light-receiving surfaces of the polarization image sensor 20 and the non-polarization image sensor 70.

[0105] The non-polarization image sensor 70 includes a plurality of pixels for capturing incident light 16. The non-polarization image sensor 70 includes a light detection layer 72, a color filter layer 76, and a microlens layer 78. The non-polarization image sensor 70 differs from the polarization image sensor 20 in that it does not include a polarizer layer.

[0106] The number of pixels of the non-polarized image sensor 70 is greater than the number of arranged pixel groups 32 of the polarized image sensor 20. Therefore, the number of pixels in the longitudinal and lateral directions of the non-polarized image sensor 70 is greater than half the number of pixels in the longitudinal and lateral directions of the polarized image sensor 20. The number of pixels in the longitudinal and lateral directions of the non-polarized image sensor 70 may also be the same as the number of pixels in the longitudinal and lateral directions of the polarized image sensor 20. The number of pixels in the longitudinal and lateral directions of the non-polarized image sensor 70 may be greater than the number of pixels in the longitudinal and lateral directions of the polarized image sensor 20, for example, may be twice or four times the number of pixels in the longitudinal and lateral directions of the polarized image sensor 20.

[0107] The light detection layer 72 is configured similarly to a two-dimensional image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The light detection layer 72 can be configured similarly to the light detection layer 22 of the polarization image sensor 20. For example, the light detection layer 72 includes one photodiode for each pixel of the non-polarization image sensor 70. If the number of pixels in the non-polarization image sensor 70 matches the number of pixels in the polarization image sensor 20, the light detection layer 72 can also have the same specifications as the light detection layer 22 of the polarization image sensor 20.

[0108] Figure 9 This is a plan view schematically illustrating the structure of the color filter layer 76 of the non-polarized image sensor 70. The color filter layer 76 includes a red (R) filter 76a, a green (Gr) filter 76b, a blue (B) filter 76c, and a green (Gb) filter 76d arranged in a Bayer pattern for each pixel 82 of the non-polarized image sensor 70. The color filter layer 76 has a structure in which pixel groups 84 having four pixels arranged in a 2×2 pattern vertically and horizontally are arranged two-dimensionally as a repeating unit.

[0109] return Figure 8 The microlens layer 78 includes a plurality of microlenses arranged two-dimensionally. The microlens layer 78 can be configured similarly to the microlens layer 28 of the polarization image sensor 20. For example, the microlens layer 78 includes one microlens for each pixel 82 of the non-polarization image sensor 70.

[0110] The image processing device 14A generates an image using the output signal of the polarization image sensor 20 and the output signal of the non-polarization image sensor 70. The image processing device 14A includes a first processing unit 92, a second processing unit 94, and an addition processing unit 96.

[0111] The first processing unit 92 performs signal processing or image processing using as input the first image signal 36 output from the polarization image sensor 20. The first processing unit 92 includes a first signal acquisition unit 38, a first interpolation unit 40, a brightness calculation unit 42, a sorting unit 44, a reference generation unit 46, and a synthesis unit 48. This unit implements the same functions as the image processing device 14 of the first embodiment.

[0112] The first processing section 92 outputs a composite RGB value (RGBm) obtained by synthesizing the RGB values ​​of the four polarization components to the addition processing section 96. The first processing section 92 may output the RGB values ​​of the four polarization components (RGBa to RGBd) to the addition processing section 96 instead of the composite RGB value (RGBm). The first processing section 92 may also output the composite RGB value (RGBm) and the RGB values ​​of the four polarization components (RGBa to RGBd) to the addition processing section 96.

[0113] The second processing unit 94 performs signal processing or image processing using as input the second image signal 86 output from the non-polarization image sensor 70 . The second processing unit 94 includes a second signal acquisition unit 98 , a second interpolation unit 100 , and a high-frequency extraction unit 102 .

[0114] The second signal acquisition unit 98 acquires the second image signal 86 output from the non-polarization image sensor 70. The second image signal 86 corresponds to the raw data output from the non-polarization image sensor 70 and is, for example, serial data representing the pixel values ​​of the pixels 82 of the non-polarization image sensor 70, read in the order of their addresses. The second image signal 86 may not be strictly RAW data, but may be RAW data subjected to correction processing such as white balance adjustment or gain adjustment. The number of bits of the pixel values ​​in the second image signal 86 is not particularly limited; for example, it may be 1 second.

[0115] The second interpolation processing section 100 generates a color image based on the second image signal 86 acquired by the second signal acquisition section 98. The second interpolation processing section 100 calculates RGB values ​​set for each pixel 82 of the non-polarization image sensor 70 by performing a known debayer process.

[0116] If the number of pixels of the non-polarization image sensor 70 is different from the number of pixels of the polarization image sensor 20, the second interpolation processing section 100 can adjust the size of the debayer image generated by the debayer process. Thus, the second interpolation processing section 100 generates a color image whose number of vertical and horizontal pixels matches that of the output image of the first processing section 92.

[0117] The high-frequency extraction unit 102 extracts high-frequency components from the RGB values ​​of each pixel in the color image output from the second interpolation processing unit 100, thereby generating a high-frequency color image. The RGB values ​​of each pixel in the high-frequency color image are array data including pixel values ​​(Rh, Gh, Bh) for the high-frequency component of R (Rh), the high-frequency component of G (Gh), and the high-frequency component of B (Bh).

[0118] The high-frequency extraction unit 102 extracts low-frequency components from the color image output by the second interpolation processing unit 100 and subtracts the low-frequency components from the original color image to calculate the high-frequency components. For example, a FIR (Finite Impulse Response) filter with a cutoff frequency fc equal to 1 / 4 of the sampling frequency fs (i.e., fc = fs / 4) can be used as a low-pass filter (LPF) for extracting low-frequency components. Figure 10 An example of the frequency characteristics of the low-pass filter used in the high-frequency extraction unit 102 is shown. The LPF used in the high-frequency extraction unit 102 is not limited to Figure 10 For the LPF shown, the cutoff frequency fc can be less than or greater than 1 / 4 of the sampling frequency fs.

[0119] Figure 11 Schematically shows the flow of image processing by the second processing unit 94. Figure 11 For simplicity, only 16 pixels, 4×4 in vertical and horizontal dimensions, are cropped and shown. Input image 104 corresponds to RAW data based on second image signal 86, with each pixel storing only a single pixel value. Input image 104 is a Bayer array image corresponding to the arrangement of color filter layer 76.

[0120] The second interpolation processing unit 100 generates a debayer image 106 based on the input image 104. The debayer image 106 is a color image in which RGB values ​​are stored in each pixel. Figure 11In the example of , the resizing process of the debayer image 106 is omitted, but the second interpolation processing unit 100 may generate a color image after resizing the debayer image 106 .

[0121] The high-frequency extraction unit 102 generates a high-frequency color image 108 from the color image output from the second interpolation processing unit 100. The high-frequency color image 108 is a color image in which high-frequency RGB values ​​(RGBh) representing high-frequency components of RGB values ​​are stored in each pixel.

[0122] return Figure 8 The addition processing unit 96 generates a high-resolution color image by adding the composite RGB value (RGBm) output from the synthesis processing unit 48 of the first processing unit 92 and the high-frequency RGB value (RGBh) output from the second processing unit 94 for each pixel. The RGB value (RGBhr) of each pixel in the high-resolution color image is obtained by adding the high-frequency RGB value (RGBh) to the composite RGB value (RGBm), where RGBhr = RGBm + RGBh. The addition processing unit 96 can also calculate the high-resolution RGB value (RGBhr) by using the weighting coefficient h using the formula: RGBhr = RGBm + h·RGBh.

[0123] The addition processing unit 96 may also generate a high-resolution color image by adding the RGB values ​​(RGBa to RGBd) of the four polarization components output from the first interpolation processing unit 40 of the first processing unit 92 and the high-frequency RGB values ​​(RGBh) output from the second processing unit 94 on a pixel-by-pixel basis. The addition processing unit 96 may also calculate the high-resolution RGB values ​​of the first polarization component by adding the high-frequency RGB values ​​(RGBh) to the RGB values ​​of the first polarization component (RGBa) (RGBhra = RGBa + RGBh). The addition processing unit 96 may also calculate the high-resolution RGB values ​​of the second polarization component by adding the high-frequency RGB values ​​(RGBh) to the RGB values ​​of the second polarization component (RGBb) (RGBhrb = RGBb + RGBh). The addition processing unit 96 may also calculate the high-resolution RGB values ​​of the third polarization component (RGBc) by adding the high-frequency RGB values ​​(RGBh) to the RGB values ​​of the third polarization component (RGBhrc = RGBc + RGBh). The addition processing unit 96 may add the high-frequency RGB value (RGBh) to the RGB value (RGBd) of the fourth polarization component to calculate the high-resolution RGB value of the fourth polarization component (RGBhrd=RGBd+RGBh).

[0124] According to this embodiment, a high-resolution color image can be generated by adding the high-frequency component obtained from the output signal of the non-polarization image sensor 70 to the color image obtained from the output signal of the polarization image sensor 20. This allows for the generation of a high-resolution color image with suppressed aliasing, without the use of an optical low-pass filter.

[0125] Because the polarization image sensor 20 uses pixel groups 34 with 4×4 pixels as a repeating unit, its resolution is likely to be lower than that of a conventional (non-polarization) image sensor using 2×2 pixels as a repeating unit. Furthermore, if an optical low-pass filter is used in the polarization image sensor 20 to suppress aliasing, the optical low-pass filter will alter the polarization state, making it impossible to detect the appropriate polarization component. Consequently, when using the polarization image sensor 20, the optical low-pass filter cannot be used, resulting in images where aliasing is noticeable.

[0126] According to this embodiment, by using a non-polarization image sensor 70 without a polarizer layer, a high-resolution image can be easily obtained compared to the polarization image sensor 20, and high-frequency components that contribute to high resolution can be extracted. By adding these high-frequency components to the image obtained using the polarization image sensor 20, a color image with high resolution and suppressed distortion can be generated.

[0127] According to this embodiment, similar to the first embodiment, a color image with suppressed whitish and black casts can be generated. By adding high-frequency components (high-frequency RGB values, RGBh) to the color image with suppressed whitish and black casts (synthesized RGB values, RGBm), a more appropriate color image with high resolution, suppressed distortion, and suppressed whitish and black casts (high-resolution RGB values, RGBhr) can be generated.

[0128] According to this embodiment, by adding the high-frequency component (RGBh) to the color images of the four polarization components (RGBa to RGBd), more suitable polarization color images (RGBhra to RGBhrd) with high resolution and suppressed aliasing can be generated.

[0129] The addition processing unit 96 of the second embodiment may not calculate the high-resolution RGB value (RGBhr) using the composite RGB value (RGBm), but may instead calculate only at least one of the high-resolution RGB values ​​(RGBhra to RGBhrd) for the four polarization components. In this case, the first processing unit 92 may not include the brightness calculation unit 42, the sorting unit 44, the reference generation unit 46, and the synthesis unit 48. The first processing unit 92 may only include the first signal acquisition unit 38 and the first interpolation unit 40.

[0130] (Third embodiment)

[0131] Figure 12 This figure schematically illustrates the structure of an imaging device 10B according to the third embodiment. The imaging device 10B according to the third embodiment includes an imaging unit 12B and an image processing device 14B. The following description of the third embodiment will focus on the differences from the third embodiment, with similarities omitted as appropriate. In the accompanying drawings, identical components to those in the first or second embodiment are denoted by the same reference numerals.

[0132] The imaging unit 12B includes an imaging lens 18, a polarization image sensor 20, a non-polarization image sensor 70B, and a spectroscopic element 80. The imaging lens 18 and the polarization image sensor 20 have the same configuration as in the first embodiment. The spectroscopic element 80 is configured similarly to the second embodiment.

[0133] The non-polarization image sensor 70B includes a light detection layer 72 and a microlens layer 78. The non-polarization image sensor 70B differs from the non-polarization image sensor 70 of the second embodiment in that it does not include a color filter layer 76. The non-polarization image sensor 70B does not include the color filter layer 76 and thus functions as a non-polarization monochrome image sensor.

[0134] The image processing device 14B includes a first processing unit 92, a second processing unit 94B, and an addition unit 96B. The first processing unit 92 is configured similarly to the second embodiment. The second processing unit 94B includes a second signal acquisition unit 98B, a second interpolation unit 100B, and a high-frequency extraction unit 102B.

[0135] The second signal acquisition unit 98B acquires the second image signal 86B output from the non-polarization image sensor 70B. The second image signal 86B corresponds to the raw data output from the non-polarization image sensor 70B and is, for example, serial data representing the pixel values ​​of pixels 82 read in the order of their addresses. The second image signal 86B may not be strictly RAW data, but may be RAW data subjected to correction processing such as gain adjustment. The number of bits in the pixel values ​​of the second image signal 86B is not particularly limited; for example, it may be 1 second.

[0136] The second interpolation processing unit 100B generates a monochrome image based on the second image signal 86 acquired by the second signal acquisition unit 98. The second interpolation processing unit 100B outputs a luminance value (Y value) set for each pixel 82 of the non-polarization image sensor 70B. If the number of pixels of the non-polarization image sensor 70B differs from the number of pixels of the polarization image sensor 20, the second interpolation processing unit 100B resizes the monochrome image to generate a monochrome image having the same number of pixels in the vertical and horizontal directions as the output image of the first processing unit 92. Alternatively, if resizing is not required, the second processing unit 94B may not include the second interpolation processing unit 100B.

[0137] The high-frequency extraction unit 102B extracts the high-frequency components of the brightness values ​​(Y values) of each pixel of the monochrome image output from the second signal acquisition unit 98B or the second interpolation processing unit 100B, and generates a high-frequency monochrome image. The high-frequency extraction unit 102B extracts the low-frequency components from the monochrome image output from the second signal acquisition unit 98B or the second interpolation processing unit 100B, and subtracts the low-frequency components from the original monochrome image, thereby calculating the high-frequency components. As an LPF for extracting the low-frequency components, the same as Figure 10 The same filter is shown as a second embodiment.

[0138] Figure 13 Schematically shows the flow of image processing performed by the second processing unit 94B. Figure 13 For simplicity, only 16 pixels, 4×4 in vertical and horizontal dimensions, are cropped and shown. Input image 110 corresponds to a pixel array based on the RAW data of second image signal 86B, with each pixel assigned a single pixel value. Input image 110 is a monochrome image representing the luminance value (Y value) of each pixel.

[0139] The high-frequency extraction unit 102B generates a high-frequency monochromatic image 112 from the monochromatic image as the input image 110. The high-frequency monochromatic image 112 is a monochromatic image in which a high-frequency luminance value (Yh) representing a high-frequency component of a luminance value is provided for each pixel.

[0140] return Figure 12 The addition processing section 96B adds the synthesized RGB value (RGBm) output from the synthesizing processing section 48 of the first processing section 92 and the high-frequency luminance value (Yh) output from the second processing section 94B for each pixel to generate a high-resolution color image.

[0141] The RGB value (RGBhr) of each pixel in a high-resolution color image is obtained by adding the high-frequency luminance value (Yh) to the composite RGB value (RGBm): RGBhr = RGBm + Yh. For example, the values ​​(Rhr, Ghr, Bhr) of each color in the high-resolution RGB value (RGBhr) are calculated by adding the common high-frequency luminance value Yh to the values ​​(Rm, Gm, Bm) of each color in the composite RGB value (RGBm). That is, Rhr = Rm + Yh, Ghr = Gm + Yh, and Bhr = Bm + Yh.

[0142] The addition processing unit 96B can also calculate the high-resolution RGB value (RGBhr) by using the weighting coefficient h, using the formula RGBhr = RGBm + h·RGBh. The weighting coefficient h can be shared by each color or different for each color. In the latter case, the weighting coefficients hR, hG, and hB for each color are used, and can be set as Rhr = Rm + hR·Yh, Ghr = Gm + hG·Yh, and Bhr = Bm + hB·Yh. The weighting coefficients hR, hG, and hB for each color can be set based on ITU-R BT.709, such as hR = 0.02126, hG = 0.7152, and hB = 0.0722.

[0143] The addition processing unit 96B can also generate a high-resolution color image by adding the RGB values ​​(RGBa to RGBd) of the four polarization components output from the first interpolation processing unit 40 of the first processing unit 92 and the high-frequency luminance value (Yh) output from the second processing unit 94B on a pixel-by-pixel basis. The addition processing unit 96B can calculate the high-resolution RGB value of the first polarization component by adding the high-frequency luminance value (Yh) to the RGB value (RGBa) of the first polarization component (RGBhra = RGBa + Yh). The addition processing unit 96B can calculate the high-resolution RGB value of the second polarization component by adding the high-frequency luminance value (Yh) to the RGB value (RGBb) of the second polarization component (RGBhrb = RGBb + Yh). The addition processing unit 96B can calculate the high-resolution RGB value of the third polarization component by adding the high-frequency luminance value (Yh) to the RGB value (RGBc) of the third polarization component (RGBhrc = RGBc + Yh). The addition processing unit 96B can add the high-frequency luminance value (Yh) to the RGB value (RGBd) of the fourth polarization component to calculate the high-resolution RGB value of the fourth polarization component (RGBhrd = RGBd + Yh). When adding the high-frequency luminance value (Yh) to the RGB values ​​(RGBa to RGBd) of the four polarization components, the addition processing unit 96B can use the weighting coefficient h or the weighting coefficients hR, hG, and hB of each color.

[0144] This embodiment can achieve the same effects as the second embodiment. According to this embodiment, since the high-frequency component obtained from the output signal of the non-polarization monochrome image sensor is added, the cost of the non-polarization image sensor 70B and the second processing unit 94B can be reduced.

[0145] Similar to the second embodiment, the addition unit 96B of the third embodiment may calculate only at least one of the high-resolution RGB values ​​(RGBhra to RGBhrd) for the four polarization components, rather than calculating the high-resolution RGB value (RGBhr) using the composite RGB value (RGBm). In this case, the first processing unit 92 may not include the brightness calculation unit 42, the sorting unit 44, the reference generation unit 46, and the synthesis unit 48. The first processing unit 92 may also include only the first signal acquisition unit 38 and the first interpolation unit 40.

[0146] (Fourth embodiment)

[0147] Figure 14 This figure schematically illustrates the configuration of an imaging device 10C according to a fourth embodiment. The imaging device 10C according to the third embodiment includes an imaging unit 12C and an image processing device 14C. The following description of the fourth embodiment will focus on the differences from the aforementioned embodiments, with similarities omitted as appropriate. In the accompanying drawings, identical components to those in the aforementioned embodiments are denoted by the same reference numerals.

[0148] The imaging unit 12C includes an imaging lens 18, a polarization image sensor 20, a spectroscopic element 80C, a distance image sensor 120, and an illumination device 122. The imaging lens 18 and the polarization image sensor 20 have the same configuration as in the first embodiment.

[0149] The spectroscopic element 80C is arranged at the rear stage of the shooting lens 18. The spectroscopic element 80C splits the incident light 16 that has passed through the shooting lens 18 into visible light 16v and infrared light 16n. The visible light 16v is incident on the polarization image sensor 20, and the infrared light 16n is incident on the distance image sensor 120. The spectroscopic element 80C is, for example, a dichroic prism, and has a dichroic mirror that selectively transmits visible light and selectively reflects infrared light. The dichroic mirror is, for example, composed of a dielectric multilayer film. The dichroic mirror is preferably a non-polarization dichroic mirror that has no polarization dependence. By adopting a non-polarization dichroic mirror, the change in the polarization state of the visible light incident on the polarization image sensor 20 can be suppressed. The spectroscopic element 80C can also be configured to selectively reflect visible light and selectively transmit infrared light. In this case, the arrangement of the polarization image sensor 20 and the distance image sensor 120 is replaced.

[0150] The polarization image sensor 20 and the distance image sensor 120 are arranged coaxially with respect to the optical axis of the incident light 16. The photographing lens 18 is arranged to form an image of the incident light 16 on the light receiving surfaces of the polarization image sensor 20 and the non-polarization image sensor 70.

[0151] The distance image sensor 120 detects reflected light from infrared illumination light 124 irradiated from the lighting device 122 toward the subject. The distance image sensor 120 includes a plurality of pixels provided with infrared filters that selectively transmit infrared light. The distance image sensor 120 is, for example, a LIDAR (Light Detection and Ranging) sensor that measures the distance to the subject using a ToF (Time of Flight) method. The distance image sensor 120 outputs a distance value based on the elapsed time from the trigger timing of the pulse signal 116 supplied from the distance processing unit 126 to the light reception timing of each pixel. The distance image sensor 120 outputs, for example, serial data of the distance value detected by each pixel as a distance image signal 118.

[0152] Illumination device 122 irradiates infrared illumination light 124 toward the subject. Illumination device 122 includes a laser diode array, such as a VCSEL (Vertical Cavity Surface Emitting Laser). Illumination device 122 drives the laser diode array based on pulse signal 116 supplied from distance processing unit 126, emitting pulsed illumination light synchronized with the trigger timing of pulse signal 116.

[0153] The image processing device 14C includes a first processing unit 92, a distance processing unit 126, and a point cloud data generating unit 128. The first processing unit 92 is configured similarly to the second embodiment. The distance processing unit 126 includes a timing control unit 130, a distance signal acquiring unit 132, and a three-dimensional position calculating unit 134.

[0154] The timing control unit 130 generates a pulse signal 116 for driving the distance image sensor 120 and the lighting device 122. The distance signal acquisition unit 132 acquires the distance image signal 118 from the distance image sensor 120. The three-dimensional position calculation unit 134 uses the distance image signal 118 as input to calculate the coordinate values ​​(x, y, z) representing the three-dimensional position of the object. The three-dimensional position calculation unit 134 calculates the coordinate values ​​(x, y, z) representing the three-dimensional position for each pixel based on the distance value detected by each pixel of the distance image sensor 120. The coordinate values ​​(x, y, z) corresponding to each pixel correspond to the three-dimensional position of each reflection point on the surface of the object caused by the infrared light incident on each pixel.

[0155] The point cloud data generation unit 128 generates point cloud data by associating the RGB values ​​output from the first processing unit 92 with the coordinate values ​​output from the distance processing unit 126 for each pixel. The point cloud data generation unit 128 can generate point cloud data in a file format that conforms to the PLY (Polygon File Format). In the PLY file format, (x, y, z, R, G, B) array data is set for each pixel.

[0156] The point cloud data generation unit 128 may generate point cloud data by associating, for each pixel, the composite RGB value (RGBm) output from the composite processing unit 48 of the first processing unit 92 and the coordinate value (x, y, z) output from the distance processing unit 126. In this case, for example, array data of (x, y, z, Rm, Gm, Bm) is set for each pixel using the PLY file format.

[0157] The point cloud data generation unit 128 may also generate point cloud data that associates the RGB values ​​(RGBa to RGBd) of the four polarization components output from the first interpolation processing unit 40 of the first processing unit 92 with the coordinate values ​​(x, y, z) output from the distance processing unit 126 for each pixel. In this case, the point cloud data associates the coordinate values ​​(x, y, z) with the RGB values ​​(Ra, Ga, Ba) of the first polarization component, the RGB values ​​(Rb, Gb, Bb) of the second polarization component, the RGB values ​​(Rc, Gc, Bc) of the third polarization component, and the RGB values ​​(Rd, Gd, Bd) of the fourth polarization component for each pixel. For example, using a PLY file format, array data of (x, y, z, Ra, Ga, Ba, Rb, Gb, Bb, Rc, Gc, Bc, Rd, Gd, Bd) is set for each pixel. The setting order of the four polarization components can be an ascending order (for example, in the order of 0 degrees, 45 degrees, 90 degrees, and 135 degrees) or a descending order (for example, in the order of 135 degrees, 90 degrees, 45 degrees, and 0 degrees).

[0158] This embodiment generates point cloud data that associates composite RGB values ​​(RGBm) with coordinate values ​​(x, y, z) for each pixel, and provides point cloud data with suppressed whitening and blackening. By modeling the three-dimensional shape of a subject in virtual space based on this point cloud data, a less awkward three-dimensional model can be provided even when virtual lighting is applied to the subject in virtual space.

[0159] According to this embodiment, point cloud data can be generated that associates the RGB values ​​(RGBa to RGBd) of the four polarization components with coordinate values ​​(x, y, z) for each pixel. By using this point cloud data, the normal vector of each pixel can be calculated based on the four polarization components. Thus, a three-dimensional model of the subject can be generated using the point cloud with the normal vector of each pixel and distance information.

[0160] (Fifth embodiment)

[0161] Figure 15 This figure schematically illustrates the structure of an imaging device 10D according to a fifth embodiment. The imaging device 10D includes an imaging unit 12D and an image processing device 14D. The following description of the fifth embodiment will focus on the differences from the above-described embodiments, with similarities omitted as appropriate. In the drawings, identical components to those in the above-described embodiments are denoted by the same reference numerals.

[0162] The imaging unit 12D includes an imaging lens 18, a spectroscopic element 140, a first image sensor 142, a second image sensor 144, a third image sensor 146, and the lighting device 122. The imaging unit 12D is a so-called three-panel camera, and uses the spectroscopic element 140 to split the incident light 16 that has passed through the imaging lens 18 into three light beams 16a, 16b, and 16c. The imaging unit 12D is configured to capture the incident light 16 using the first image sensor 142, the second image sensor 144, and the third image sensor 146, respectively.

[0163] Beam splitting element 140 is a so-called three-plate prism. First image sensor 142, second image sensor 144, and third image sensor 146 are polarization image sensors, non-polarization image sensors, or distance image sensors. Specifically, one of first image sensor 142, second image sensor 144, and third image sensor 146 is a polarization image sensor, another of first image sensor 142, second image sensor 144, and third image sensor 146 is a non-polarization image sensor, and yet another of first image sensor 142, second image sensor 144, and third image sensor 146 is a distance image sensor.

[0164] The polarization image sensor included in the imaging unit 12D is configured similarly to the polarization image sensor 20 of the aforementioned embodiment. The polarization image sensor outputs a first image signal 36. The non-polarization image sensor included in the imaging unit 12D is configured similarly to the non-polarization image sensor 70 of the aforementioned second embodiment or the non-polarization image sensor 70B of the aforementioned third embodiment. The non-polarization image sensor outputs a second image signal 86. The range image sensor included in the imaging unit 12D is configured similarly to the range image sensor 120 of the aforementioned fourth embodiment. The range image sensor operates in synchronization with the pulse signal 116 and outputs a range image signal 118.

[0165] The lighting device 122 is configured similarly to the fourth embodiment described above and operates in synchronization with the pulse signal 116 to irradiate the subject with infrared illumination light 124 .

[0166] The image processing device 14D includes a first processing unit 92, a second processing unit 94, an addition processing unit 96, a distance processing unit 126, and a point cloud data generation unit 128D. The first processing unit 92, the second processing unit 94, and the addition processing unit 96 can be configured similarly to the second or third embodiment described above. The distance processing unit 126 can be configured similarly to the fourth embodiment described above.

[0167] The point cloud data generating unit 128D generates point cloud data by associating the high-resolution RGB values ​​(RGBhr or RGBhra to RGBhrd) output from the adding unit 96 with the coordinate values ​​(x, y, z) output from the distance processing unit 126 for each pixel.

[0168] The point cloud data generator 128D may also generate point cloud data by associating high-resolution RGB values ​​(Rhr, Ghr, Bhr) using composite RGB values ​​(RGBm) with coordinate values ​​(x, y, z) for each pixel. In this case, for example, a PLY file format is used, and array data of (x, y, z, Rhr, Ghr, Bhr) is set for each pixel.

[0169] The point cloud data generator 128D may also generate point cloud data that associates high-resolution RGB values ​​(RGBhra to RGBhrd) using the RGB values ​​(RGBa to RGBd) of the four polarization components with coordinate values ​​(x, y, z) for each pixel. In this case, the point cloud data associates the coordinate values ​​(x, y, z) for each pixel with the high-resolution RGB values ​​(Rhra, Ghra, Bhra) of the first polarization component, the high-resolution RGB values ​​(Rhrb, Ghrb, Bhrb) of the second polarization component, the high-resolution RGB values ​​(Rhrc, Ghrc, Bhrc) of the third polarization component, and the high-resolution RGB values ​​(Rhrd, Ghrd, Bhrd) of the fourth polarization component. For example, using a PLY file format, array data of (x, y, z, Rhra, Ghra, Bhra, Rhrb, Ghrb, Bhrb, Rhrc, Ghrc, Bhrc, Rhrd, Ghrd, Bhrd) is set for each pixel. The setting order of the four polarization components can be an ascending order (for example, in the order of 0 degrees, 45 degrees, 90 degrees, and 135 degrees) or a descending order (for example, in the order of 135 degrees, 90 degrees, 45 degrees, and 0 degrees).

[0170] This embodiment generates point cloud data that associates high-resolution RGB values ​​(RGBhr) with coordinate values ​​(x, y, z) for each pixel using composite RGB values ​​(RGBm). By adding high-frequency components to the RGB values ​​in this point cloud data, it is possible to provide more precise point cloud data with high resolution, suppressed distortion, and reduced whitening and blackening. By modeling the three-dimensional shape of a subject in virtual space based on this point cloud data, a less awkward three-dimensional model can be created, even when virtual lighting is applied to the subject in virtual space.

[0171] According to this embodiment, point cloud data can be generated that associates high-resolution RGB values ​​(RGBhra to RGBhrd) using the RGB values ​​of the four polarization components (RGBa to RGBd) with coordinate values ​​(x, y, z) for each pixel. Since high-frequency components are added to the RGB values ​​of each of the four polarization components in the point cloud data, it is possible to provide more optimal point cloud data with high resolution and reduced distortion. By utilizing this point cloud data, the normal vector for each pixel can be calculated based on the four polarization components. Thus, a three-dimensional model of the subject can be generated using the point cloud with the normal vector for each pixel and distance information.

[0172] Figure 16 It is a diagram schematically showing the imaging unit 136 of the first configuration example according to the fifth embodiment. Figure 16 The photographing unit 136 can be used as Figure 15The imaging unit 12D includes an imaging lens 138 , a spectroscopic element 140 , a first image sensor 142 , a second image sensor 144 , a third image sensor 146 , and a phase difference plate 148 .

[0173] The spectroscopic element 140 includes a first prism 152, a second prism 154, and a third prism 156. The first prism 152 includes a first incident surface 158, a first splitting surface 160, and a first exit surface 162. The second prism 154 includes a second incident surface 164, a second splitting surface 166, and a second exit surface 168. The third prism 156 includes a third incident surface 170 and a third exit surface 172. An air gap is provided between the first splitting surface 160 and the second incident surface 164.

[0174] Incident light 180 incident on first incident surface 158 is split by first splitting surface 160 into first reflected light 182 and first transmitted light 184. First reflected light 182 reflected by first splitting surface 160 undergoes total internal reflection at first incident surface 158, then passes through first exit surface 162 toward first image sensor 142. First transmitted light 184 passing through first splitting surface 160 is split by second splitting surface 166 into second reflected light 186 and second transmitted light 188. Second reflected light 186 reflected by second splitting surface 166 undergoes total internal reflection at second incident surface 164, then passes through second exit surface 168 toward second image sensor 144. Second transmitted light 188, having passed through second splitting surface 166, passes through third incident surface 170 and third exit surface 172 and toward third image sensor 146.

[0175] exist Figure 16 In the first configuration example, the first image sensor 142 can be configured as a polarization image sensor 20. The first reflected light 182 directed toward the first image sensor 142 is reflected by the first split surface 160 and the first incident surface 158, and thus the polarization state of the first reflected light 182 may change. In other words, the polarization state of the first reflected light 182 may change from the polarization state of the incident light 180. In particular, when the first reflected light 182 is totally reflected within the first incident surface 158, which serves as the interface between the prism and air, the polarization state of the first reflected light 182 may change significantly. If the polarization state of the first reflected light 182 changes from the polarization state of the incident light 180, the polarization image sensor 20 may not be able to accurately measure the polarization state of the incident light 180.

[0176] exist Figure 16In the first configuration example, a phase difference plate 148 is provided between the first image sensor 142 and the beam splitter 140 to compensate for changes in the polarization state of the first reflected light 182 caused by the beam splitter 140. The phase difference provided by the phase difference plate 148 is set to reduce or cancel the phase difference between the s-polarization component and the p-polarization component of the first reflected light 182 resulting from at least one of reflection on the first splitting surface 160 and reflection on the first incident surface 158. The magnitude of the phase difference provided by the phase difference plate 148 is not particularly limited and is, for example, approximately 120 degrees.

[0177] When the first image sensor 142 is set as the polarization image sensor 20, the first dividing surface 160 preferably includes a non-polarization beam splitter that has no wavelength dependence. Here, "non-polarization" means that the influence on polarized light is small enough to be ignored. The non-polarization beam splitter is constructed so that the change in the polarization state before and after reflection and before and after transmission of the beam splitter is small enough to be ignored. As a non-polarization beam splitter that has no wavelength dependence, for example, a metal thin film can be used. As a non-polarization beam splitter that has no wavelength dependence, a dielectric multilayer film designed in a manner to suppress changes in the polarization state can also be used. By providing a non-polarization beam splitter on the first dividing surface 160, changes in the polarization state of the first reflected light 182 reflected by the first dividing surface 160 can be suppressed.

[0178] When the first image sensor 142 is configured as the polarization image sensor 20, one of the second image sensor 144 and the third image sensor 146 is the non-polarization image sensor 70 or 70B, and the other is the distance image sensor 120. In this case, the second dividing surface 166 preferably includes a dichroic mirror that separates visible light and infrared light. A dielectric multilayer film, for example, can be used as the dichroic mirror. When the second image sensor 144 is the non-polarization image sensor 70 or 70B, the dichroic mirror provided on the second dividing surface 166 is designed to selectively reflect visible light and selectively transmit infrared light. When the second image sensor 144 is the distance image sensor 120, the dichroic mirror provided on the second dividing surface 166 is designed to selectively transmit visible light and selectively reflect infrared light. Alternatively, the second dividing surface 166 may include a beam splitter that is not wavelength-dependent, such as a semi-transparent mirror, instead of a dichroic mirror.

[0179] according to Figure 16In the first configuration example, by using the spectroscopic element 140, an image of the incident light 180 can be captured using each of the polarization image sensor 20, the non-polarization image sensor 70 or 70B, and the distance image sensor 120, and a polarization image, a non-polarization image, and a distance image can be obtained. According to this embodiment, by setting the first image sensor 142 as the polarization image sensor 20, it is possible to configure a structure in which a dichroic mirror is not arranged on the optical path toward the polarization image sensor 20. Thus, a polarization image that is not affected by changes in the polarization state in the dichroic mirror can be obtained. In addition, by providing a phase difference plate 148 between the spectroscopic element 140 and the polarization image sensor 20 (first image sensor 142), changes in the polarization state of the first reflected light 182 generated on at least one of the first dividing surface 160 and the first incident surface 158 can be suppressed. Thus, a polarization image in which changes in the polarization state relative to the incident light 180 are suppressed can be obtained.

[0180] Figure 17 This diagram schematically illustrates imaging unit 136A of a second configuration example according to the fifth embodiment. This second configuration example differs from the first embodiment described above in that a phase shift plate 148A is provided between second image sensor 144 and spectroscopic element 140. The following description of the second configuration example will focus on differences from the first configuration example, with similarities omitted as appropriate.

[0181] The imaging unit 136A includes an imaging lens 138, a spectroscopic element 140, a first image sensor 142, a second image sensor 144, a third image sensor 146, and a phase difference plate 148A. In the second embodiment, the second image sensor 144 is a polarization image sensor 20. In the second embodiment, one of the first image sensor 142 and the third image sensor 146 is a non-polarization image sensor 70 or 70B, and the other is a distance image sensor 120.

[0182] When the first image sensor 142 is a non-polarization image sensor 70 or 70B and the third image sensor 146 is a distance image sensor 120, the first dividing surface 160 preferably includes a non-polarization beam splitter that is not wavelength-dependent. Providing a non-polarization beam splitter on the first dividing surface 160 can suppress changes in the polarization state of the first transmitted light 184 passing through the first dividing surface 160. When the first image sensor 142 is a non-polarization image sensor 70 or 70B and the third image sensor 146 is a distance image sensor 120, the second dividing surface 166 is preferably a non-polarization dichroic mirror. The dichroic mirror provided on the second dividing surface 166 is designed to selectively reflect visible light and selectively transmit infrared light. Providing a non-polarization dichroic mirror on the second dividing surface 166 can suppress changes in the polarization state of the second reflected light 186 reflected by the second dividing surface 166. Alternatively, the second dividing surface 166 may include a non-polarization beam splitter that is not wavelength-dependent, such as a half-mirror, instead of a dichroic mirror.

[0183] When the first image sensor 142 is the distance image sensor 120 and the third image sensor 146 is the non-polarization image sensor 70 or 70B, the first dividing surface 160 is preferably a non-polarization dichroic mirror. As the non-polarization dichroic mirror, for example, a dielectric multilayer film designed to suppress the change in polarization state from the visible region to the infrared region can be used. The dichroic mirror provided on the first dividing surface 160 is designed to selectively transmit visible light and selectively reflect infrared light. By providing the non-polarization dichroic mirror on the first dividing surface 160, the change in polarization state of the first transmitted light 184 passing through the first dividing surface 160 can be suppressed. In addition, the first dividing surface 160 may also not include a dichroic mirror, but instead include a non-polarization beam splitter with no wavelength dependence. When the first image sensor 142 is the distance image sensor 120 and the third image sensor 146 is the non-polarization image sensor 70 or 70B, the second dividing surface 166 preferably includes a non-polarization beam splitter with no wavelength dependence. By providing a non-polarization beam splitter on the second split surface 166 , it is possible to suppress changes in the polarization state of the second reflected light 186 reflected by the second split surface 166 .

[0184] The phase difference of the phase difference plate 148A is set to reduce or cancel the phase difference between the s-polarization component and the p-polarization component of the second reflected light 186 caused by at least one of the transmission of the first split surface 160, the reflection of the second split surface 166, and the reflection of the second incident surface 164. The phase difference plate 148A compensates for the change in the polarization state of the second reflected light 186 caused by the spectroscopic element 140. In particular, when the second reflected light 186 is totally reflected inside the second incident surface 164, which is the interface between the prism and the air, the polarization state of the second reflected light 186 may change significantly. The magnitude of the phase difference imparted by the phase difference plate 148A is not particularly limited, and is, for example, about 120 degrees.

[0185] Also in Figure 17 In the second configuration example, by using the spectroscopic element 140, an image of the incident light 180 can be captured using each of the polarization image sensor 20, the non-polarization image sensor 70 or 70B, and the distance image sensor 120, and a polarization image, a non-polarization image, and a distance image can be obtained. According to this embodiment, by providing a phase difference plate 148A between the spectroscopic element 140 and the polarization image sensor 20 (the second image sensor 144), changes in the polarization state of the second reflected light 186 generated by at least one of the first dividing surface 160, the second dividing surface 166, and the second incident surface 164 can be suppressed. According to this embodiment, by providing a non-polarization dichroic mirror on the first dividing surface 160 or the second dividing surface 166, a polarization image in which changes in the polarization state in the dichroic mirror are suppressed can be obtained. According to this embodiment, by providing a non-polarization beam splitter on the first dividing surface 160 or the second dividing surface 166, a polarization image in which changes in the polarization state in the beam splitter are suppressed can be obtained.

[0186] Figure 18 This diagram schematically illustrates imaging unit 136B of the third configuration example of the fifth embodiment. This third configuration example differs from the first and second configuration examples described above in that phase shift plates 148 and 148A are not provided. The following description of the third configuration example focuses on the differences between the first and second configuration examples, omitting any commonalities where appropriate.

[0187] The imaging unit 136B includes a first image sensor 142, a second image sensor 144, a third image sensor 146, and a spectroscopic element 140. In the third embodiment, the third image sensor 146 is a polarization image sensor 20. In the third embodiment, one of the first image sensor 142 and the second image sensor 144 is a non-polarization image sensor 70 or 70B, and the other of the first image sensor 142 and the second image sensor 144 is a distance image sensor 120.

[0188] When the first image sensor 142 is a non-polarization image sensor 70 or 70B and the second image sensor 144 is a distance image sensor 120, the first dividing surface 160 preferably includes a non-polarization beam splitter that is not wavelength-dependent. Providing a non-polarization beam splitter on the first dividing surface 160 can suppress changes in the polarization state of the first transmitted light 184 that passes through the first dividing surface 160. When the first image sensor 142 is a non-polarization image sensor 70 or 70B and the second image sensor 144 is a distance image sensor 120, the second dividing surface 166 is preferably a non-polarization dichroic mirror. The dichroic mirror provided on the second dividing surface 166 is designed to selectively transmit visible light and selectively reflect infrared light. Providing a non-polarization dichroic mirror on the second dividing surface 166 can suppress changes in the polarization state of the second transmitted light 188 that passes through the second dividing surface 166. Alternatively, the second dividing surface 166 can include a non-polarization beam splitter that is not wavelength-dependent, such as a half-mirror, instead of a dichroic mirror.

[0189] When the first image sensor 142 is the distance image sensor 120 and the second image sensor 144 is the non-polarization image sensor 70 or 70B, the first dividing surface 160 is preferably a non-polarization dichroic mirror. The dichroic mirror provided on the first dividing surface 160 is designed to selectively transmit visible light and selectively reflect infrared light. By providing the non-polarization dichroic mirror on the first dividing surface 160, changes in the polarization state of the first transmitted light 184 passing through the first dividing surface 160 can be suppressed. In addition, the first dividing surface 160 can also be provided with a non-polarization beam splitter that is not wavelength-dependent without a dichroic mirror. When the first image sensor 142 is the distance image sensor 120 and the second image sensor 144 is the non-polarization image sensor 70 or 70B, the second dividing surface 166 preferably includes a non-polarization beam splitter that is not wavelength-dependent. By providing the non-polarization beam splitter on the second dividing surface 166, changes in the polarization state of the second transmitted light 188 passing through the second dividing surface 166 can be suppressed.

[0190] In the third configuration example, by using the spectroscopic element 140, it is also possible to capture the incident light 180 using the polarization image sensor 20, the non-polarization image sensor 70 or 70B, and the distance image sensor 120, respectively, to obtain a polarization image, a non-polarization image, and a distance image. According to this embodiment, by setting the third image sensor 146 as the polarization image sensor 20, it is possible to set a structure in which internal total reflection does not occur on the optical path toward the polarization image sensor 20. As a result, a polarization image that is not affected by changes in the polarization state caused by internal total reflection can be obtained. According to this embodiment, by providing non-polarization dichroic mirrors or beam splitters on the first dividing surface 160 and the second dividing surface 166, changes in the polarization state of the second transmitted light 188 caused by transmission through the first dividing surface 160 and the second dividing surface 166 can be suppressed. As a result, even if a phase difference plate is not provided between the spectroscopic element 140 and the polarization image sensor 20 (third image sensor 146), an appropriate polarization image can be obtained.

[0191] (Sixth embodiment)

[0192] Figure 19 Schematically shows the configuration of an endoscope system 200 according to the sixth embodiment. The endoscope system 200 includes an endoscope 202 and an image processing device 204. The endoscope 202 includes an insertion portion 212 having a distal end portion 210, an operation portion 214, and a connection portion 216.

[0193] The insertion portion 212 is the portion inserted into the interior of the object being observed. The insertion portion 212 is, for example, made of a flexible member, and is configured so that the orientation of the distal end portion 210 can be adjusted by bending the area near the distal end portion 210. In this case, the endoscope 202 is a flexible endoscope. Alternatively, the insertion portion 212 may be made of a non-flexible member. In this case, the endoscope 202 is a rigid endoscope.

[0194] The distal end portion 210 is a portion facing the internal object of the observation target, and is provided at the distal end of the insertion portion 212. Inside the distal end portion 210, an imaging unit 220 is provided.

[0195] The operation portion 214 is a portion gripped by a user who uses the endoscope 202. The operation portion 214 is provided with an operation knob (not shown) for changing the orientation of the distal end portion 210 and the like.

[0196] The connection portion 216 is an interface for connecting the endoscope 202 to the image processing device 204. The image signal output from the imaging unit 220 is transmitted to the image processing device 204 via the connection portion 216 through the transmission cable 230 provided inside the insertion portion 212 and the operation portion 214.

[0197] The image processing device 204 performs image processing with the image signal output from the photographing unit 220 and transmitted through the transmission cable 230 as input.

[0198] The endoscope system 200 may include any of the imaging devices 10A, 10B, 10C, and 10D according to the above-described embodiments. The imaging unit 220 may be configured similarly to the imaging units 12, 12A, 12B, 12C, 12D, 136, 136A, and 136B according to the above-described embodiments. The image processing device 204 may be configured similarly to any of the image processing devices 14, 14A, 14B, 14C, and 14D according to the above-described embodiments.

[0199] The transmission cable 230 can transmit at least any one of the first image signal 36, the second image signal 86, 86B, and the distance image signal 118 output from the imaging unit 220 to the image processing device 204. The transmission cable 230 can transmit the pulse signal 116 from the image processing device 204 to the imaging unit 220.

[0200] According to this embodiment, a clearer color image can be generated using the image signal obtained from the imaging unit 220 provided at the front end portion 210 of the endoscope 202. For example, by synthesizing the RGB values ​​of the four polarization components, a color image with suppressed whitening or blackening can be generated. In addition, by adding the high-frequency components obtained from the output signals of the non-polarized image sensors 70 and 70B to the color image obtained from the output signal of the polarized image sensor 20, a high-resolution color image can be generated. Thus, a color image with high resolution and suppressed folding and deformation can be generated. In addition, by adding the high-frequency components to the color images of each of the four polarization components, a more appropriate polarized color image with high resolution and suppressed folding and deformation can be generated. In addition, by further using a distance image sensor, point group data can be generated that corresponds the coordinate values ​​representing the three-dimensional position of the subject to the RGB values ​​based on the clear color image. Thus, a more appropriate three-dimensional model of the subject can be generated.

[0201] (Seventh embodiment)

[0202] Figure 20 This figure schematically illustrates the configuration of an endoscope system 200A according to a seventh embodiment. The endoscope system 200A of the seventh embodiment differs from the aforementioned sixth embodiment in that it is a binocular system. The following description of the seventh embodiment will focus on the differences from the sixth embodiment, with similarities omitted as appropriate. In the accompanying drawings, components identical to those of the sixth embodiment are denoted by the same reference numerals.

[0203] The endoscope system 200A includes an endoscope 202A and an image processing device 204A. The endoscope 202A includes an insertion portion 212 having a distal end portion 210 , an operation portion 214 , and a connection portion 216 .

[0204] A first imaging unit 222 and a second imaging unit 224 are provided within the front end portion 210. The first imaging unit 222 and the second imaging unit 224 are provided, for example, side by side. The first imaging unit 222 and the second imaging unit 224 can be configured similarly to the imaging units 12, 12A, 12B, 12C, 12D, 136, 136A, and 136B of the aforementioned embodiments, respectively.

[0205] A first transmission cable 232 and a second transmission cable 234 are provided inside the insertion portion 212 and the operation portion 214. The first transmission cable 232 transmits image signals output from the first imaging unit 222. The second transmission cable 234 transmits image signals output from the second imaging unit 224.

[0206] The image processing device 204A includes a first image processing unit 242 and a second image processing unit 244. The first image processing unit 242 performs image processing using as input an image signal output from the first imaging unit 222 and transmitted via the first transmission cable 232. The second image processing unit 244 performs image processing using as input an image signal output from the second imaging unit 224 and transmitted via the second transmission cable 234. The first image processing unit 242 and the second image processing unit 244 can each be configured similarly to any of the image processing devices 14, 14A, 14B, 14C, and 14D according to the above-described embodiments.

[0207] According to this embodiment, a stereoscopic image can be generated using image signals acquired from the first imaging unit 222 and the second imaging unit 224 provided in the distal end portion 210 of the endoscope 202A. According to this embodiment, the left and right color images constituting the stereoscopic image can be made clearer.

[0208] As mentioned above, the present invention has been described with reference to the above-mentioned embodiments. However, the present invention is not limited to the above-mentioned embodiments, and structures obtained by appropriately combining or replacing the structures shown in the respective display examples are also included in the present invention.

[0209] Several aspects of the invention are described below.

[0210] According to a first aspect of the present disclosure, a shooting device is provided, comprising: a polarization image sensor, in which pixel groups are arranged in two dimensions, and an RGB filter is arranged in a Bayer arrangement according to each pixel group, wherein the pixel group includes 2×2 pixels for detecting four polarization components that are different for each pixel; an interpolation processing unit, which separates the pixel values ​​output from the polarization image sensor according to each polarization component to generate four Bayer array images corresponding to the four polarization components, and calculates the RGB values ​​of the four polarization components for each pixel by debayering and up-converting the four Bayer array images respectively; a brightness calculation unit, which calculates the brightness values ​​of the four polarization components for each pixel according to the RGB values ​​of the four polarization components; and a sorting processing unit, which sorts the brightness values ​​of the four polarization components in the order of the brightness values ​​of the four polarization components. Each pixel sorts the four polarization components; a reference generation unit calculates, for each pixel, a reference brightness value obtained by synthesizing the brightness values ​​of multiple polarization components including at least the second and third digits; and a synthesis processing unit, a) when the brightness value is consistent with a predetermined threshold, outputs a reference RGB value obtained by synthesizing the RGB values ​​of the multiple polarization components; b) when the reference brightness value is less than the threshold, outputs a synthesized RGB value obtained by synthesizing the upper RGB value and the reference RGB value, wherein the upper RGB value is obtained by synthesizing the RGB values ​​of the first and second polarization components; c) when the reference brightness value is less than the threshold, outputs a synthesized RGB value obtained by synthesizing the lower RGB value and the reference RGB value, wherein the lower RGB value is obtained by synthesizing the RGB values ​​of the third and fourth polarization components.

[0211] In the first aspect, the synthesis processing unit may also b) increase the synthesis weight of the upper RGB value as the reference brightness value becomes smaller for pixels whose reference brightness value is smaller than the threshold value, and c) increase the synthesis weight of the lower RGB value as the reference brightness value becomes larger for pixels whose reference brightness value is larger than the threshold value.

[0212] In the first aspect, the shooting device may also include: a non-polarized image sensor, in which the number of pixels arranged in two dimensions is greater than the number of pixel groups arranged in the polarized image sensor; a spectroscopic element, which divides the incident light into light toward the polarized image sensor and light toward the non-polarized image sensor; a high-frequency extraction unit, which uses the output image of the image sensor to extract the high-frequency component of each pixel; and an addition processing unit, which adds the high-frequency component to the RGB value output from the synthesis processing unit according to each pixel to generate a high-resolution image.

[0213] In the first aspect, the shooting device may further include: a distance image sensor for measuring the distance to the subject; a spectroscopic element for dividing the incident light into light toward the polarization image sensor and light toward the distance image sensor; a three-dimensional position calculation unit for calculating, for each pixel, a coordinate value representing the three-dimensional position of the subject using an output signal from the distance image sensor; and a point group data generation unit for generating, for each pixel, point group data by corresponding the RGB value output from the synthesis processing unit and the coordinate value calculated by the three-dimensional position calculation unit.

[0214] In the first aspect, the imaging device may further include: a non-polarized image sensor having a two-dimensional arrangement of pixels having a number greater than the number of arrangement of the pixel group of the polarized image sensor; a distance image sensor for measuring the distance to an object; a spectroscopic element for dividing incident light into light toward the polarized image sensor, light toward the non-polarized image sensor, and light toward the distance image sensor; a high-frequency extraction unit for extracting a high-frequency component of each pixel using an output image of the image sensor; an addition processing unit for adding the high-frequency component to the RGB value output from the synthesis processing unit for each pixel to generate a high-resolution image; a three-dimensional position calculation unit for calculating a coordinate value representing the three-dimensional position of the subject for each pixel using the output signal of the distance image sensor; and a point group data generation unit for generating point group data, the point group data being data in which the RGB value of the high-resolution image generated by the addition processing unit corresponds to the coordinate value calculated by the three-dimensional position calculation unit for each pixel.

[0215] According to a second aspect of the present disclosure, a polarization image sensor is provided, comprising: a polarization image sensor having pixel groups arranged in two dimensions, an RGB filter arranged in a Bayer pattern according to each pixel group, the pixel group comprising 2×2 pixels for detecting four polarization components that are different for each pixel; a non-polarization image sensor having pixels arranged in two dimensions, the number of pixels being greater than the number of pixel groups arranged in the polarization image sensor; a spectroscopic element for dividing incident light into light toward the polarization image sensor and light toward the non-polarization image sensor; an interpolation processing unit for separating the pixel values ​​output from the polarization image sensor according to each polarization component to generate four Bayer array images corresponding to the four polarization components, and calculating the RGB values ​​of the four polarization components for each pixel by debayering and up-converting the four Bayer array images respectively; a high-frequency extraction unit for extracting the high-frequency component of each pixel using the output image of the non-polarization image sensor; and an addition processing unit for adding the high-frequency component to the RGB values ​​of the four polarization components output from the interpolation processing unit according to each pixel to generate four high-resolution images corresponding to the four polarization components.

[0216] In the first aspect or the second aspect, the non-polarized image sensor may be a color image sensor in which RGB color filters are Bayer-arranged for each pixel, the high-frequency extraction unit may extract the high-frequency component from the RGB value of each pixel calculated by debayering the Bayer array image output from the non-polarized image sensor, and the addition processing unit may add the RGB value representing the high-frequency component to the RGB value of each of the four polarization components for each pixel.

[0217] In the first aspect or the second aspect, the non-polarized image sensor may be a monochrome image sensor that does not include a color filter, the high-frequency extraction unit may extract a high-frequency component from the brightness value of each pixel output from the monochrome image sensor, and the addition processing unit may add the brightness value representing the high-frequency component to each of the RGB values ​​of the four polarization components for each pixel.

[0218] In the second embodiment, the shooting device may also include a distance image sensor for measuring the distance to the subject, and the spectroscopic element may further divide the incident light into light toward the distance image sensor. The shooting device may also include: a three-dimensional position calculation unit, which uses the output signal of the distance image sensor to calculate the coordinate value representing the three-dimensional position of the subject for each pixel; and a point group data generation unit, which generates point group data, wherein the point group data is data that corresponds, for each pixel, the RGB value of each of the four high-resolution images output from the addition processing unit to the coordinate value calculated by the three-dimensional position calculation unit.

[0219] A third aspect of the present invention is an endoscope system having the imaging device of the first or second aspect. The endoscope system includes: an endoscope having an insertion portion having a front end facing a subject, a transmission cable disposed within the insertion portion; and an image processing device that acquires signals via the transmission cable.

[0220] In the third aspect, the polarization image sensor can be arranged in the front end portion, the transmission cable can transmit the output signal of the polarization image sensor, the image processing device can include the interpolation processing unit, the brightness calculation unit, the sorting processing unit, the reference generation unit and the synthesis processing unit, and can obtain the output signal of the polarization image sensor.

[0221] In the third aspect, the polarization image sensor, the non-polarization image sensor and the spectroscopic element can be arranged in the front end portion, the transmission cable can transmit the output signals of the polarization image sensor and the non-polarization image sensor, the image processing device can include the interpolation processing unit, the high-frequency extraction unit and the addition processing unit, and can obtain the output signals of the polarization image sensor and the non-polarization image sensor.

[0222] Industrial Application Possibilities

[0223] According to the present invention, a clear image can be generated using a polarization image sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0224] 10, 10A, 10B, 10C, 10D…shooting device, 12, 12A, 12B, 12C, 12D…shooting unit, 14, 14A, 14B, 14C, 14D…image processing device, 16…incident light, 18…shooting lens, 20…polarization image sensor, 22…light detection layer, 24…polarizer layer, 26…color filter layer, 28…microlens layer, 30…pixel, 32…pixel group, 34…pixel group, 38…signal acquisition unit, 40…interpolation processing unit, 42…brightness calculation unit, 44…sorting processing unit, 46…reference generation unit, 48…combination processing unit, 60…reference synthesis unit, 62…upper synthesis unit, 64…lower synthesis unit, 66… Coefficient calculation unit, 68…output synthesis unit, 70, 70B…non-polarized image sensor, 72…light detection layer, 76…color filter layer, 78…microlens layer, 80, 80C…spectrographic element, 82…pixel, 84…pixel group, 96, 96B…addition processing unit, 102, 102B…high-frequency extraction unit, 120…distance image sensor, 128, 128D…point group data generation unit, 134…three-dimensional position calculation unit, 140…spectrographic element, 200, 200A…endoscope system, 202, 202A…endoscope, 204, 204A…image processing device, 210…front end portion, 212…insertion portion, 220…shooting unit, 230…transmission cable.

Claims

1. A photographing device, comprising: A polarization image sensor having pixel groups arranged two-dimensionally, and an RGB color filter arranged in a Bayer pattern for each pixel group, the pixel group including 2×2 pixels for detecting four polarization components that differ for each pixel; an interpolation processing unit that separates the pixel values ​​output from the polarization image sensor according to each polarization component to generate four Bayer array images corresponding to the four polarization components, and calculates the RGB value of each of the four polarization components for each pixel by performing debayering and up-conversion on the four Bayer array images; a brightness calculation unit, which calculates the brightness value of each of the four polarization components for each pixel based on the RGB values ​​of each of the four polarization components; a sorting processing unit, for each pixel, sorting the four polarization components according to the order of the brightness values ​​of the four polarization components; A reference generating unit that calculates, for each pixel, a reference brightness value obtained by synthesizing brightness values ​​of a plurality of polarization components including at least the second and third digits; as well as a synthesis processing unit that outputs a reference RGB value obtained by synthesizing the RGB values ​​of the plurality of polarization components when a) the luminance value matches a predetermined threshold value; When b) the reference brightness value is less than the threshold value, the composite RGB value obtained by synthesizing the upper RGB value and the reference RGB value is output, and the upper RGB value is obtained by synthesizing the RGB values ​​of the first and second polarization components; when c) the reference brightness value is less than the threshold value, the composite RGB value obtained by synthesizing the lower RGB value and the reference RGB value is output, and the lower RGB value is obtained by synthesizing the RGB values ​​of the third and fourth polarization components.

2. The photographing device according to claim 1, wherein: The synthesis processing unit b) for pixels whose reference luminance value is less than the threshold, increasing the synthesis weight of the upper RGB value as the reference luminance value becomes smaller, c) For pixels whose reference luminance value is greater than the threshold value, the synthesis weight of the lower RGB values ​​is increased as the reference luminance value becomes larger.

3. The photographing device according to claim 1 or 2, further comprising: a non-polarization image sensor having two-dimensionally arranged pixels having a greater number of pixels than the number of arranged pixels of the pixel groups of the polarization image sensor; a light splitting element, which splits the incident light into light directed toward the polarized image sensor and light directed toward the non-polarized image sensor; a high-frequency extraction unit that extracts a high-frequency component of each pixel using an output image of the image sensor; as well as The addition processing section adds the high-frequency component to the RGB value output from the synthesis processing section for each pixel, thereby generating a high-resolution image.

4. A photographing device comprising: A polarization image sensor having pixel groups arranged two-dimensionally and an RGB color filter arranged in a Bayer pattern for each pixel group, wherein the pixel group includes 2×2 pixels for detecting four polarization components that differ for each pixel; a non-polarization image sensor having two-dimensionally arranged pixels having a greater number of pixels than the number of arranged pixels of the pixel groups of the polarization image sensor; a light splitting element, which splits the incident light into light directed toward the polarized image sensor and light directed toward the non-polarized image sensor; an interpolation processing unit that separates the pixel values ​​output from the polarization image sensor according to each polarization component to generate four Bayer array images corresponding to the four polarization components, and calculates the RGB value of each of the four polarization components for each pixel by performing debayering and up-conversion on the four Bayer array images; a high-frequency extraction unit that extracts a high-frequency component of each pixel using an output image of the non-polarized image sensor; as well as The addition processing unit adds the high-frequency component to the RGB values ​​of the four polarization components output from the interpolation processing unit for each pixel, thereby generating four high-resolution images corresponding to the four polarization components, respectively.

5. The photographing device according to claim 3 or 4, wherein: The non-polarized image sensor is a color image sensor in which RGB color filters are arranged in a Bayer pattern for each pixel. The high-frequency extraction unit extracts high-frequency components from the RGB value of each pixel calculated by debayering the Bayer array image output by the non-polarization image sensor. The addition processing unit adds the RGB value representing the high-frequency component and the RGB value of each of the four polarization components for each pixel.

6. The photographing device according to claim 3 or 4, wherein: The non-polarized image sensor is a monochrome image sensor without a color filter, The high-frequency extraction unit extracts a high-frequency component from the brightness value of each pixel output by the monochrome image sensor. The addition processing unit adds the luminance value indicating the high-frequency component to each of the RGB values ​​of the four polarization components for each pixel.

7. The photographing device according to any one of claims 3 to 6, further comprising: Distance image sensor, measures the distance to the subject, The light splitting element further splits the incident light into light directed toward the distance image sensor. The shooting device also includes: a three-dimensional position calculation unit that calculates a coordinate value indicating the three-dimensional position of the object for each pixel using an output signal of the range image sensor; and The point cloud data generating unit generates point cloud data in which the RGB values ​​of the four high-resolution images output by the adding unit and the coordinate values ​​calculated by the three-dimensional position calculating unit are associated with each other for each pixel.

8. The photographing device according to claim 1 or 2, further comprising: Distance image sensor, measuring the distance to the subject; a light splitting element, which splits the incident light into light directed toward the polarization image sensor and light directed toward the distance image sensor; a three-dimensional position calculation unit that calculates a coordinate value representing the three-dimensional position of the object for each pixel using an output signal of the range image sensor; as well as The point cloud data generating unit generates point cloud data in which the RGB value output by the synthesis processing unit and the coordinate value calculated by the three-dimensional position calculating unit are associated with each other for each pixel.

9. An endoscope system comprising the photographing device according to claim 1, 2 or 8, the endoscope system comprising: An endoscope comprising an insertion portion having a front end portion facing a subject, the polarization image sensor being disposed within the front end portion, and a transmission cable for transmitting an output signal of the polarization image sensor being disposed within the insertion portion; as well as An image processing device includes the interpolation processing section, the brightness calculation section, the sorting processing section, the reference generation section, and the synthesis processing section, and acquires the output signal via the transmission cable.

10. An endoscope system comprising the photographing device according to any one of claims 3 to 7, the endoscope system comprising: an endoscope comprising an insertion portion having a front end portion facing a subject, the polarization image sensor, the non-polarization image sensor, and the spectroscopic element being disposed within the front end portion, and a transmission cable for transmitting output signals of the polarization image sensor and the non-polarization image sensor being disposed within the insertion portion; and The image processing device includes the interpolation processing unit, the high-frequency extraction unit, and the addition processing unit, and acquires the output signal via the transmission cable.

Images