Image processing device, camera device, image processing method, and recording medium

Abstract

The present invention provides an image processing device, an imaging device, an image processing method and an image processing program for generating a plurality of polarization image data based on polarization image data acquired from an imaging element. A processor (200B) of the image processing device (200) performs the following processing: an acquisition process for acquiring first image data from an imaging element (100) in which four first polarizers with different polarization directions are regularly arranged in pixels arranged two-dimensionally; a first polarization image data generation process for performing a demosaicing process on the first image data to generate four first polarization image data with different polarization directions; and a second polarization image data generation process for generating four or less second polarization image data using the relationship between the four first polarization image data and the polarization directions of the four first polarizers stored in a memory (200C).

Description

Technical Field

[0001] The present invention relates to an image processing device, an imaging device, an image processing method and an image processing program, and more particularly to a technology for generating a plurality of polarization image data based on polarization image data acquired from an imaging element. Background Art

[0002] The imaging element described in Patent Document 1 includes a plurality of light receiving cells arranged in an array and each detecting light of a specific color and a specific polarization angle. Among the plurality of cells, adjacent cells detect different colors and polarization angles.

[0003] Specifically, a plurality of units are provided with a polarizer (polarization filter) that transmits light having any polarization angle of 0°, 45°, 90°, or 135°, and also provided with any color filter of red, green, or blue (RGB).

[0004] Thereby, RGB images (polarization images) can be generated at polarization angles of 0°, 45°, 90°, and 135°.

[0005] Patent Document 1 describes an image processing unit that demosaic a polarization RAW image to generate a plurality of polarization images at different polarization angles. Here, demosaicing refers to a process of generating polarization images at the same polarization angle from a pixel group captured by an imaging element.

[0006] Previous technical literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application Publication No. 2016-171368 Summary of the invention

[0009] Technical issues to be solved by the invention

[0010] One embodiment of the technology according to the present invention provides an image processing device, an imaging device, an image processing method, and an image processing program for satisfactorily generating a plurality of polarization image data from polarization image data acquired from an imaging element.

[0011] Means for solving technical problems

[0012] The invention involved in the first embodiment is an image processing device, which has a processor and a memory that stores the relationship between polarization directions, and the processor performs image processing, and the processor performs the following processing: acquisition processing, when n is an integer greater than 4, acquiring the first image data from an imaging element in which n first polarizers with different polarization directions are regularly arranged in a plurality of pixels composed of photoelectric conversion elements arranged two-dimensionally; first polarization image data generation processing, de-mosaicing the first image data to generate n first polarization image data corresponding to different polarization directions; and second polarization image data generation processing, using the relationship between n first polarization image data and the polarization direction stored in the memory to generate n or less second polarization image data.

[0013] In the image processing device according to the second aspect of the present invention, preferably, the relationship between the polarization directions is a relationship between a characteristic of the n first polarizers and a physical quantity of a pixel corresponding to the first polarizers.

[0014] In the image processing device according to the third aspect of the present invention, preferably, when n is 4,

[0015] When four polarization image data separated from the first image data for each polarization direction are represented by I1, I2, I3, and I4, respectively, the processor performs the following calculation to generate four demosaiced first polarization image data.

[0016] [Formula 1]

[0017]

[0018] in,

[0019]

[0020] (where a is a parameter)

[0021] In the image processing device according to the fourth aspect of the present invention, preferably, when n is 4, the memory stores the following matrix as the relationship between the polarization directions of the n first polarizers:

[0022] [Formula 2]

[0023]

[0024] (where l1~l3 and c0~c3 are coefficients)

[0025] When the first pixel value originally existing at each pixel position of the n first polarization image data is set to x0, and the three second pixel values ​​generated by demosaicing processing at the pixel position are set to x 1demo 、x2dem o 、x 3demo When the processor calculates the three pixel values ​​(x ) generated at the pixel position for generating the second polarization image data of less than 4 pixels by the following formula: 1r 、x 2r 、x 3r ).

[0026] [Formula 3]

[0027]

[0028] In the image processing device involved in the fifth aspect of the present invention, when four polarization information of adjacent pixels obtained when non-polarized light with uniform light distribution is incident on the imaging element are defined as x0 to x3, coefficients c0 to c3 and polarization information x0 to x3 satisfy the following equation.

[0029] [Formula 4]

[0030]

[0031] As an example, x i The brightness value of the light.

[0032] In the image processing device according to the sixth aspect of the present invention, when the parameter i is set to 1 to 3 and the value of l i The coefficients l1, l2, l3 in the matrix shown in [Formula 2] are represented by c i When the coefficients c1, c2, and c3 are expressed, l i It is expressed by the following formula.

[0033] [Formula 10]

[0034]

[0035] Remove {c0,c1,c2,c3} T A vector of three elements consisting of the i+1th component of

[0036] The matrix that converts the 4 polarization information into the other n physical quantities

[0037] Remove A T A matrix consisting of the i+1th row and j+1th column of A with 3 rows and 3 columns

[0038] ~: cofactor matrix

[0039] (Wherein, when the matrix A is set to be a matrix of m rows and 4 columns that converts the four polarization information obtained from the imaging element into another m physical quantities, the matrix B in the formula [Formula 10] is obtained by removing AT The cofactor matrix of the 3-row and 3-column matrix B consisting of the (i+1)th row and (j+1)th column of A)

[0040] In the image processing device according to the seventh aspect of the present invention, preferably, the matrix A is a unit matrix having 4 rows and 4 columns.

[0041] In the image processing device according to the eighth aspect of the present invention, when the parameter i is set to 1 to 3 and the value of l i The coefficients l1, l2, l3 in the matrix shown in [Formula 2] are represented by c i When the coefficients c1, c2, and c3 are expressed, l i It is expressed by the following formula.

[0042] [Formula 18]

[0043]

[0044] The camera device involved in the 9th mode of the present invention comprises: a photographic optical system; when n is an integer greater than 4, an imaging element, in which n first polarizers with different polarization directions are regularly arranged in a plurality of pixels composed of photoelectric conversion elements arranged two-dimensionally; and any image processing device in the 1st to 8th modes, wherein the processor obtains the first image data from the imaging element.

[0045] The camera device involved in the 10th mode of the present invention comprises: a photographic optical system, including three second polarizers and three wavelength selection elements, the pupil areas of the three second polarizers are divided into a first pupil area, a second pupil area and a third pupil area, and the light passing through the first pupil area, the second pupil area and the third pupil area is polarized in different directions, and the three wavelength selection elements allow light of different wavelength bands to pass through the first pupil area, the second pupil area and the third pupil area; an imaging element, in which four first polarizers with different polarization directions are regularly arranged in a plurality of pixels composed of photoelectric conversion elements arranged two-dimensionally; and an image processing device of the 6th mode, the processor obtains the first image data from the imaging element, and the matrix A is an interference removal matrix when calculating three image data corresponding to the three wavelength bands of light after removing the interference of the three wavelength bands of light based on the first image data.

[0046] The invention involved in the 11th embodiment is an image processing method, which uses a processor and a memory that stores the relationship between polarization directions to perform image processing, and the image processing method includes: a step of obtaining, by the processor, when n is an integer greater than 4, a first image data from an imaging element in which n first polarizers with different polarization directions are regularly arranged in a plurality of pixels composed of photoelectric conversion elements arranged two-dimensionally; a step of de-mosaicing the first image data to generate n first polarization image data with different polarization directions; and a step of generating n or less second polarization image data using the relationship between the n first polarization image data and the polarization directions stored in the memory.

[0047] The image processing program involved in the 12th mode of the present invention implements the following functions through a computer: when n is an integer greater than 4, a function of obtaining the first image data from an imaging element in which n first polarizers with different polarization directions are regularly arranged in a plurality of pixels composed of photoelectric conversion elements arranged two-dimensionally; a function of de-mosaicing the first image data to generate n first polarization image data with different polarization directions; and a function of generating n or less second polarization image data using the relationship between the n first polarization image data and the polarization directions of the n first polarizers. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a diagram showing a schematic configuration of a first embodiment of an imaging device according to the present invention.

[0049] Figure 2 Yes means Figure 1 A diagram showing a schematic structure of an imaging element.

[0050] Figure 3 Yes means Figure 2 The 1 pixel shown ( Figure 2 A cross-sectional view of the schematic structure of the device (the dotted line portion).

[0051] Figure 4 This is a diagram showing an example of an arrangement pattern of polarization filter elements provided in each pixel block of an imaging element.

[0052] Figure 5 Yes means Figure 1 A diagram showing a schematic structure of an arrangement of pixels of an imaging element shown.

[0053] Figure 6 Yes means Figure 1 A block diagram showing a schematic structure of an image processing device shown in FIG.

[0054] Figure 7 1 is a schematic diagram showing a method of separating polarization image data I1 to I4 for each polarization direction from polarization RAW image data.

[0055] Figure 8 1 is a schematic diagram showing a method of generating four first polarization image data by performing convolution with kernel K on each of the polarization image data I1 to I4 to interpolate pixel values ​​of black portions.

[0056] Fig. 9 The diagram shows eight adjacent pixels (nine pixels in total) centered around the pixel P1.

[0057] Fig.10 It is a schematic diagram showing the relationship between the pixel values ​​of four pixels P1, P2, P3, and P4 with different polarization directions.

[0058] Fig.11 It is a diagram showing a schematic configuration of a second embodiment of an imaging device according to the present invention.

[0059] Fig.12 Yes means Fig.11 A front view of an embodiment of a bandpass filter unit is shown.

[0060] Fig.13 This is a graph showing an example of the transmission wavelength characteristics of each bandpass filter.

[0061] Fig.14 Yes means Fig.11 A front view of an embodiment of a polarization filter unit is shown. DETAILED DESCRIPTION

[0062] Hereinafter, preferred embodiments of an image processing device, an imaging device, an image processing method, and an image processing program according to the present invention will be described with reference to the accompanying drawings.

[0063] <First embodiment>

[0064] [Structure of the imaging device]

[0065] Figure 1 It is a diagram showing a schematic configuration of a first embodiment of an imaging device according to the present invention.

[0066] The imaging device 1 - 1 of the first embodiment is a polarization camera that captures four polarization images having mutually different polarization directions, and includes a photographing optical system 10 - 1 , an imaging element 100 , and an image processing device 200 .

[0067] The photographing optical system 10 - 1 is a general photographing lens that forms an optical image of a subject on the light receiving surface of the imaging element 100 .

[0068] Furthermore, the photographing optical system 10-1 has a focus adjustment mechanism and an aperture (not shown). The focus adjustment mechanism adjusts the focus by moving the entire photographing optical system 10-1 forward and backward along the optical axis L, for example.

[0069] <Imaging element>

[0070] Figure 2 Yes means Figure 1 A diagram showing a schematic structure of an imaging element. Figure 3 Yes means Figure 2 The 1 pixel shown ( Figure 2 A cross-sectional view of the schematic structure of the device (the dotted line portion).

[0071] The imaging element 100 is a CMOS (Complementary Metal-Oxide Semiconductor) type image sensor, which has a pixel array layer 110, a polarization filter element array layer 120, and a microlens array layer 140. The layers are sequentially arranged from the image plane side toward the object side with the pixel array layer 110, the polarization filter element array layer 120, and the microlens array layer 140. In addition, the imaging element 100 is not limited to the CMOS type, and can also be an XY address type or CCD (Charge Coupled Device) type image sensor.

[0072] The pixel array layer 110 is formed by two-dimensionally arranging a plurality of photodiodes 112 as photoelectric conversion elements. One photodiode 112 forms one pixel. The photodiodes 112 are regularly arranged in the horizontal direction (x-axis direction) and the vertical direction (y-axis direction).

[0073] The polarization filter array layer 120 is provided with n first polarizers (polarization filter elements 122A, 122B, 122C, and 122D) having different polarization directions of transmitted light in a regular pattern. Here, n is an integer greater than or equal to 4, and in this example, n=4 is shown.

[0074] Figure 4 This is a diagram showing an example of an arrangement pattern of polarization filter elements provided in each pixel block of an imaging element.

[0075] like Figure 4 As shown, a pixel block PB (X, Y) is formed by four adjacent (2×2) pixels P1, P2, P3, and P4, and the above-mentioned polarization filter elements 122A, 122B, 122C, and 122D are respectively provided in each pixel P1, P2, P3, and P4 in the pixel block PB.

[0076] The polarization directions of the four polarization filter elements 122A, 122B, 122C, and 122D differ from each other by 45°.

[0077] For example, the polarization filter element 122A allows only the incident light Figure 4 When the light component in the left-right direction (x-axis direction) is transmitted, when the polarization direction θ1 at this time is set to 0°, the polarization direction θ2 of the polarization filter element 122B is 45°, the polarization direction θ3 of the polarization filter element 122C is 90°, and the polarization direction θ4 of the polarization filter element 122D is 135°. In addition, the arrangement pattern of the four polarization filter elements 122A, 122B, 122C, and 122D is not limited to Figure 4 Example shown.

[0078] Figure 5 Yes means Figure 1 A diagram showing a schematic structure of an arrangement of pixels of an imaging element shown.

[0079] like Figure 5 As shown, the imaging element 100 has four types of pixels P1 to P4 on the light receiving surface as described above. Four adjacent (2×2) pixel blocks PB (X, Y) are regularly arranged in the horizontal direction (x-axis direction) and the vertical direction (y-axis direction).

[0080] When the imaging device 1-1 receives a photographing command input from a release switch or the like, it performs exposure control in the imaging element 100. Through this exposure control, the optical image of the subject formed on the light receiving surface of the imaging element 100 is converted into an electrical signal by the imaging element 100. Charges corresponding to the amount of light incident on the photodiode 112 are accumulated in each pixel of the imaging element 100, and electrical signals corresponding to the amount of charge accumulated in each pixel are read from the imaging element 100 as image signals (first image data).

[0081] Here, in this example, the first image data is image data having four polarization information for each of pixels P1 to P4. Hereinafter, the first image data will be referred to as "polarization RAW image data".

[0082] Return to Figure 1 The image processing device 200 acquires polarization RAW image data from the imaging element 100 and performs image processing (RAW development processing) described later to generate four polarization image data (second polarization image data) for each polarization direction.

[0083] <Image processing device>

[0084] Figure 6 Yes means Figure 1 A block diagram showing a schematic structure of an image processing device shown in FIG.

[0085] like Figure 6 As shown, the image processing apparatus 200 includes an analog signal processing unit 200A, a processor 200B and a memory 200C.

[0086] The analog signal processing unit 200A reads the analog pixel signal output from the imaging element 100 , performs predetermined signal processing (eg, correlated double sampling processing, amplification processing, etc.), converts the analog pixel signal into a digital signal, and outputs the digital signal.

[0087] The processor 200B performs an acquisition process of acquiring polarization RAW image data from the analog signal processing unit 200A. In addition, when the imaging element 100 is a CMOS type imaging element, the analog signal processing unit and the like are usually built into the CMOS type imaging element. In this case, the processor 200B can acquire polarization RAW image data from the CMOS type imaging element.

[0088] The processor 200B performs a demosaic process on the acquired polarization RAW image data to generate four polarization image data (first polarization image data) having different polarization directions.

[0089] <<De-mosaic Processing>>

[0090] Next, demosaic processing of polarization RAW image data will be described.

[0091] When four polarization image data separated by polarization direction from the polarization RAW image data are represented by I1, I2, I3, and I4, respectively, the processor 200B performs the following calculation to perform first polarization image data generation processing to generate four demosaiced first polarization image data.

[0092] [Formula 1]

[0093]

[0094] in,

[0095]

[0096] (where a is a parameter)

[0097] Polarized image data I1 among the four polarized image data (I1, I2, I3, I4) is image data having only the pixel value of pixel P1 and the pixel values ​​of other pixel positions are zero. Similarly, polarized image data I2 is image data having only the pixel value of pixel P2 and the pixel values ​​of other pixel positions are zero, polarized image data I3 is image data having only the pixel value of pixel P3 and the pixel values ​​of other pixel positions are zero, and polarized image data I4 is image data having only the pixel value of pixel P4 and the pixel values ​​of other pixel positions are zero.

[0098] Formula [Formula 1] represents a de-mosaicing process in which the pixel value at the pixel position with a zero value is estimated (interpolated) based on the pixel values ​​of the surrounding pixels of the pixel at the pixel position with a zero value by performing a convolution operation (convolution) on each polarization image data I1~I4 and a 7×7 kernel K with a weighting coefficient.

[0099] Regarding the coefficient of the kernel K, a larger weighting coefficient is assigned to a pixel position closer to the pixel position of the estimated pixel value.

[0100] Furthermore, a in the kernel K is a parameter, and the sharpness can be controlled by adjusting the parameter a. For example, when the parameter a is close to 0, the sharpness is small and the noise is small, and when the parameter a is a large negative value, the sharpness is high and the noise is large.

[0101] In addition, the size of the kernel K, the weighting coefficient, and the parameter a shown in the formula [Formula 1] are only examples, and the present invention is not limited to them.

[0102] Figure 7 Schematic diagram showing a method of separating polarization image data I1 to I4 for each polarization direction from polarization RAW image data. By separating polarization image data I1 to I4 for each polarization direction, subsequent processing becomes easier.

[0103] exist Figure 7 In the polarization image data I1 to I4 shown, the pixel value of the black portion is 0. The demosaicing process is a process of estimating (interpolating) the pixel value of the black portion from the pixel values ​​of the surrounding pixels of the black portion, embedding the interpolated pixel value in the black portion, and generating four first polarization image data.

[0104] Figure 8 1 is a schematic diagram showing a method of generating four first polarization image data by performing convolution with kernel K on each of the polarization image data I1 to I4 to interpolate pixel values ​​of black portions.

[0105] The kernel K is stored in the memory 200C, for example. The processor 200B reads the kernel K from the memory 200C and performs a demosaicing process for generating four first polarization image data by convolution with the kernel K for each of the polarization image data I1 to I4.

[0106] In the above-mentioned demosaicing process, according to Figure 4 and Figure 5 The spatial arrangement relationship of the pixels P1 to P4 of the four first polarizers having different polarization directions is shown in FIG. 1 , and a predetermined pixel position ( ) is generated for each polarization image data I1 to I4 by interpolation processing. Figure 8 The pixel value at the pixel position (the black part shown).

[0107] <<Method for generating second polarization image data>>

[0108] Next, a method for generating n or less second polarization image data having higher interpolation accuracy than the first polarization image data using the relationship between the n first polarization image data and the polarization directions of the n first polarizers will be described. The relationship between the polarization directions of the n first polarizers will be described later.

[0109] For the sake of convenience, the following description will be made on the case where n is 4.

[0110] When n is 4, Figure 6 The memory 200C shown stores the following matrix as the relationship between the polarization directions of n (4) first polarizers having different polarization directions.

[0111] [Formula 2]

[0112]

[0113] (where l1~l3 and c0~c3 are coefficients)

[0114] Furthermore, when n is 4, four first polarization image data are generated by demosaicing processing.

[0115] At the same pixel position of the four first polarization image data, there are an original pixel value (hereinafter referred to as “first pixel value x0”) and three pixel values ​​generated by demosaicing processing (hereinafter referred to as “second pixel value (x 1demo 、x 2demo 、x 3demo )”).

[0116] Figure 6 The processor 200B shown in FIG. 200B calculates the polarization direction of the four first polarizers according to the relationship between the polarization directions of the four first polarizers (the matrix of [Formula 2]) and the four pixel values ​​(the first pixel value (x0), the three second pixel values ​​(x1), and the third pixel value (x2)) at each pixel position stored in the memory 200C. 1demo 、x 2demo 、x 3demo )) Perform the following operation:

[0117] [Formula 3]

[0118]

[0119] The calculation corrects the three second pixel values ​​(x 1dem o 、x 2demo 、x 3demo ) and the three pixel values ​​(x 1r 、x 2r 、x 3r ).

[0120] Thus, the processor 200B generates four or less (four in this example) second polarization image data (I ) having an interpolation accuracy higher than the four first polarization image data subjected to the demosaicing process. 1r ,I 2r ,I 3r ,I 4r )'s second polarization image data generation process.

[0121] The four second polarization image data (I 1r ,I 2r ,I 3r ,I 4r ) can be output to a display device or recording device (not shown) described later for display or recording.

[0122] <<Relationship between polarization directions>>

[0123] The relationship between the polarization directions of the n first polarizers may be the relationship between the characteristics of the n first polarizers and the physical quantity of the pixel corresponding to the first polarizer. As an example, the relationship between the polarization directions of the first polarizers when n=4 shown in the matrix of [Formula 2] is described.

[0124] When non-polarized light (for example, natural light) with uniform light quantity distribution in the plane is incident on the imaging element 100, Figure 4 and Figure 5 When the four polarization information (pixel values) obtained from the four adjacent (2×2) pixels P1, P2, P3, and P4 shown are set to x0~x3, the coefficients c0~c3 in the matrix shown in [Formula 3] and the pixel values ​​x0~x3 satisfy the following formula.

[0125] [Formula 4]

[0126]

[0127] In other words, there are coefficients c0 to c3 that satisfy the formula [Formula 4].

[0128] Here, regarding uniformity, a distribution (brightness difference) within the range of correctable pixel values ​​is allowed. As an example, it is confirmed to be 20% or less. In addition, non-polarized light means mixed polarized light in multiple directions, rather than polarized light in a single direction.

[0129] Fig. 9 The diagram shows eight adjacent pixels (nine pixels in total) centered around the pixel P1. Fig.10 It is a schematic diagram showing the relationship between the pixel values ​​of four pixels P1, P2, P3, and P4 with different polarization directions.

[0130] like Fig.10 As shown, when the interval of the polarization direction is divided into 4 equal parts (45° intervals), the c in formula [Formula 4] is i c0=c2=1, c1=c3=-1.

[0131] However, in reality, the above relationship may slightly deviate due to the influence of manufacturing errors of the imaging element 100 and the like.

[0132] Therefore, an overall full white input (uniform non-polarized light) is made incident on the imaging element 100 , and calibration is performed using the output result of the imaging element 100 .

[0133] In an image that is completely white, there is no sudden change, so we only need to check the relationship between adjacent pixels. Then, we determine the c that satisfies the formula [Formula 4] i . i The value after constant multiplication also satisfies the same condition, so c i Therefore, by setting c i The sum of c is 1. i .

[0134] Formula [Formula 4] is the four pixel values ​​(x 0dem o 、x 1demo 、x 2demo 、x 3demo ) also holds true between pixel values ​​(x 0demo 、x 1demo 、x 2demo 、x 3demo ) and the coefficients c0~c3 can be expressed by the following formula.

[0135] [Formula 5]

[0136] c0x 0demo +c1x 1demo +c2x 2demo +c3x 3demo =0

[0137] The following formula [Formula 5'] is a formula obtained by transforming the above formula [Formula 5].

[0138] [Formula 5']

[0139] x 0demo =-(1 / c0)(c1x 1demo +c2x 2demo +c3x 3demo )

[0140] Furthermore, in a pixel of a certain polarization direction (for example, pixel P1 with a polarization direction of 0°) among the four pixels P1, P2, P3, and P4 with different polarization directions, the error between the originally existing first pixel value and the second pixel value (pixel value of the same polarization direction) inferred through de-mosaicing processing is set to δ.

[0141] Similarly, in the three pixels whose polarization directions are offset by 45° from the pixels of one of the four pixels P1, P2, P3, and P4 (for example, pixels P2, P3, and P4 with polarization directions of 45°, 90°, and 135°), the errors between the originally existing first pixel value and the second pixel value inferred through de-mosaicing are set to l1δ, l2δ, and l3δ, respectively.

[0142] The coefficients l1, l2, and l3 are the same as l1, l2, and l3 in the matrix of [Formula 2]. The calculation method of the coefficients l1, l2, and l3 will be described later.

[0143] At present, at the pixel position of pixel P1, when the true value (the pixel value to be obtained) of pixel P2 whose polarization direction differs from pixel P1 by 45° is set to x 1r When the pixel value x 1r It can be expressed by the following formula.

[0144] [Formula 6]

[0145] x 1r =x 1demo +l1δ

[0146] The error δ in the above equation [Equation 6] is the difference between the original pixel value x0 of the pixel P1 and the pixel value x estimated by the demosaicing process. 0demo The error between them can be expressed by the following formula.

[0147] [Formula 7]

[0148] δ=x0-x 0demo

[0149] If the right side of Formula 7 is substituted into δ of Formula 6, Formula 6 can be expressed as follows.

[0150] [Formula 8]

[0151] x 1r =x 1demo +l1(x0-x 0demo )

[0152] Furthermore, if we substitute the right side of Formula 5' into the x of Formula 8, 0demo After being rearranged in , the formula [Formula 8] can be expressed as follows.

[0153] [Formula 9]

[0154] x 1r =l1x0+{(1+l1c1 / c0)x 1demo +(l1c2 / c0)x 2demo +(l1c3 / c0)x 3demo}

[0155] The above formula [Formula 9] is equivalent to calculating x in formula [Formula 3] 1r of numbers.

[0156] Furthermore, at the pixel position of pixel P1, the pixel value x of pixel P3 whose polarization direction differs from that of pixel P1 by 90° can also be calculated in the same manner. 2r (pixel value without error l2δ) and pixel value x of pixel P4 whose polarization direction differs from pixel P1 by 135° 3r (pixel value without error l3δ).

[0157] And, Formula (3) is the value of these pixels without errors (x 1r 、x 2r 、x 3r ), the matrix on the right side of [Formula 3] is the matrix shown in [Formula 2].

[0158] In addition, the coefficients l1 to l3 shown in Formula 2 and Formula 3 can be 1 respectively.

[0159] <<Calculation method of coefficients l1, l2, l3>>

[0160] When l i The coefficients l1, l2, l3 in the matrix shown in [Formula 2] are represented by c i When the coefficients c1, c2, and c3 are expressed, l i It can be expressed by the following formula.

[0161] [Formula 10]

[0162]

[0163] Remove {c0,c1,c2,c3} T A vector of three elements consisting of the i+1th component of

[0164] The matrix that converts the 4 polarization information into the other n physical quantities Remove A T A matrix consisting of the i+1th row and j+1th column of A with 3 rows and 3 columns

[0165] ~: cofactor matrix

[0166] (Wherein, when the matrix A is set to be a matrix of m rows and 4 columns that provides the conversion of the 4 polarization information obtained from the imaging element into the other 4 physical quantities, the matrix B in the formula [Formula 10] is obtained by removing A T The cofactor matrix of the 3-row and 3-column matrix B consisting of the (i+1)th row and (j+1)th column of A).

[0167] Currently, matrix A is set to the unit matrix shown in [Formula 11].

[0168] [Formula 11]

[0169]

[0170] A T A can be represented by the matrix shown in [Formula 12].

[0171] [Formula 12]

[0172]

[0173] Remove A T The matrix B is formed by the (i+1)th row and (j+1)th column of A. i , j can be expressed by [Formula 13].

[0174] [Formula 13]

[0175]

[0176] The cofactor matrix B in formula [Formula 10] i,j The matrix B based on formula [Formula 13] can be expressed as follows.

[0177] [Formula 14]

[0178]

[0179] And, c i Since it is a vector excluding the (i+1)th component, it can be expressed by the following formula.

[0180] [Formula 15]

[0181]

[0182] Therefore, the coefficient l1 of the formula [Formula 10] can be expressed by the following formula.

[0183] [Formula 16]

[0184]

[0185] Similarly, the coefficients l2 and l3 of the formula [Formula 10] can also be expressed by the following formulas respectively.

[0186] [Formula 17]

[0187]

[0188] Therefore, the coefficients l1 to l3 of the formula [Formula 10] can be expressed by the following formula.

[0189] [Formula 18]

[0190]

[0191] Furthermore, the polarization directions (θ1 to θ4) of the four polarization filter elements 122A to 122D provided in the imaging element 100 of this example are 0°, 45°, 90° and 135°, which are obtained by dividing 180° into n (=4) equal parts and are staggered at 45° scales, but the polarization directions are not limited to this.

[0192] Next, the coefficient c under the cos square model i The export of .

[0193] When n is an integer greater than or equal to 4, the formula [Formula 4] can be represented by the following formula.

[0194] [Formula 19]

[0195]

[0196] In formula [Formula 19], φ i It is an angle representing the polarization directions of a plurality of mutually different pixels of the imaging element.

[0197] In order for the formula [Formula 19] to hold true at any θ, the following formula must be satisfied.

[0198] [Formula 20]

[0199]

[0200] Since the variables are relatively lengthy, for example, if c0=1, the formula [Formula 20] can be rewritten as the following formula.

[0201] [Formula 21]

[0202]

[0203] The formula [Formula 21] can be expressed as follows.

[0204] [Formula 22]

[0205]

[0206] Here,

[0207] [Formula 23]

[0208] If ΦC=-φ0

[0209] Φ={φ1 φ2…φ n}

[0210]

[0211] , so that ||C||→min, it can be expressed as

[0212] C=-Φ T (ΦΦ T ) -1 φ0.

[0213] For example, if φ0=0°, φ1=45°, φ2=90°, φ4=135°, then c0=1, c1=-1, c2=1, c3=-1; if φ0=0°, φ1=10°, φ2=30°, φ4=60°, then c0=1, c1=-1.6527, c2=0.8794, c3=-0.2267.

[0214] <Second embodiment>

[0215] Fig.11 It is a diagram showing a schematic configuration of a second embodiment of an imaging device according to the present invention.

[0216] The imaging device 1 - 2 of the second embodiment is a multispectral camera that captures three or more images (multispectral images) in different wavelength bands, and includes a photographic optical system 10 - 2 , an imaging element 100 , and an image processing device 200 - 2 .

[0217] In addition, Fig.11 In, with Figure 1 The same parts as those of the imaging device 1-1 of the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted. Figure 6 The image processing device 200 shown is the same, but the coefficients used for the calculations are different.

[0218] The imaging device 1 - 2 of the second embodiment is different from the imaging device 1 - 1 of the first embodiment mainly in that an imaging optical system 10 - 2 is different from the imaging optical system 10 - 1 of the imaging device 1 - 1 of the first embodiment.

[0219] <<Photographic optical system>>

[0220] The photographic optical system 10-2 is composed of a plurality of lenses 12. The photographic optical system 10-2 has a bandpass filter unit 16 and a polarization filter unit 18 near its pupil. In addition, the photographic optical system 10-2 has a focus adjustment mechanism (not shown). The focus adjustment mechanism adjusts the focus by moving the entire photographic optical system 10-2 forward and backward along the optical axis L, for example.

[0221] The pupil area of ​​the imaging optical system 10 - 2 is divided into a first pupil area, a second pupil area, and a third pupil area. The bandpass filter unit 16 has three wavelength selection elements that allow light of different wavelength bands to pass through the first pupil area, the second pupil area, and the third pupil area.

[0222] Fig.12 Yes means Fig.11 A front view of an embodiment of a bandpass filter unit is shown.

[0223] Fig.12 The bandpass filter unit 16 shown has a first pupil area Z1 that allows light of the first band λ1 to pass through, a second pupil area Z2 that allows light of the second band λ2 to pass through, and a third pupil area Z3 that allows light of the third band λ3 to pass through, and functions as a wavelength selection unit that allows light of different bands to pass through the first pupil area Z1, the second pupil area Z2, and the third pupil area Z3.

[0224] That is, the bandpass filter unit 16 is configured so that its center is consistent with the optical axis L of the photographic optical system 10-2, and is composed of a light-shielding frame 16A having three fan-shaped opening areas 16A1, 16A2, and 16A3 with a central angle of 90° and three wavelength selection elements, namely, bandpass filters 16B1, 16B2, and 16B3, which are arranged in each opening area 16A1, 16A2, and 16A3 of the frame 16A.

[0225] Fig.13 This is a graph showing an example of the transmission wavelength characteristics of each bandpass filter.

[0226] Each of the bandpass filters 16B1 to 16B3 transmits light of a different wavelength band. Specifically, the bandpass filter 16B1 transmits light of the wavelength band λ1. The bandpass filter 16B2 transmits light of the wavelength band λ2. The bandpass filter 16B3 transmits light of the wavelength band λ3.

[0227] Furthermore, the polarization filter unit 18 includes three second polarizers for polarizing the light passing through the pupil-divided first pupil region Z1 , the second pupil region Z2 , and the third pupil region Z3 in mutually different directions.

[0228] Fig.14 Yes means Fig.11A front view of an embodiment of a polarization filter unit is shown.

[0229] and Fig.11 The bandpass filter unit 16 shown is identical to Fig.14 The polarization filter unit 18 shown is configured so that its center is consistent with the optical axis L of the photographic optical system 10-2, and is composed of a light-shielding frame 18A having three fan-shaped opening areas 18A1, 18A2, and 18A3 with a central angle of 90° and three second polarizers (polarization filters 18B1, 18B2, and 18B3) that polarize light passing through the first pupil area Z1, the second pupil area Z2, and the third pupil area Z3 in different directions.

[0230] The bandpass filter unit 16 and the polarization filter unit 18 are arranged close to each other, and the projection shapes of the first pupil area Z1, the second pupil area Z2, and the third pupil area Z3 of each are the same. Therefore, the light that has passed through the bandpass filter 16B1 of the bandpass filter unit 16 passes only through the polarization filter 18B1 of the polarization filter unit 18, and similarly, the light that has passed through the bandpass filter 16B2 passes only through the polarization filter 18B2, and the light that has passed through the bandpass filter 16B3 passes only through the polarization filter 18B3.

[0231] Furthermore, the polarization directions (θ1, θ2, θ3) of the polarization filters 18B1, 18B2, 18B3 of the polarization filter unit 18 of this example are the same as the polarization directions (θ1, θ2, θ3) of the polarization filter elements 122A, 122B, 122C of the pixels P1, P2, P3 arranged in the imaging element 100.

[0232] Therefore, the light with polarization direction θ1 of band λ1 passing through the first pupil region Z1 is incident on the pixel P1 of the imaging element 100 with the same polarization direction θ1 and on the photodiodes 112 of the pixels P2 and P4 whose polarization directions differ from the polarization direction θ1 by ±45°. Similarly, the light with polarization direction θ2 of band λ2 passing through the second pupil region Z2 is incident on the pixel P2 of the imaging element 100 with the same polarization direction θ2 and on the photodiodes 112 of the pixels P3 and P1 whose polarization directions differ from the polarization direction θ2 by ±45°. And the light with polarization direction θ3 of band λ3 passing through the third pupil region Z3 is incident on the pixel P3 of the imaging element 100 with the same polarization direction θ3 and on the photodiodes 112 of the pixels P2 and P4 whose polarization directions differ from the polarization direction θ3 by ±45°.

[0233] Light with polarization direction θ1 in band λ1, light with polarization direction θ2 in band λ2, and light with polarization direction θ3 in band λ3 passing through the first pupil area Z1, the second pupil area Z2, and the third pupil area Z3 of the photographic optical system 10-2, respectively, are incident on the photodiode of pixel P1 of the imaging element 100.

[0234] That is, the light in each wavelength band λ1 to λ3 is incident on (interferes with) the photodiode of the pixel P1 of the imaging element 100. Similarly, the light in each wavelength band λ1 to λ3 is also incident on the photodiode of the pixels P2 to P4 of the imaging element 100.

[0235] Therefore, the image processing device 200-2 ( Fig.11 ) performs interference removal processing to generate image signals in each band λ1~λ3.

[0236] When noise removal is performed, the three second pixel values ​​(x ) generated by demosaicing are corrected at each pixel position by performing the calculation shown in Formula 3. 1demo 、x 2demo 、x 3demo ) and the three pixel values ​​(x 1r 、x 2r 、x 3r In this case, the three pixel values ​​(x 1r 、x 2r 、x 3r ) are the pixel values ​​of each band λ1~λ3.

[0237] Furthermore, when performing interference removal, only one of the three pupil areas Z1, Z2, and Z3 of the bandpass filter unit 16 and the polarization filter unit 18 is opened, the other pupil areas are blocked, and the entire white input is incident on the imaging element 100 to obtain the output result of the imaging element 100. This step is performed for each of the three pupil areas Z1, Z2, and Z3. Using the output result of the imaging element 100 obtained in this way, the interference relationship between adjacent pixels is checked, and c that satisfies the formula [Formula 4] is determined. i .

[0238] Then, the coefficient l shown in formula [Formula 10] is obtained. i The matrix A (the basic matrix used to obtain the cofactor matrix B) is used as the interference removal matrix.

[0239] Figure 6The memory 200C of the image processing device 200 shown in FIG. 2 stores a matrix shown in [Formula 2] when performing interference removal. When performing interference removal, the processor 200B can perform interference removal based on the relationship between the polarization directions of the four first polarizers for interference removal stored in the memory 200C (the matrix of [Formula 2]) and the four pixel values ​​(the first pixel value (x0), the three second pixel values ​​(x1), and the third pixel value (x2)) at each pixel position. 1demo 、x 2demo 、x 3demo )) The calculation shown in Formula 3 is performed to obtain three spectral images (multi-spectral images) corresponding to the respective wavelength bands λ1 to λ3.

[0240] In addition, the three first pupil areas Z1, second pupil areas Z2, and third pupil areas Z3 of the bandpass filter unit 16 and the polarization filter unit 18 of the photographic optical system 10-2 are respectively fan-shaped with a central angle of 90°, but may also be fan-shaped with a central angle of 120°. Furthermore, the three pupil areas are not limited to fan-shaped, and may also be three pupil areas composed of a circle and two annular bands centered on the optical axis L.

[0241] [other]

[0242] The imaging element 100 of this embodiment does not have a color filter, but a red filter, a green filter, or a blue filter may be repeatedly arranged in each pixel in a Bayer arrangement, for example, for each 2×2 pixel block PB. In this way, multiple polarization images with different polarization directions can be obtained for each color (red, green, and blue), and by setting a bandpass filter that selects 3 or less bands in each band of the red filter, the green filter, or the blue filter, a maximum of 9 (=3×3) multispectral images can be captured.

[0243] Furthermore, the present invention is not limited to the camera device, and for example also includes Figure 6 The image processing device 200 and Figure 6 The image processing method and image processing program of the image processing device 200 shown.

[0244] In this case, the image processing method includes a processor 200B and a memory 200C, and the processor 200B executes the following steps. That is, the processor 200B executes the steps of acquiring polarization RAW image data (first image data) from the imaging element 100, performing a demosaicing process on the first image data to generate four first polarization image data with different polarization directions, and generating four or less second polarization image data using the relationship between the four first polarization image data and the polarization directions of the four first polarizers stored in the memory 200C.

[0245] The image processing program implements the above-mentioned image processing method through a computer.

[0246] In the present embodiment, for example, the hardware structure of the processing unit of the processor 200B that performs various processes is various processors as shown below. The various processors include commonly used processors that execute software (programs) to perform the functions of various processing units, namely CPUs (Central Processing Units), processors that can change the circuit structure after manufacturing, such as FPGAs (Field Programmable Gate Arrays), namely Programmable Logic Devices (PLDs), and processors that have circuit structures specially designed for performing specific processes, such as ASICs (Application Specific Integrated Circuits), namely dedicated circuits, etc.

[0247] One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a CPU and an FPGA). Furthermore, multiple processing units may be composed of one processor. As an example of multiple processing units composed of one processor, first, there is the following method, such as represented by computers such as clients or servers: one processor is composed of a combination of one or more CPUs and software, and the functions of multiple processing units are performed by the processor. Secondly, there is the following method, such as represented by a system on chip (System On Chip: SoC), etc.: a processor that uses one IC (Integrated Circuit) chip to implement the functions of the entire system including multiple processing units. In this way, various processing units are constructed as a hardware structure using one or more of the above-mentioned various processors.

[0248] Furthermore, more specifically, the hardware structure of these various processors is a circuit (circuitry) formed by combining circuit elements such as semiconductor elements.

[0249] Furthermore, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications can be made without departing from the spirit of the present invention.

[0250] Explanation of symbols

[0251] 1-1, 1-2-camera device, 10-1, 10-2-photographic optical system, 12-lens, 16-bandpass filter unit, 16A-frame, 16A1, 16A2, 16A3-opening area, 16B1, 16B2, 16B3-bandpass filter, 18-polarization filter unit, 18A-frame, 18A1, 18A2, 18A3-opening area, 18B1, 18B2, 18B3-polarization filter, 100-imaging element, 110-pixel array layer, 112-photodiode, 120-polarization filter element array layer, 122A, 122B, 122C, 122D-polarization filter element, 140-microlens array layer, 200-image processing device, 200A-analog signal processing unit, 200B-processor, 200C-memory, L-optical axis.

Claims

1. An image processing device comprising a processor and a memory storing a relationship between polarization directions, wherein the processor performs image processing, The processor performs the following processing: Acquisition processing, when n is an integer greater than or equal to 4, acquiring first image data from an imaging element in which n first polarizers having mutually different polarization directions are regularly arranged in a plurality of pixels constituted by photoelectric conversion elements arranged two-dimensionally; a first polarization image data generation process of performing a demosaic process on the first image data to generate n first polarization image data corresponding to different polarization directions; and a second polarization image data generation process, generating n or less second polarization image data using the relationship between the n first polarization image data and the polarization direction stored in the memory, When n is 4, When the four polarization image data separated from the first image data according to each polarization direction are set to I1, I2, I3, and I4 respectively, the processor performs the following calculation to generate the four first polarization image data after the demosaicing process, [Formula 1] in, Among them, a is a parameter.

2. The image processing device according to claim 1, wherein: The relationship between the polarization directions is a relationship between the characteristics of the n first polarizers and a physical quantity of a pixel corresponding to the first polarizer.

3. The image processing device according to claim 1 or 2, wherein: When n is 4, The memory stores the following matrix as the relationship between the polarization directions of the n first polarizers, [Formula 2] Among them, l1~l3, c0~c3 are coefficients, When the first pixel value originally existing at each pixel position of the n first polarization image data is set to x0, The three second pixel values ​​generated by the demosaicing process at the pixel position are respectively set as x 1demo 、x 2demo 、x 3demo hour, The processor calculates three pixel values ​​x generated at the pixel position for generating four or less second polarization image data by the following formula: 1r 、x 2r 、x 3r , [Formula 3] 4. The image processing device according to claim 3, wherein: When the four pixel values ​​of adjacent pixels obtained when non-polarized light with uniform light distribution is incident on the imaging element are respectively set to x0 to x3, the coefficients c0 to c3 and the four pixel values ​​x0 to x3 satisfy the following equation: [Formula 4] 5. The image processing device according to claim 3, wherein: When the parameter i is set to 1 to 3 and l i The coefficients l1, l2, l3 in the matrix shown in [Formula 2] are represented by c i When the coefficients c1, c2, and c3 are expressed, the l i It is expressed by the following formula: [Formula 10] Remove {c0, c1, c2, c3} T A vector of three elements consisting of the i+1th component of The matrix that converts the 4 polarization information into the other n physical quantities Remove A T A matrix consisting of the i+1th row and j+1th column of A with 3 rows and 3 columns ~: cofactor matrix When the matrix A is set to be a matrix of m rows and 4 columns that converts the four polarization information obtained from the imaging element into the other four physical quantities, the matrix B in the formula [Formula 10] is obtained by removing A. T The cofactor matrix of the 3-row and 3-column matrix B consisting of the (i+1)th row and (j+1)th column of A.

6. The image processing device according to claim 5, wherein: The matrix A is a unit matrix with 4 rows and 4 columns.

7. The image processing device according to claim 3, wherein: When the parameter i is set to 1 to 3, l i The coefficients l1, l2, l3 in the matrix shown in [Formula 2] are represented by c i When the coefficients c1, c2, and c3 are expressed, the l i It is expressed by the following formula: [Formula 18] 8. An image processing device comprising a processor and a memory storing the relationship between polarization directions, wherein the processor performs image processing. The processor performs the following processing: Acquisition processing, when n is an integer greater than or equal to 4, acquiring first image data from an imaging element in which n first polarizers having mutually different polarization directions are regularly arranged in a plurality of pixels constituted by photoelectric conversion elements arranged two-dimensionally; a first polarization image data generation process of performing a demosaic process on the first image data to generate n first polarization image data corresponding to different polarization directions; and a second polarization image data generation process, generating n or less second polarization image data using the relationship between the n first polarization image data and the polarization direction stored in the memory, When n is 4, The memory stores the following matrix as the relationship between the polarization directions of the n first polarizers, [Formula 2] in, l1~l3, c0~c3 are coefficients, When the first pixel value originally existing at each pixel position of the n first polarization image data is set to x0, The three second pixel values ​​generated by the demosaicing process at the pixel position are respectively set as x 1demo 、x 2demo 、x 3demo hour, The processor calculates three pixel values ​​x generated at the pixel position for generating four or less second polarization image data by the following formula: 1r 、x 2r 、x 3r , [Formula 3] 9. A camera device comprising: Photographic optical systems; An imaging element, when n is an integer greater than or equal to 4, wherein n first polarizers having mutually different polarization directions are regularly arranged in a plurality of pixels composed of photoelectric conversion elements arranged two-dimensionally; and The image processing device according to any one of claims 1 to 8, The processor acquires first image data from the imaging element.

10. A camera device comprising: A photographic optical system comprising three second polarizers and three wavelength selection elements, wherein the pupil regions of the three second polarizers are divided into a first pupil region, a second pupil region, and a third pupil region, and light passing through the first pupil region, the second pupil region, and the third pupil region is polarized in mutually different directions, and the three wavelength selection elements allow light of mutually different wavelength bands to pass through the first pupil region, the second pupil region, and the third pupil region; An imaging element, wherein four first polarizers having different polarization directions are regularly arranged in a plurality of pixels composed of photoelectric conversion elements arranged two-dimensionally; and The image processing device according to claim 5, The processor acquires the first image data from the imaging element, The matrix A is an interference removal matrix when three image data corresponding to light of three wavelength bands after removing interference of light of three wavelength bands are calculated based on the first image data.

11. An image processing method, which uses a processor and a memory storing the relationship between polarization directions to perform image processing, the image processing method comprising: By means of the processor, When n is an integer greater than or equal to 4, a step of acquiring first image data from an imaging element in which n first polarizers having mutually different polarization directions are regularly arranged in a plurality of pixels constituted by photoelectric conversion elements arranged two-dimensionally; A step of performing a demosaicing process on the first image data to generate n first polarization image data with different polarization directions; and a step of generating n or less second polarization image data using the relationship between the n first polarization image data and the polarization direction stored in the memory, When n is 4, When the four polarization image data separated from the first image data according to each polarization direction are set to I1, I2, I3, and I4 respectively, the processor performs the following calculation to generate the four first polarization image data after the demosaicing process, [Formula 1] in, Among them, a is a parameter.

12. An image processing method, which uses a processor and a memory storing the relationship between polarization directions to perform image processing, the image processing method comprising: By means of the processor, When n is an integer greater than or equal to 4, a step of acquiring first image data from an imaging element in which n first polarizers having mutually different polarization directions are regularly arranged in a plurality of pixels constituted by photoelectric conversion elements arranged two-dimensionally; A step of performing a demosaicing process on the first image data to generate n first polarization image data with different polarization directions; and a step of generating n or less second polarization image data using the relationship between the n first polarization image data and the polarization direction stored in the memory, When n is 4, The memory stores the following matrix as the relationship between the polarization directions of the n first polarizers, [Formula 2] Among them, l1~l3, c0~c3 are coefficients, When the first pixel value originally existing at each pixel position of the n first polarization image data is set to x0, The three second pixel values ​​generated by the demosaicing process at the pixel position are respectively set as x 1demo 、x 2demo 、x 3demo hour, The processor calculates three pixel values ​​x generated at the pixel position for generating four or less second polarization image data by the following formula: 1r 、x 2r 、x 3r , [Formula 3] 13 . A recording medium, which is a non-transitory recording medium readable by a computer and has recorded thereon a program for causing a computer to execute the image processing method according to claim 11 or 12.

Images