Light field imaging device and light field imaging method

Abstract

The present invention relates to a light field imaging device and a light field imaging method, wherein the light field imaging device comprises a diffraction grating component and a pixel array, wherein the incident wavefront of a scene is received by the diffraction grating component, the incident wavefront is diffracted, and a diffraction wavefront is generated, so that the pixel array receives a diffraction pattern corresponding to the diffraction wavefront, and light field image data is obtained according to the diffraction pattern. Since the diffraction grating has gratings in two directions perpendicular to each other, the angle of the incident wavefront in two spatial directions can be calculated, and its performance will not be reduced due to changes in the polarization state of the incident light, the utilization rate of the pixel unit area is improved, the size is reduced, the power consumption is reduced, the resolution is improved, the light field image data is obtained more efficiently, and the quality of the two-dimensional image is maintained while increasing the depth information.

Description

Technical Field

[0001] The present invention relates to the field of imaging technology, and in particular to a light field imaging device and a light field imaging method. Background Art

[0002] With the development of science and technology, more and more industries are no longer satisfied with the flat information provided by two-dimensional displays, but require a true reflection of the three-dimensional real world. As a result, optical three-dimensional sensing technology has great significance and broad application prospects in more and more fields, especially in industrial inspection, face recognition, smart home, machine vision and virtual reality.

[0003] Exemplary optical 3D sensing technologies include passive 3D sensing and active 3D sensing. Among them, the active solution requires the use of an active light source. Since an additional infrared light source is required, the power consumption of the device is increased, and it is easily affected by ambient light, which limits its use in outdoor environments. Passive solutions include binocular vision and light field recording. When necessary, passive solutions can also use active light sources to improve the effect. However, binocular vision requires obvious stripes and calibration, and the amount of calculation is large, which makes it inefficient. Another exemplary method is light field imaging, such as the light field camera developed by a US company, which can record all optical information within the focal length range when taking pictures. However, the device needs to add a microlens array in front of the sensor to obtain light field information, sacrificing the resolution of the image in the two-dimensional plane direction.

[0004] Therefore, the exemplary optical three-dimensional sensing device has problems such as low resolution in the two-dimensional plane direction, high power consumption and low imaging efficiency due to the need to set up additional light sources, calibration objects or microlens arrays, etc. Summary of the invention

[0005] Based on this, it is necessary to provide a light field imaging device and a light field imaging method that can improve resolution and imaging efficiency and reduce product power consumption.

[0006] In order to achieve the purpose of the present invention, the present invention adopts the following technical solution:

[0007] A light field imaging device, comprising:

[0008] A diffraction grating component is configured to receive an incident wavefront of a scene, diffract the incident wavefront, and generate a diffracted wavefront; the diffraction grating component includes a diffraction grating having at least two mutually perpendicular grating axes and a refractive index that changes periodically in the direction of the grating axis;

[0009] The pixel array is located below the diffraction grating component and is configured to receive a diffraction pattern corresponding to the diffraction wavefront and acquire light field image data according to the diffraction pattern.

[0010] In one embodiment, the diffraction grating is a two-dimensional phase grating.

[0011] In one of the embodiments, the diffraction grating includes a first grating structure and a second grating structure that are periodically arranged;

[0012] The first grating structure includes a first protrusion and a first groove periodically arranged in the grating axis direction, and the second grating structure includes a second protrusion and a second groove periodically arranged in the grating axis direction, the first protrusion is arranged corresponding to the second protrusion, and the first groove is arranged corresponding to the second groove; in the direction perpendicular to the grating axis, the second protrusion is higher than the first protrusion, and the second groove is higher than the first groove.

[0013] In one embodiment, the first protrusion is flush with the second groove.

[0014] In one of the embodiments, in a direction perpendicular to the grating axis, a ratio of a height difference between the first protrusion and the first groove to a height difference between the second protrusion and the first groove is 1:2.

[0015] In one embodiment, the number of grating periods is at least three.

[0016] In one embodiment, the diffraction grating is offset or aligned with the pixel unit.

[0017] In one of the embodiments, the pixel cells in the pixel array have a pixel size along the grating axis, and the pixel size is less than or equal to half the grating period.

[0018] In one embodiment, the incident wavefront includes one or more of ultraviolet light, visible light, near infrared light, infrared band, terahertz and microwave.

[0019] In one embodiment, the pixel array is front-side illuminated or back-side illuminated.

[0020] In one embodiment, the pixel array is located in the near field region of the diffraction grating assembly.

[0021] In one embodiment, the diffraction grating assembly includes a plurality of the diffraction gratings, and the plurality of the diffraction gratings are arranged in a two-dimensional matrix array.

[0022] In one of the embodiments, the diffraction grating assembly further includes a substrate, the substrate includes a spectral filtering material, and the diffraction grating assembly is configured to receive an incident wavefront of a scene, perform spectral filtering and diffraction on the incident wavefront, and generate a diffracted wavefront.

[0023] In one embodiment, the light field imaging device further comprises:

[0024] The filter plate is disposed between the diffraction grating assembly and the pixel array, and is configured to perform spectral filtering on the diffracted wavefront.

[0025] In one embodiment, the light field imaging device further comprises:

[0026] The microlens array is disposed above the pixel array and is configured to focus the diffracted wavefront onto corresponding pixel units.

[0027] In one embodiment, the light field imaging device further comprises:

[0028] The optical control device is arranged in the optical path of the incident wavefront between the scene and the diffraction grating component.

[0029] In one embodiment, the light field imaging device further comprises:

[0030] A light source is configured to project light onto the scene to form the incident wavefront.

[0031] A light field imaging method based on the light field imaging device as described above comprises:

[0032] receiving an incident wavefront of a scene, diffracting the incident wavefront to generate a diffracted wavefront;

[0033] receiving a diffraction pattern corresponding to the diffraction wavefront, and acquiring light field image data according to the diffraction pattern;

[0034] Depth information of the scene is acquired according to the light field image data to construct a three-dimensional scene.

[0035] The above-mentioned light field imaging device includes a diffraction grating component and a pixel array, and receives the incident wavefront of the scene through the diffraction grating component, diffracts the incident wavefront, and generates a diffraction wavefront, so that the pixel array receives the diffraction pattern corresponding to the diffraction wavefront, and obtains light field image data according to the diffraction pattern. Since the diffraction grating has gratings in two directions perpendicular to each other, it can calculate the angle of the incident wavefront in two spatial directions, and its performance will not be reduced due to changes in the polarization state of the incident light. The utilization rate of the pixel unit area is improved, the size is reduced, the power consumption is reduced, the resolution is improved, the light field image data is more efficiently acquired, and the quality of the two-dimensional image can be maintained while increasing the depth information.

[0036] The above-mentioned light field imaging method receives the incident wavefront of the scene, diffracts the incident wavefront to generate a diffracted wavefront, receives the diffraction pattern corresponding to the diffraction wavefront, efficiently obtains high-resolution light field image data according to the diffraction pattern, and obtains the depth information of the scene according to the light field image data, thereby constructing a three-dimensional scene with high resolution and stronger sense of reality, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic structural diagram of a light field imaging device according to one embodiment;

[0038] Figure 2 is a schematic structural diagram of a diffraction grating in one embodiment;

[0039] Figure 3 for Figure 2 The diffracted wavefront in the implementation corresponds to the diffraction pattern of the grating period;

[0040] Figure 4 For the corresponding Figure 3 Schematic diagram of a two-dimensional diffraction grating;

[0041] Figure 5 is a schematic structural diagram of a diffraction grating in another embodiment;

[0042] Figure 6 For the corresponding Figure 5 Schematic diagram of a two-dimensional diffraction grating;

[0043] Figure 7 for Figure 5 The diffracted wavefront in the implementation corresponds to the diffraction pattern of the grating period;

[0044] Figure 8a is a schematic diagram of the structure in which a diffraction grating and a pixel unit are arranged relative to each other in one embodiment;

[0045] Figure 8b is a schematic structural diagram of the relative arrangement of the diffraction grating and the pixel unit in another embodiment;

[0046] Figure 8c is a schematic structural diagram of the relative arrangement of the diffraction grating and the pixel unit in another embodiment;

[0047] Figure 8d is a schematic structural diagram of the relative arrangement of the diffraction grating and the pixel unit in another embodiment;

[0048] Fig. 9 is an array top view of a pixel array in one embodiment;

[0049] Fig.10a is a schematic diagram of an incident wavefront incident on a diffraction grating in one embodiment;

[0050] Fig.10b is a schematic diagram of the lateral displacement of the diffraction pattern corresponding to the change in the angle of the incident light in the x direction;

[0051] Fig.10c is a schematic diagram of the lateral displacement of the diffraction pattern corresponding to the change in the angle of the incident light in the y direction;

[0052] Fig.11a is a schematic diagram of a relative displacement of a pixel unit and a diffraction grating by π / 2 and 3π / 2;

[0053] Fig.11b Schematic diagram of a pixel unit and two diffraction gratings being relatively offset by π / 2 and 3π / 2 respectively;

[0054] Fig.12 is a schematic structural diagram of a light field imaging device according to another embodiment;

[0055] Fig.13 is a schematic structural diagram of a light field imaging device according to another embodiment;

[0056] Fig.14 is a schematic structural diagram of a light field imaging device according to another embodiment;

[0057] Fig.15 A flow chart of a light field imaging method according to an embodiment. DETAILED DESCRIPTION

[0058] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0060] See also Figure 1 , Figure 1 Schematic diagram of the structure of the light field imaging device in this embodiment.

[0061] In this embodiment, the light field imaging device 10 includes a diffraction grating component 11 and a pixel array 12 .

[0062] The diffraction grating component 11 is configured to receive an incident wavefront of a scene, diffract the incident wavefront, and generate a diffracted wavefront; the diffraction grating component 11 includes at least one diffraction grating having at least two mutually perpendicular grating axes and a refractive index that changes periodically in the direction of the grating axis.

[0063] The pixel array 12 is located below the diffraction grating assembly 11 and is configured to receive a diffraction pattern corresponding to the diffraction wavefront and acquire light field image data according to the diffraction pattern.

[0064] In this embodiment, the grating axis is defined as the direction in which the grating is periodically repeated. The diffraction grating assembly 11 includes at least one diffraction grating, each of which has at least two mutually perpendicular grating axes, and the diffraction grating has a periodically repeated grating in at least two mutually perpendicular directions, and the refractive index corresponding to the grating has a periodic change in each grating axis direction. As a result, the diffraction grating has symmetry and presents a refractive index modulation pattern in space, and its cross-sectional profile in the grating axis direction is approximately a step function or a square wave function.

[0065] Specifically, after the incident wavefront passes through the diffraction grating, its amplitude and / or phase is modulated by the diffraction grating to form a diffraction wavefront. The diffraction wavefront forms a diffraction wavefront. When the diffraction wavefront and the diffraction grating are at a specific distance, a diffraction pattern with the same period and shape as the diffraction grating is formed, and the light field intensity distribution along the grating axis is formed. The specific distance is defined as t is the period of the grating, λ is the wavelength of the incident light, and n is the refractive index of the medium in which the light propagates; the type of incident wavefront is not limited and can include ultraviolet light, visible light, or near-infrared light, as well as infrared bands, terahertz, and microwaves. Furthermore, a diffraction pattern / intensity distribution with the same period as the diffraction grating will be at Z T Integer multiples of / 2 appear and are clearest.

[0066] In one embodiment, the diffraction grating is a phase grating, and after the incident wavefront passes through the phase grating, its phase is modulated by the diffraction grating to form a diffraction wavefront; in another embodiment, the diffraction grating is an intensity grating, and after the incident wavefront passes through the intensity grating, its amplitude is modulated by the diffraction grating to form a diffraction wavefront. Optionally, the diffraction grating in this embodiment is a phase grating, and further, it is a two-dimensional phase grating, which has a higher light transmittance and can improve the collection efficiency of the pixel array 12 for light field image data. In one embodiment, the diffraction grating can be a transmission grating or a reflection grating, and optionally, the diffraction grating in this embodiment is a transmission grating.

[0067] In one embodiment, please refer to Figure 2 The diffraction grating 100 includes a first grating structure 101 and a second grating structure 102 which are periodically arranged in the direction of the grating axis G1.

[0068] Specifically, the first grating structure 101 includes a first protrusion 101a and a first groove 101b periodically arranged in the direction of the grating axis G2, and the second grating structure 102 includes a second protrusion 102a and a second groove 102b periodically arranged in the direction of the grating axis G2, the first protrusion 101a is arranged correspondingly to the second protrusion 102a, and the first groove 101b is arranged correspondingly to the second groove 102b; in the direction perpendicular to the grating axis, the second protrusion is higher than the first protrusion, and the second groove is higher than the first groove. Therefore, the diffraction grating 100 has symmetry, and its performance will not be reduced due to the change of the polarization state of the incident light.

[0069] Among them, the number of grating periods is at least three. The shapes of the first protrusion 101a and the second protrusion 102a are not limited, including columns and blocks. The protrusions are made of dielectric materials, including silicon oxide, glass, polymer materials, insulating materials or photoresists; the first groove 101b and the second groove 102b are filled with materials with a refractive index different from the refractive index of the protrusions; optionally, the refractive index of each protrusion of the diffraction grating 100 is higher than the refractive index of the groove filler. In one embodiment, the material of the protrusion is silicon dioxide with a refractive index of about 1.45, and the groove may not be filled with any material, which is equivalent to using air as a filler. It should be noted that the filling ratio of different refractive index parts in the diffraction grating 100 can be different, and the ratio is not limited to 1:1.

[0070] Therefore, in the grating axis direction, due to the refractive index n of the first protrusion 101a 1a Greater than the refractive index n of the first groove 101b 1b There is a predetermined phase difference between the adjacent first protrusions 101a and first grooves 101b, and the diffraction wavefront passing through the first grating structure 101 has a diffraction pattern corresponding to the grating period (see Figure 3 Similarly, there is a predetermined phase difference between the adjacent second protrusions 102a and second grooves 102b, and the diffraction wavefront passing through the second grating structure 102 has a diffraction pattern corresponding to the grating period (see Figure 3 ). By setting the height difference between the protrusion and the groove, the refractive index of the protrusion and the refractive index of the groove filler, the optical path difference between the protrusion and the groove can be preset.

[0071] The height of the first groove 101b is set to h1, the height of the first protrusion 101a is set to h2, the height of the second groove 102b is set to h3, and the height of the second protrusion 102a is set to h4. When expressed in a two-dimensional plane, Figure 4. Among them, h1, h2, h3 and h4 can be different from each other or partially the same. In one embodiment, the ratio of the height difference (h2-h1) of the first protrusion 101a relative to the first groove 101b to the height difference (h4-h1) of the second protrusion 102a relative to the first groove 101b is 1:2. It should be noted that the dielectric materials of different heights are not limited to the aforementioned height ratio relationship.

[0072] The grating period t1 of the first grating structure 101 corresponds to the sum of the widths of the first protrusion 101a and the first groove 102b in the grating axis direction, and the grating period t2 of the second grating structure 102 corresponds to the sum of the widths of the second protrusion 102a and the second groove 102b in the grating axis direction. The grating period t1 and the grating period t2 may be the same or different.

[0073] The grating duty cycle (herein, the duty cycle is defined as the ratio of the width of the protrusion to the grating period) can be set according to actual needs. For example, the first grating structure 101 has a specific duty cycle of 50%, that is, the ratio of the width of the first protrusion 101a to the grating period t1 is 1:2. The duty cycle of the first grating structure 101 and the duty cycle of the second grating structure 102 can be the same or different.

[0074] In one embodiment, when the first protrusion 101a is flush with the second groove 102b and the grating period t1 of the first grating structure 101 is the same as the grating period of the second grating structure 102, that is, h2=h3 and t1=t2, after the incident wavefront is diffracted by the diffraction grating 100, the diffraction pattern obtained has at least the following characteristics: in the direction in which the first protrusion 101a and the first groove 101b are periodically arranged, the diffraction pattern is in the shape of a first step function; in the direction in which the second protrusion 102a and the second groove 102b are periodically arranged, the diffraction pattern is in the shape of a second step function; the first step function and the second step function have the same period, the same grating duty cycle, and the height of the protrusion of the first step function is equal to the height of the groove of the second step function.

[0075] In another embodiment, see Figure 5 , when the first protrusion 101a is flush with the second groove 102b and the shapes of the first protrusion 101a and the second protrusion 102a are both square, then t1=t2 and the duty ratio of the first grating structure 101 and the duty ratio of the second grating structure 102 are both equal to 50%, the diffraction grating 100 not only has two mutually perpendicular grating axes (G1 and G2), but also has a third grating axis G3 that is 45 degrees from the above two grating axes (see Figure 5 and Figure 6), the structure of the diffraction grating 100 is symmetrical in any direction, and its performance will not be reduced due to the change of the polarization state of the incident light. After the incident wavefront is diffracted by the diffraction grating 100, the obtained diffraction pattern has at least the characteristic (see Figure 7 ): In the direction (G1) in which the first protrusions 101a and the first grooves 101b are periodically arranged, the diffraction pattern is in the shape of a first square wave function F1; in the direction (G1) in which the second protrusions 102a and the second grooves 102b are periodically arranged, the diffraction pattern is in the shape of a second square wave function F2; in the direction (G2) in which the first grating structure 101 and the second grating structure 102 are periodically arranged, the diffraction pattern is in the shape of a third square wave function F3 and a fourth square wave function F4; the first square wave function F1 and the third wave function F3 are exactly the same, the second square wave function F2 and the fourth square wave function F4 are exactly the same, and the diffraction pattern is a symmetrical structure in any direction.

[0076] It should be noted that the diffraction grating 100 of the above embodiment can be integrated into an imaging system including a pixel array at a low cost. For example, it can be directly loaded onto a manufactured pixel array or added during the manufacturing process of the pixel array.

[0077] It should be noted that the diffraction grating assembly 11 of the above embodiment may include a plurality of diffraction gratings arranged in a two-dimensional matrix array, and two adjacent diffraction gratings differ in at least one or more of grating period, grating duty cycle and height.

[0078] In this embodiment, the pixel array 12 is located below the diffraction grating component 11, and is configured to receive the diffraction pattern corresponding to the diffraction wavefront and obtain the light field image data of the diffraction pattern. The light field image data includes information such as the wavelength, incident angle, and incident position of the incident wavefront; the pixel array 12 can be a front-side illumination (FSI) or a back-side illumination (BSI). Optionally, the back-side illumination has a higher light field image data acquisition efficiency, and the distance between the diffraction grating component 11 and the pixel array 12 is closer, and the influence of the high diffraction order is smaller.

[0079] In this embodiment, the pixel array 12 receives a diffraction pattern having the same period and shape as the diffraction grating, and has a light field intensity distribution along the grating axis. The diffraction pattern and the light field intensity distribution are related to information such as the geometric shape of the diffraction grating, the wavelength and incident angle of the incident wavefront, and the position of the incident wavefront. Therefore, the light field image data corresponding to the diffraction pattern can be obtained by reading the light field intensity distribution.

[0080] Specifically, in order to read the light field intensity distribution of the diffraction pattern, a pixel array 12 is provided below the diffraction grating assembly 11. The pixel array 12 includes a two-dimensional array of photodetectors, and the photodetector may be an image sensor. As a single sensor, a dual camera is not required, and the device consumes little power. Optionally, the image sensor may be a pn junction diode, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) image sensor, an APD (Avalanche Photo Diode), a SPAD (Single-Photon Avalanche Diode), or other photodetectors.

[0081] Specifically, the diffraction grating and the pixel unit may be arranged in an offset manner or in an aligned manner. The following description is made with reference to the first grating structure 101 .

[0082] exist Figure 8a In the embodiment, the first grating structure 101 has a duty cycle of 50%, and the pixel units correspond to the protrusions and grooves one by one and are aligned. Figure 8b In, with Figure 8a Compared to the embodiment of the present invention, the first grating structure 101 has a duty cycle of 50%, but the pixel array is laterally offset by a quarter of the grating period relative to the diffraction grating. As a result, each pixel unit is vertically aligned below the transition between a corresponding protrusion and a corresponding adjacent groove. Figure 8c In the embodiment, the duty cycle of the first grating structure 101 is 50%, but Figure 8a In the embodiment of the invention, the ratio R of the grating period to the pixel size along the grating axis is equal to 4 instead of 2. Therefore, there are two pixel units corresponding to each protrusion and each groove. Figure 8d In FIG. 1 , the ratio R of the grating period to the pixel size along the grating axis is equal to two, but the duty cycle of the first grating structure 101 is different from 50%.

[0083] It should be noted that, when the diffraction grating assembly has a plurality of diffraction gratings, the relative arrangements of different diffraction gratings and pixel units may be the same or different.

[0084] In one embodiment, the photodetector array is arranged in a row-column matrix, see Fig. 9The top view of the array includes pixel columns C1, C2, C3, ..., Cx and pixel rows R1, R2, R3, ..., Ry. The number of rows and columns defines the resolution of the pixel array. The photodetector array can convert the diffracted wavefront incident on it into a photoelectric signal. The intensity of the electrical signal is proportional to the intensity of the optical signal. The obtained electrical signal can be directly used to acquire addresses, read data and process data independently of the photodetector array, or it can be used to acquire addresses, read data and process data with the assistance of other electronic components, thereby obtaining the light field intensity distribution.

[0085] In one embodiment, the size d of the pixel unit in the grating axis direction is less than or equal to half the grating period of the diffraction grating 100, so that one diffraction period corresponds to at least two pixel units, and the pixel array can fully sample the light field intensity distribution. The size d can be the size along any grating axis direction.

[0086] In one embodiment, the pixel array 12 is located in the near field region below the diffraction grating component 11. On the one hand, the diffraction pattern of the incident wavefront after passing through the diffraction grating component 11 can be calculated by the Fresnel diffraction theory in this range; on the other hand, the near field region can ensure that high-order diffraction orders will not generate crosstalk between adjacent pixel units. In one embodiment, the pixel array 12 is located in the range of 0.5 μm-20 μm below the diffraction grating component 11.

[0087] In this embodiment, the diffraction pattern / light field intensity distribution has the same period and distribution as the diffraction grating, and when the angle of the incident light changes along the grating axis direction, the diffraction pattern / light field intensity distribution will shift laterally along the incident direction, while keeping its period and shape unchanged: Taking the first grating structure 101 as an example, please refer to Figure 10a-Figure 10c (where 000 represents the diffraction pattern, the solid line diffraction pattern in the figure corresponds to the solid line incident wavefront, and the dotted line diffraction pattern corresponds to the dotted line incident wavefront). When the incident light is projected on the yz plane at a certain angle θ with the z axis y When the diffraction pattern moves a certain distance laterally along the y-axis, the projection on the xz plane will have a certain angle θ with the z-axis. x , the diffraction pattern will move laterally along the x-axis. Therefore, the incident angle of the incident wavefront can be obtained by the light field intensity distribution combined with the measurement of the lateral displacement of the diffraction pattern. Among them, because the diffraction grating has gratings in two directions perpendicular to each other, the angle of the incident light in two spatial directions can be calculated, the utilization rate of the pixel unit area can be improved, the size can be reduced, the power consumption can be reduced, the resolution can be improved, and the light field image data can be read more efficiently. At the same time, the diffraction grating is symmetrical, which maintains the quality of the two-dimensional image while increasing the depth information, and its performance will not be reduced due to changes in the polarization state of the incident light.

[0088] In one embodiment, a simulation test is performed on the lateral displacement of the diffraction pattern, and the light field intensity distribution detected on the pixel array 100Zt / 2 away from the diffraction grating with a repetition period of 3 has the following results: when the angle of the incident light deviates by 10 degrees along the x direction (θ=10°, ), the light field intensity distribution also moves horizontally along the x direction as a whole; when the incident angle is deflected by 10 degrees along the y direction (θ=10°, ), the light field intensity has a corresponding lateral translation along the y direction; when the incident angle is θ = 10°, When , the light field moves along the direction of 45° on the xy plane.

[0089] Specifically, the relationship between the incident angle θ of the incident wavefront and the intensity I(θ) of the diffraction pattern received by the pixel row and / or pixel column can be expressed as: I(θ) = I0·A(θ)·[1-m·cos(βθ+α)], where I0 is the intensity of the incident wavefront; A(θ) is the window function, and the transmission coefficient that changes with the incident angle can be obtained through the incident angle and the window function; m is the modulation coefficient of the grating, β is the sensitivity of the intensity to the change of the incident angle, and α is the phase offset between the pixel row or column and the diffraction grating. First, assume that the incident light has an angle θ only in the x direction. x , causing the diffraction pattern to shift along the x-axis by d x Similarly, when the angle θ y When it is in the y direction, the diffraction pattern moves along the y direction by d y Since the two directions are orthogonal, when the incident angle in the x and y directions is (θ x ,θ y ), the overall movement of the diffraction pattern is (d x , d y ).

[0090] In one embodiment, in order to calculate the intensity I0 and angle θ of the incident wavefront based on the intensity I(θ) of the read pixel rows and / or pixel columns, the intensity values ​​I(0), I(π / 2), I(π) and I(3π / 2) of the corresponding pixel rows or pixel columns when α=0, π / 2, π, 3π / 2 can be obtained through two diffraction gratings (diffraction grating 100 and diffraction grating 200) with different phase offsets. When the diffraction grating 100 and the pixel unit (the pixel unit size is half the grating period) are aligned, the intensity when the relative offset α between the diffraction grating and the pixel array is 0 and π can be read from adjacent pixel rows or pixel columns, and the intensity value read from each corresponding pixel unit is recorded as P(m, n). Fig.11aAs shown in the figure (the solid line represents the diffraction grating 200, and the dotted line represents the pixel unit), in order to obtain the intensity information when the offset α is π / 2 and 3π / 2, in the diffraction grating 200, the pixel unit and the boundary of the diffraction grating are offset by a quarter period (please refer to Figure 8b The structure diagram of the offset setting between the diffraction grating and the pixel unit), the adjacent units correspond to the intensities when the phase offset α is π / 2 and 3π / 2 respectively, and the intensity value read by the corresponding pixel unit is recorded as P'(m,n).

[0091] In another embodiment, if Fig.11b As shown in the figure (solid line 001 represents diffraction grating 100, solid line 002 represents diffraction grating 200, and dotted line represents pixel unit), diffraction grating 100 and diffraction grating 200 can also be offset by a quarter period in both x and y directions, and the data read out by the corresponding pixel unit is recorded as P'(m, n). Further, in order to prevent two or more incident angles from generating the same diffraction pattern and causing aliasing, diffraction gratings with different periods in adjacent areas can also be selected to uniquely determine the angle of the incident light.

[0092] Therefore, when calculating the angle between the incident light and the x-axis, the relationship between the intensity values ​​I(0), I(π / 2), I(π) and I(3π / 2) and the intensity read by each pixel unit is as follows:

[0093]

[0094]

[0095]

[0096]

[0097] When calculating the angle between the incident light and the y-axis, the relationship between the intensity values ​​I(0), I(π / 2), I(π) and I(3π / 2) and the intensity read by each pixel unit is as follows:

[0098]

[0099]

[0100]

[0101]

[0102] Through the intensity information of each pixel row and pixel column, the angle of the incident light in the x and y directions can be calculated respectively. For any case, the following formula is met:

[0103]

[0104]

[0105] When the solid angle of the incident angle is (θ, ), θ x ,θ y and solid angle (θ, ) has the following relationship:

[0106]

[0107]

[0108] tanθ=(tan 2 θ x +tan 2 θ y ) 1 / 2

[0109]

[0110] Therefore, the solid angle of the incident light can be obtained by calculating the angle of the incident light in the x and y directions.

[0111] It should be noted that when the different diffraction gratings in the above embodiments are arranged in an array, for a single point light source, the position of the point light source in three-dimensional coordinates can be located by the intensity and direction of the light reaching each pixel unit. For more complex actual scenes, the light field distribution of the scene can be obtained through the above implementation, and then combined with the assistance of computers (see Fig.12 , where 20 refers to a computer) or other embedded computing function chips, etc., calculate the depth information of the scene through a reconstruction algorithm and reconstruct the three-dimensional scene. In other applications, depth map extraction, surface reconstruction, image refocusing and other applications can also be performed based on light field image data.

[0112] The light field imaging device of the above embodiment includes a diffraction grating component and a pixel array, and receives the incident wavefront of the scene through the diffraction grating component, diffracts the incident wavefront, and generates a diffraction wavefront, so that the pixel array receives the diffraction pattern corresponding to the diffraction wavefront, and obtains the light field image data of the diffraction pattern. Since the diffraction grating has gratings in two directions perpendicular to each other, it is possible to calculate the angle of the incident wavefront in two spatial directions, and its performance will not be reduced due to changes in the polarization state of the incident light. The utilization rate of the pixel unit area is improved, the size is reduced, the power consumption is reduced, the resolution is improved, the light field image data is acquired more efficiently, and the quality of the two-dimensional image is maintained while increasing the depth information.

[0113] Further, in order to realize the spectral filtering function, based on the above embodiment, the diffraction grating component may further include a substrate, the substrate includes a spectral filtering material, and the diffraction grating component is configured to receive an incident wavefront of a scene, perform spectral filtering and diffraction on the incident wavefront, and generate a diffracted wavefront. Specifically, the substrate includes a transparent spectral filtering material, allowing the incident wavefront to pass through. For example, the material of the substrate can be selected from silicon oxide, polymer, colloidal particles, SU-8 photoresist, and glass.

[0114] In another embodiment, the diffraction gratings in different regions can be made to selectively transmit different wavelengths by directly doping with dyes, so that the diffraction gratings have both diffraction and filtering functions.

[0115] Further, in order to realize the spectral filtering function, based on the above embodiment, please refer to Fig.13 (Take a diffraction grating as an example), the light field imaging device 10 may further include a filter plate 13. The filter plate 13 is disposed between the diffraction grating component 11 and the pixel array 12, and is configured to perform spectral filtering on the diffracted wavefront.

[0116] In one embodiment, the filter plate 13 includes one or more filters, which can select the wavelength of the incident light, can image a single wavelength, or can use multiple filters to obtain color information of the scene.

[0117] In another embodiment, the filter board 13 includes a color filter array. The color filter array is aligned with the pixel array so that each color filter covers at least one pixel unit; or each color filter can be optically coupled to at least two pixel units. In practical applications, the color filter array can be arranged spatially according to a mosaic color pattern, or can be arranged in other patterns according to other practical needs.

[0118] Furthermore, based on the above embodiments, in order to improve the accuracy of light field image data acquired by the pixel array, avoid information crosstalk between pixel units, and improve the collection efficiency of angled incident light, the light field imaging device also includes a microlens array.

[0119] In one embodiment, see Fig.14 The microlens array 14 is disposed above the pixel array 12 and is configured to focus the diffracted wavefront onto the corresponding pixel unit. Each microlens unit in the microlens array 14 is optically coupled to the corresponding pixel unit and focuses the diffracted wavefront incident thereon onto the corresponding pixel array unit.

[0120] In another embodiment, the microlens array may also be disposed on a filter plate, and each microlens unit in the microlens array is optically coupled to a corresponding filter unit to focus the diffracted wavefront incident thereon onto the corresponding filter unit.

[0121] Furthermore, on the basis of the above embodiments, the light field imaging device may also include an optical control device. The optical control device is arranged in the optical path of the incident wavefront between the scene and the diffraction grating assembly, so that the incident wavefront can be regulated by the optical control device, corresponding to different needs in practice and adapting to different usage scenarios, broadening the scope of use of the light field imaging device, and further improving the accuracy of the pixel array in acquiring light field image data. Among them, the regulation includes convergence, steering, transmission, reflection, refraction, dispersion, and alignment. In practical applications, the optical control device can select an optical device with one or more functions of convergence, steering, transmission, reflection, refraction, dispersion, alignment, etc. according to the needs. For example, when the demand is convergence, a convex lens can be selected as the optical control device.

[0122] It should be noted that the light field imaging device in the above embodiment also includes a light source, which is configured to project light onto a scene to form an incident wavefront. The light source can be an active light source or a passive light source. In one embodiment, when the light source is an active light source, the light of the active light source is projected onto an object in the scene to form a reflection or transmission, thereby forming an incident wavefront of the scene; further, the incident wavefront can be filtered by a diffraction grating component or a filter plate with a spectral filtering function, thereby obtaining light field information of a single wavelength or other specific wavelengths.

[0123] This embodiment also provides a light field imaging method based on the light field imaging device described in the above embodiment.

[0124] See also Fig.15 The light field imaging method includes: step S101, receiving an incident wavefront of a scene, diffracting the incident wavefront to generate a diffracted wavefront; step S102, receiving a diffraction pattern corresponding to the diffraction wavefront, and obtaining light field image data according to the diffraction pattern; step S103, obtaining depth information of the scene according to the light field image data, and constructing a three-dimensional scene.

[0125] The above-mentioned light field imaging method receives the incident wavefront of the scene, diffracts the incident wavefront to generate a diffracted wavefront, receives the diffraction pattern corresponding to the diffraction wavefront, efficiently obtains light field image data according to the diffraction pattern, and obtains the depth information of the scene according to the light field image data, thereby constructing a three-dimensional scene with high resolution and stronger sense of reality, thereby improving the user experience.

[0126] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A light field imaging device, characterized in that: include: A diffraction grating component is configured to receive an incident wavefront of a scene, diffract the incident wavefront, and generate a diffracted wavefront; the diffraction grating component includes a diffraction grating having at least two mutually perpendicular grating axes and a refractive index that changes periodically in the direction of the grating axis; a pixel array, located below the diffraction grating assembly, configured to receive a diffraction pattern corresponding to the diffraction wavefront and acquire light field image data according to the diffraction pattern; Wherein, the diffraction grating includes a first grating structure and a second grating structure which are periodically arranged; The first grating structure includes a first protrusion and a first groove periodically arranged in the grating axis direction, and the second grating structure includes a second protrusion and a second groove periodically arranged in the grating axis direction, the first protrusion is arranged corresponding to the second protrusion, and the first groove is arranged corresponding to the second groove; in the direction perpendicular to the grating axis, the second protrusion is higher than the first protrusion, and the second groove is higher than the first groove; the refractive index of each protrusion is higher than the refractive index of each groove filler.

2. The light field imaging device according to claim 1, characterized in that: The diffraction grating is a two-dimensional phase grating.

3. The light field imaging device according to claim 1, characterized in that: The first protrusion is flush with the second groove.

4. The light field imaging device according to claim 3, characterized in that: In a direction perpendicular to the grating axis, a ratio of a height difference between the first protrusion and the first groove to a height difference between the second protrusion and the first groove is 1:

2.

5. The light field imaging device according to claim 1, characterized in that: The number of grating periods is at least three.

6. The light field imaging device according to claim 1, characterized in that: The diffraction grating and the pixel unit are offset or aligned.

7. The light field imaging device according to claim 1, characterized in that: The pixel cells in the pixel array have a pixel size along the grating axis, and the pixel size is less than or equal to half the grating period.

8. The light field imaging device according to any one of claims 1 to 7, characterized in that: The incident wavefront includes one or more of ultraviolet light, visible light, near infrared light, infrared band, terahertz and microwave.

9. The light field imaging device according to any one of claims 1 to 7, characterized in that: The pixel array is front-side illuminated or back-side illuminated.

10. The light field imaging device according to any one of claims 1 to 7, characterized in that: The pixel array is located in the near-field region of the diffraction grating assembly.

11. The light field imaging device according to any one of claims 1 to 7, characterized in that: The diffraction grating assembly includes a plurality of the diffraction gratings, and the plurality of the diffraction gratings are arranged in a two-dimensional matrix array.

12. The light field imaging device according to any one of claims 1 to 7, characterized in that: The diffraction grating assembly further includes a substrate, wherein the substrate includes a spectral filtering material. The diffraction grating assembly is configured to receive an incident wavefront of a scene, perform spectral filtering and diffraction on the incident wavefront, and generate a diffracted wavefront.

13. The light field imaging device according to any one of claims 1 to 7, characterized in that: The light field imaging device further comprises: The filter plate is disposed between the diffraction grating assembly and the pixel array, and is configured to perform spectral filtering on the diffracted wavefront.

14. The light field imaging device according to any one of claims 1 to 7, characterized in that: The light field imaging device further comprises: The microlens array is disposed above the pixel array and is configured to focus the diffracted wavefront onto corresponding pixel units.

15. The light field imaging device according to any one of claims 1 to 7, characterized in that: The light field imaging device further comprises: The optical control device is arranged in the optical path of the incident wavefront between the scene and the diffraction grating component.

16. The light field imaging device according to any one of claims 1 to 7, characterized in that: The light field imaging device further comprises: A light source is configured to project light onto the scene to form the incident wavefront.

17. A light field imaging method based on the light field imaging device according to any one of claims 1 to 16, characterized in that: include: receiving an incident wavefront of a scene, diffracting the incident wavefront to generate a diffracted wavefront; receiving a diffraction pattern corresponding to the diffraction wavefront, and acquiring light field image data according to the diffraction pattern; Depth information of the scene is acquired according to the light field image data to construct a three-dimensional scene.

Images