Under-display light field sensor, device having the under-display light field sensor, and image reconstruction method

Abstract

An under-display light field sensor, a device having the under-display light field sensor, and an image reconstruction method. The under-display light field sensor is used to sense a fingerprint or a touch located above or near a display panel. The under-display light field sensor includes a light field image sensor and a plurality of microlenses. The light field image sensor includes a plurality of sub-arrays of a plurality of sensing pixels. The sensing pixels are located under the display panel. The sensing pixels in the same sub-array are used to capture a plurality of images of an object located above or near the display panel by sensing light in different directions. The microlenses are located above the sub-arrays of the sensing pixels. Each of the microlenses corresponds to one of the sub-arrays of the sensing pixels. Thereby, the influence of the display panel structure on the sensed image can be reduced.

Description

Technical Field

[0001] The embodiments described in this disclosure relate to a sensing technology, particularly to an under-display light field sensor for sensing fingerprints or touches, a device having the under-display light field sensor, and an image reconstruction method. Background Art

[0002] With the development of technology, fingerprint sensing and fingerprint identification functions have been applied to many fields. In some related technologies, under-display sensors are used to sense fingerprints on a display panel. However, due to the structure of the display panel, interference is generated and affects the (multiple) sensed images. Summary of the Invention

[0003] Some embodiments of this disclosure relate to an under-display light field sensor. The under-display light field sensor is used to sense a fingerprint or a touch located above or near a display panel. The under-display light field sensor includes a light field image sensor and a plurality of microlenses. The light field image sensor includes a plurality of sub-arrays of a plurality of sensing pixels. The sensing pixels are located under the display panel. The sensing pixels in the same sub-array are used to capture a plurality of images of an object located above or near the display panel by sensing light in different directions. The microlenses are located above the sub-arrays of the sensing pixels. Each of the microlenses corresponds to one of the sub-arrays of the sensing pixels.

[0004] In some embodiments, the under-display light field sensor further includes a sensing circuit. The sensing circuit includes a readout circuit and an image processing circuit. The readout circuit is used to read out a plurality of sensing signals of the sensing pixels. The sensing signals represent the images captured by the light field image sensor from the different directions. The image processing circuit is used to generate a plurality of sub-pixels based on the sensing signals of the sensing pixels and combine or superimpose the sub-pixels of the object to reconstruct an image of the object.

[0005] In some embodiments, the image processing circuit includes a sub-image data generator, an offset module, and an image combining module. The sub-image data generator is used to group the signals of the corresponding sensing pixels in the sub-arrays to generate the sub-images of the object. The corresponding sensing pixels in the sub-arrays are defined as the sensing pixels located at the same position in the sub-arrays and are grouped into one of the sub-images. The offset module is used to offset each of the sub-images of the object by a displacement amount. The image combining module is used to reconstruct an image data by combining or superimposing the offset sub-images of the object.

[0006] In some embodiments, each of the microlenses has an axis. The axis is offset from the geometric center of a corresponding one of the sub-arrays of the sensing pixels.

[0007] In some embodiments, each of the microlenses has an axis. The axis is aligned with a geometric center of a corresponding one of the sub-arrays of the sensing pixels.

[0008] In some embodiments, the under-display light field sensor further includes an imaging lens. The imaging lens is located above the microlenses. The imaging lens is configured to form an image of an object, and the object is located above or near the light field image sensor.

[0009] In some embodiments, the under-display light field sensor is implemented as a fingerprint sensor.

[0010] Some embodiments of the present disclosure relate to an apparatus having an under-display light field sensor. The under-display light field sensor is configured to sense a fingerprint or a touch. The apparatus includes a display panel. The display panel includes an opposite display surface and a back surface. The under-display light field sensor includes a plurality of sub-arrays of sensing pixels and a plurality of microlenses. The sensing pixels face the back surface. The microlenses are located between the sub-arrays of the sensing pixels and the back surface. Each of the microlenses corresponds to one of the sub-arrays of the sensing pixels. The sensing pixels located in the same sub-array are configured to capture a plurality of images of an object by sensing light in different directions. The object is located above or near the display panel.

[0011] In some embodiments, the apparatus further includes a sensing circuit. The sensing circuit includes a readout circuit and an image processing circuit. The readout circuit is configured to read out sensing signals of the sensing pixels. The sensing signals represent the images captured by the light field image sensor from the different directions. The image processing circuit is configured to generate a plurality of sub-images based on the sensing signals of the sensing pixels and combine or superimpose sub-pixels of the object to reconstruct an image of the object.

[0012] In some embodiments, the image processing circuit includes a sub-image data generator, an offset module, and an image combining module. The sub-image data generator is configured to group signals of corresponding sensing pixels in the sub-arrays to generate the sub-images of the object. The corresponding sensing pixels in the sub-arrays are defined as the sensing pixels located at the same position in the sub-arrays and are grouped into one of the sub-images. The offset module is configured to offset each of the sub-images of the object by a displacement amount. The image combining module is configured to reconstruct an image data by combining or superimposing the offset sub-images of the object.

[0013] Some embodiments of the present disclosure relate to a method for reconstructing an image. The image is sensed by an under-display light field sensor to sense a fingerprint or an object. The fingerprint or the object is located above or near a display panel. The under-display light field sensor includes a plurality of sub-arrays of sensing pixels and a plurality of microlenses. The sensing pixels are located under the display panel. The sensing pixels in the same sub-array are used to capture a plurality of images of an object by sensing light in different directions. The object is located above or near the display panel. The microlenses are located above the sub-arrays of the sensing pixels. Each of the microlenses corresponds to one of the sub-arrays of the sensing pixels. The method includes: obtaining an initial image and generating a plurality of sub-images of the object based on the initial image; obtaining a plurality of corresponding displacement amounts of the sub-images of the object; using the corresponding displacement amounts to correct the sub-images of the object to generate a plurality of corrected sub-images; and reconstructing the corrected sub-images to generate a reconstructed image.

[0014] In some embodiments, the initial image includes a plurality of n×m matrices. Each matrix includes a plurality of elements a ij . n represents the columns in each matrix, m represents the rows in each matrix, i represents the column of the element a ij and is an integer from 1 to n, and j represents the row of the element a ij and is an integer from 1 to m. Generating the sub-images of the object based on the initial image includes extracting the elements a ij with the same i value and the same j value from each of the matrices to generate one of the sub-images. The number of the sub-images of the object is the same as the product of n and m.

[0015] In some embodiments, the method further includes: processing at least one of the sub-images of the object before superimposing the corrected sub-images to generate a reconstructed image.

[0016] In some embodiments, processing at least one of the sub-images of the object includes reducing noise, enhancing contrast, adjusting brightness, filtering some frequency components in the signal, or any combination of the above.

[0017] In some embodiments, generating the sub-images of the object based on the initial image includes grouping a plurality of signals of the corresponding sensing pixels in the sub-arrays to generate the sub-images of the object. The corresponding sensing pixels in the sub-arrays are defined as the sensing pixels located at the same position in the sub-arrays and are grouped into one of the sub-images.

[0018] In some embodiments, the sensing pixels (a, b) of the sub-arrays are represented. a represents the corresponding sub-array, and b represents the sensing pixel in the corresponding sub-array. The sensing pixels having the same b value are defined as being at the same position in the corresponding sub-array. Generating the sub-images of the object from the initial image includes clustering the signals of the sensing pixels having the same b value to generate one of the sub-images of the object.

[0019] In some embodiments, correcting the sub-images of the object by the displacement amounts to generate the corrected sub-images includes offsetting the sub-images of the object by the displacement amounts.

[0020] In some embodiments, reconstructing the corrected sub-images includes superimposing or combining the offset sub-images of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] To make the above and other objects, features, advantages, and embodiments of the disclosure more apparent and understandable, the descriptions of the accompanying drawings are as follows:

[0022] Figure 1 is a schematic diagram of a fingerprint recognition system according to some embodiments of the disclosure;

[0023] Figure 2A is a schematic diagram of an under-display light field sensor according to some embodiments of the disclosure;

[0024] Figure 2B is according to some embodiments of the disclosure Figure 2A a top view of the under-display light field sensor in;

[0025] Figures 3A to 3C is a schematic diagram of an under-display light field sensor according to some embodiments of the disclosure;

[0026] Figure 4 is a schematic diagram of a fingerprint recognition system according to some embodiments of the disclosure;

[0027] Figure 5 is a schematic diagram of a fingerprint recognition system according to some embodiments of the disclosure;

[0028] Figure 6 is a schematic diagram of a fingerprint recognition system with multiple microlenses offset outward according to some embodiments of the disclosure;

[0029] Figure 7 is a schematic diagram of a fingerprint recognition system with multiple microlenses offset inward according to some embodiments of the disclosure;

[0030] Figure 8is a schematic diagram of a sensing circuit according to some embodiments of the present disclosure;

[0031] Figure 9 is a flowchart of an image reconstruction method according to some embodiments of the present disclosure;

[0032] Figure 10 is a schematic diagram of multiple sub - arrays of multiple sensing pixels according to some embodiments of the present disclosure;

[0033] Figure 11 is a schematic diagram of a sub - image according to some embodiments of the present disclosure;

[0034] Figure 12 is a schematic diagram of a sub - image according to some embodiments of the present disclosure;

[0035] Figure 13 is a schematic diagram of a sub - image according to some embodiments of the present disclosure;

[0036] Figure 14 is a schematic diagram of multiple sub - images according to some embodiments of the present disclosure;

[0037] Figure 15 is a schematic diagram of multiple displacement amounts according to some embodiments of the present disclosure;

[0038] Figure 16 is a schematic diagram of a combined image according to some embodiments of the present disclosure;

[0039] Figure 17 is a flowchart of an image combination method according to some embodiments of the present disclosure; and

[0040] Figure 18 is a flowchart of an image combination method according to some embodiments of the present disclosure. Detailed Description of the Invention

[0041] All terms used herein have their ordinary meanings. The definitions of the above - mentioned terms in commonly used dictionaries, including examples of the use of any of the terms discussed herein, are only illustrative and should not limit the scope and meaning of the present disclosure. Similarly, the present disclosure is not limited only to the various embodiments shown in this specification.

[0042] The term "coupled" used herein may also refer to "electrically coupled", and the term "connected" may also refer to "electrically connected". "Coupled" and "connected" may also refer to two or more elements cooperating or interacting with each other.

[0043] Reference Figure 1 . Figure 1FIG. 0 is a schematic diagram of a fingerprint recognition system 100 according to some embodiments of the present disclosure. For Figure 1 example, the fingerprint recognition system 100 includes a display panel 110 and an under-display light field sensor 120. The display panel 110 includes an opposite display surface S1 and a back surface S2. The display surface S1 is used to display an image. The under-display light field sensor 120 can be disposed under the display panel 110 and face the back surface S2. The under-display light field sensor 120 (e.g., implemented as a fingerprint sensor) is used to sense a target object (e.g., the fingerprint FP of a finger F) on or near the display panel 110. For example, the under-display light field sensor 120 can sense light rays L emitted from an object (e.g., the fingerprint FP) and emitted from different directions. The sensed information can be used to identify the object (e.g., the fingerprint FP).

[0044] In some embodiments, the object sensed by the under-display light field sensor 120 is not limited to the fingerprint FP, and the object sensed by the under-display light field sensor 120 can be other objects.

[0045] In some related technologies, due to the structure of the display panel, interference will be generated and affect the (multiple) sensed images. One solution is to use hardware or software to eliminate the interference caused by the structure of the display panel according to the difference between the spatial frequency of the display panel structure and the spatial frequency of the target object. However, when the spatial frequency of the display panel structure is close to the spatial frequency of the target object, the above method will not be able to eliminate the interference. In addition, due to problems such as Moire pattern, Newton ring, or MURA pattern, the interference cannot be completely eliminated.

[0046] Compared with the above related technologies, in the fingerprint recognition system 100 of the present disclosure, since the light rays L are emitted from an object (e.g., the fingerprint FP) and emitted from different directions and the light rays L pass through different positions of the structure of the display panel 110, the light rays L will not be blocked by the same position of the structure of the display panel 110. Accordingly, the influence caused by the structure of the display panel 110 can be reduced.

[0047] In some embodiments, the under-display light field sensor 120 can sense the intensity and direction of the light rays L, and the sensed information can be collected and used to identify the object (e.g., the fingerprint FP). Since the intensity and direction of the light rays L are sensed and recorded, the distance information of the object (e.g., the fingerprint FP) can be obtained according to the phase difference corresponding to the recorded information.

[0048] Reference Figure 2A and Figure 2B . Figure 2ASchematic diagram of the under-display light field sensor 120 according to some embodiments of the present disclosure. Figure 2B Shown in accordance with some embodiments of the present disclosure Figure 2A Top view of the under-display light field sensor 120 therein. For Figure 2A example, the under-display light field sensor 120 includes a light field image sensor 121 and microlenses 122. The light field image sensor 121 includes sensing pixels 1211 and a substrate 1212. The sensing pixels 1211 can be disposed under the display panel 110 and on top of the substrate 1212, and face Figure 1 the back surface S2 of the display panel 110 therein. For Figure 2B example, the sensing pixels 1211 are divided into sub-arrays SA1-SAN, where N is a positive integer greater than 1. The sensing pixels 1211 in each sub-array SA1-SAN are used to capture images of an object (e.g., fingerprint FP) from different directions shown in Figure 1 therein. The microlenses 122 are attached to the sub-arrays SA1-SAN, and each microlens 122 corresponds to one of the sub-arrays SA1-SAN. In this example, the under-display light field sensor 120 includes 40,000 sub-arrays, and each sub-array SA1-SAN includes 25 sensing pixels 1211.

[0049] The number of sub-arrays (e.g., 40,000) in the under-display light field sensor 120 is for illustrative purposes only and the present disclosure is not limited to this number. Various numbers of sub-arrays in the under-display light field sensor 120 are within the scope of the present disclosure.

[0050] The number of sensing pixels 1211 (e.g., 25) in each sub-array SA1-SAN is for illustrative purposes only and the present disclosure is not limited to this number. Various numbers of sensing pixels 1211 in each sub-array SA1-SAN are within the scope of the present disclosure.

[0051] In some embodiments, the light field image sensor 121 is implemented as a backside illumination (BSI) sensor or a front side illumination (FSI) sensor. In some embodiments, the light field image sensor 121 is applied with an imaging lens. In some embodiments, the sensing pixels 1211 are global shutter pixels or rolling shutter pixels. In some embodiments, in addition to the light source (e.g., backlight module) of the display panel 110 itself, an additional light source can be added to the under-display light field sensor 120.

[0052] Refer to Figure 3A . Figure 3ASchematic diagram of the under-display light field sensor 120A according to some embodiments of the present disclosure. For Figure 3A example, in some embodiments, the light shielding structure LSS1 includes multiple parts, and these parts are disposed between two adjacent microlenses 122. By configuring the light shielding structure LSS1, the light of one of the two adjacent microlenses 122 will not be interfered by the light of the other. Therefore, light crosstalk can be avoided.

[0053] Reference Figure 3B . Figure 3B Schematic diagram of the under-display light field sensor 120B according to some embodiments of the present disclosure. For Figure 3B example, in some embodiments, the light shielding structure LSS2 includes multiple parts, and these parts are disposed under two adjacent microlenses 122. In different embodiments, the distance from the light shielding structure LSS2 to the microlens 122 can be adjusted according to design requirements. In addition, the light shielding structure LSS2 can be extended to a longer length, for example, from the position between two adjacent microlenses 122 to the positions of these pixels. Similarly, by configuring the light shielding structure LSS2, the light of one of the two adjacent microlenses 122 will not be interfered by the light of the other. Therefore, light crosstalk can be avoided.

[0054] Reference Figure 3C . Figure 3C Schematic diagram of the under-display light field sensor 120C according to some embodiments of the present disclosure. In different embodiments, the distance from the light shielding structure LSS3 to the microlens 122 can be adjusted according to design requirements. In addition, the light shielding structure LSS3 can be extended to a longer length, for example, from the position between two adjacent microlenses 122 to the positions of these pixels. In some embodiments, Figure 3A the light shielding structure LSS1 in Figure 3B and the light shielding structure LSS2 in Figure 3C can be combined to form the new light shielding structure LSS3 in

[0055] Reference Figure 4 . Figure 4 Schematic diagram of the fingerprint recognition system 400 according to some embodiments of the present disclosure. Similarly, the fingerprint recognition system 400 can not only reduce the influence of the structure of the display panel 110, but also the under-display light field sensor 120 of the fingerprint recognition system 400 can be used to perform a hovering touch function. In other words, the finger F does not need to touch the display panel 110. For Figure 4For example, the finger F is located above the display panel 110 and the display panel 110 includes panel pixels 111. The imaging lens IMLS is disposed above the microlens 122 and below the panel pixel 111. The imaging lens IMLS is configured to form an image of an object (e.g., fingerprint FP) on the light field image sensor 121. The imaging lens IMLS can provide a wider field of view (FOV) and better resolution. The light field image sensor 121 senses the light ray L passing through the imaging lens IMLS and the under-display light field sensor 120 calculates the vertical distance (e.g., the distance in the direction Z) from the under-display light field sensor 120 to the object (e.g., fingerprint FP). Together with the coordinates of the under-display light field sensor 120 on the plane formed by the direction X and the direction Y, the movement of the finger F in the three-dimensional space can be tracked to achieve the hovering touch function. In this example, each microlens 122 has an axis AX, and this axis AX is substantially aligned with the geometric center GC of the corresponding sub-array SA.

[0056] Reference Figure 5 。 Figure 5 is a schematic diagram of a fingerprint identification system 500 illustrated in accordance with some embodiments of the present disclosure. Similarly, the fingerprint identification system 500 can not only reduce the influence of the structure of the display panel 110, but also reduce the overall height. In other words, compared with Figure 4 the fingerprint identification system 400 in Figure 5 the fingerprint identification system 500 in Figure 5 is not configured with the imaging lens IMLS, so that

[0057] Reference Figure 6 。 Figure 6 is a schematic diagram of a fingerprint identification system 600 with multiple microlenses 622 offset outward illustrated in accordance with some embodiments of the present disclosure. Taking Figure 6 as an example, the microlens 622 has an axis AX, and the axis AX is offset outward from the geometric center GC of the sub-array SA. In this configuration, the sensing range (e.g., the range of the light ray L) can be larger.

[0058] Reference Figure 7 。 Figure 7 is a schematic diagram of a fingerprint identification system 700 with multiple microlenses 722 offset inward illustrated in accordance with some embodiments of the present disclosure. Taking Figure 7 as an example, the microlens 722 has an axis AX, and the axis AX is offset inward from the geometric center GC of the sub-array SA. In this configuration, the light ray L is concentrated in a smaller sensing range and the sampling frequency can be increased. Therefore, the signal-to-noise ratio (SNR) of the fingerprint identification system 700 can be improved.

[0059] Reference Figure 8 。 Figure 8 is a schematic diagram of a sensing circuit 800 illustrated in accordance with some embodiments of the present disclosure. The sensing circuit 800 is coupled to the sensing pixel 1211. For Figure 8 example, the sensing circuit 800 includes a readout circuit 810 and an image processing circuit 820. The image processing circuit 820 is coupled to the readout circuit 810. The image processing circuit 820 includes a sub-image data generator 822, an offset module 824, and an image combining module 826. The offset module 824 is coupled to the sub-image data generator 822 and the image combining module 826. In some embodiments, the sub-image data generator 822, the offset module 824, and the image combining module 826 can be implemented using hardware (e.g., circuits), software, firmware, or a combination thereof.

[0060] Reference Figures 1 to 3C and Figure 8 。The readout circuit 810 reads the sensing signal SS of the sensing pixel 1211 of the light field sensor 120 under the display. The image processing circuit 820 reconstructs an image of an object (e.g., fingerprint FP) based on the sensing signal SS. The sensing signal SS represents an image of an object (e.g., fingerprint FP) captured by the light field image sensor 121 from different directions.

[0061] Reference Figure 9 。 Figure 9 is a flowchart of an image reconstruction method 900 illustrated in accordance with some embodiments of the present disclosure. The image reconstruction method 900 is Figure 8 executed by the image processing circuit 820 and the image reconstruction method 900 is used to reconstruct an image. The image reconstruction method 900 includes operations S910, S920, S930, and S940.

[0062] In operation S910, Figure 8 the sub-image data generator 822 in Figure 10 obtains an initial image of an object (e.g., fingerprint FP) and generates a plurality of sub-images based on the initial image. For example, the readout circuit 810 reads the sensing signal SS from the sensing pixel 1211, and then the sub-image data generator 822 groups the sensing signals SS of the corresponding sensing pixels 1211 in the sub-arrays SA1 - SAN to generate sub-images of the object (e.g., fingerprint FP). The corresponding sensing pixels 1211 in the sub-arrays SA1 - SAN are defined as the sensing pixels 1211 located at the same position in the sub-arrays SA1 - SAN, as Figure 10 shown. Figure 10 is a schematic diagram of the sub-arrays SA1 - SAN of the sensing pixel 1211 illustrated in accordance with some embodiments of the present disclosure.

[0063] For Figure 10For example, each sensing pixel 1211 is represented by (a, b), where "a" represents the sub-array SA1-SAN in which this sensing pixel 1211 is located, and "b" represents a sensing pixel 1211 in a sub-array SA1-SAN. If each sub-array SA1-SAN contains n×m (e.g., 5×5) sensing pixels 1211, the initial image obtained by the light field image sensor 121 contains multiple n×m matrices, where "n" represents the columns in each matrix and "m" represents the rows in each matrix. Each matrix contains elements a ij , where "i" represents the column of this element and is an integer from 1 to n, and "j" represents the row of this element and is an integer from 1 to m.

[0064] In this example, each sub-array SA1-SAN contains 5×5 sensing pixels 1211. Therefore, the initial image obtained by the light field image sensor 121 contains multiple 5×5 matrices, and each matrix corresponds to a microlens. Each matrix contains elements a ij , where "i" represents the column of this element and is an integer from 1 to 5, and "j" represents the row of this element and is an integer from 1 to 5.

[0065] The sensing pixels 1211 with the same b value in the sub-arrays SA1-SAN are defined as the sensing pixels 1211 at the same position in the sub-arrays SA1-SAN, and the sensing pixels 1211 with the same b value are grouped together to generate a sub-image of an object (e.g., fingerprint FP), as Figures 11 to 13 shown. Since b is an integer from 1 to n×m, n×m sub-images can be obtained. In other words, the number of generated sub-images is equal to the product of n and m. In this example, 5×5 sub-images can be obtained.

[0066] Refer to Figure 11 , 12 and Figure 13 . Figures 11 to 13 is a schematic diagram of sub-images MSUB1-MSUB2 and MSUB25 of an object (e.g., fingerprint FP) illustrated according to some embodiments of the present disclosure. For Figure 11 example, the sub-image MSUB1 is generated based on the initial image and extracts the elements a ij in each matrix that have the same i value and j value (e.g., i is 1 and j is 1). For Figure 12 example, the sub-image MSUB2 is generated based on the initial image and extracts the elements a ij in each matrix that have the same i value and j value (e.g., i is 1 and j is 2). For Figure 13 example, the sub-image MSUB25 is generated based on the initial image and extracts the elements a ijand generated. Based on the same rule, 5×5 sub-images of an object (e.g., fingerprint FP) can be obtained, as Figure 14 shown. Figure 14 is a schematic diagram of sub-images MSUB1-MSUB25 illustrated according to some embodiments of the present disclosure.

[0067] Refer to Figure 15 . Figure 15 is a schematic diagram of a plurality of displacement amounts illustrated according to some embodiments of the present disclosure. In Figure 9 operations S920 and S930, Figure 8 the offset module 824 in obtains the displacement amounts of sub-images MSUB1-MSUB25 of an object (e.g., fingerprint FP), and then uses these displacement amounts to correct the sub-images MSUB1-MSUB25 of the object (e.g., fingerprint FP) to generate corrected sub-images.

[0068] For example, Figure 1 the object (e.g., fingerprint FP) in is captured by the under-display light field sensor 120 in the aforementioned sub-images MSUB1-MSUB25. The position of the object (e.g., fingerprint FP) in the sub-images MSUB1-MSUB25 can be determined. The position of the finger F in one of the sub-images MSUB1-MSUB25 is determined as the reference position, and then the displacement amounts of each sub-image MSUB1-MSUB25 can be obtained based on this reference position, as Figure 15 shown. If a displacement amount is large, it means the object (e.g., fingerprint FP) is relatively close, and if a displacement amount is small, it means the object (e.g., fingerprint FP) is relatively far away. In order to obtain the absolute displacement amount, a reference value can be designed or calculated, or the reference value can be obtained by executing a calibration program and a comparison program before leaving the factory. After obtaining the displacement amounts of the sub-images MSUB1-MSUB25, the offset module 824 offsets each sub-image MSUB1-MSUB25 using the corresponding displacement amount to correct the sub-images MSUB1-MSUB25.

[0069] Refer to Figure 16 and Figure 17 . Figure 16 is a schematic diagram of the combined image CI illustrated according to some embodiments of the present disclosure. Figure 17 is a flowchart of an image combining method 1700 illustrated according to some embodiments of the present disclosure. In Figure 9 operation S940, Figure 8 the image combining module 826 in reconstructs the corrected sub-images by combining or superimposing the offset sub-images of the object (e.g., fingerprint FP) and then generates a reconstructed image. Taking Figure 17For example, the image combination method 1700 includes operations S1710, S1720, S1740, and S1750. Operation S1710 is used to divide the initial image into sub-images MSUB1-MSUB25 according to the rules discussed above. Operation S1720 is used to obtain the displacement amounts of the sub-images MSUB1-MSUB25, as Figure 14 shown. Operation S1740 is used to offset the sub-images MSUB1-MSUB25 according to the corresponding displacement amounts, as Figure 15 shown. Operation S1750 is used to combine the offset sub-images MSUB1-MSUB25 to generate Figure 16 the combined image CI in

[0070] and further generate a reconstructed image of the object (e.g., fingerprint FP). Figure 18 . Figure 18 is a flowchart of an image combination method 1800 illustrated according to some embodiments of the present disclosure. For Figure 18 example, the image combination method 1800 includes operations S1810, S1820, S1830, S1840, and S1850. Operations S1810 and S1820 are similar to Figure 17 operations S1710 and S1720 in

[0071] Operation S1830 is used to perform image processing on each of the sub-images MSUB1-MSUB25. The image processing is, for example, reducing the noise of the sub-images MSUB1-MSUB25, enhancing the contrast of the sub-images MSUB1-MSUB25, adjusting the brightness of the sub-images MSUB1-MSUB25, or filtering frequency components (e.g., low-frequency components) from the sub-images MSUB1-MSUB25, or a combination of the above. Then, operation S1840 is used to offset the processed sub-images according to the corresponding displacement amounts. Then, operation S1850 is used to combine the processed and offset sub-images MSUB1-MSUB25 to generate the combined image CI and further generate a reconstructed image of the object (e.g., fingerprint FP).

[0072] In summary, the under-display light field sensor, the device having the under-display light field sensor, and the image reconstruction method of the present disclosure can reduce the influence caused by the display panel structure.

[0073] Various functional elements and blocks have been disclosed herein. For those of ordinary skill in the art, the functional blocks can be implemented by circuits (either dedicated circuits or general-purpose circuits operating under the control of one or more processors and encoded instructions), which generally include transistors or other circuit elements for controlling the operation of electrical circuits corresponding to the functions and operations described herein. Further understanding is that, generally, the specific structure and interconnection of circuit elements can be determined by a compiler, such as a Register Transfer Language (RTL) compiler. The Register Transfer Language compiler operates on a script that is quite similar to assembly language code and compiles the script into a form for layout or fabrication of the final circuit.

[0074] Although the present disclosure has been disclosed as above in embodiments, it is not intended to limit the present disclosure. Any person of ordinary skill in the art can make various modifications and refinements without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the scope defined by the appended claims.

Claims

1. A display bottom light field sensor, characterized in that, For sensing a fingerprint or a touch located above or near a display panel, the under-display light field sensor includes: An optical field image sensor including a plurality of sub-arrays of sensing pixels located under the display panel, wherein the sensing pixels in the same sub-array among the sub-arrays are used to capture a plurality of images of an object located above or near the display panel by sensing light in a plurality of different directions; A plurality of microlenses located above the sub-arrays of the sensing pixels, each of the microlenses corresponding to one of the sub-arrays of the sensing pixels; An imaging lens located above the microlenses and disposed under the display panel, wherein the sensing pixels capture the images of the object by sensing light in the different directions passing through the display panel and the imaging lens; A plurality of light shielding structures, each of the light shielding structures being disposed between two adjacent microlenses among the microlenses; and A sensing circuit for generating a plurality of sub-images based on the sensing signals of the sensing pixels and combining or superimposing the plurality of offset sub-images of the object to reconstruct a reconstructed image of the object.

2. The display bottom light field sensor according to claim 1, wherein Wherein the sensing circuit includes: A readout circuit for reading out the sensing signals of the sensing pixels, wherein the sensing signals represent the images captured by the optical field image sensor from the different directions; and An image processing circuit for generating the sub-images based on the sensing signals of the sensing pixels and combining or superimposing the offset sub-images of the object to reconstruct the reconstructed image of the object.

3. The display bottom light field sensor according to claim 2, characterized in that, Wherein the image processing circuit includes: A sub-image data generator for grouping the sensing signals of the corresponding sensing pixels in the sub-arrays to generate the sub-images, wherein the corresponding sensing pixels in the sub-arrays are defined as the sensing pixels located at the same position in the sub-arrays and are grouped into one of the sub-images; An offset module for offsetting the sub-images by a plurality of displacement amounts; and An image combining module for reconstructing an image data by combining or superimposing the offset sub-images of the object.

4. The display bottom light field sensor according to claim 1, wherein Wherein each of the microlenses has an axis that is offset from the geometric center of a corresponding one of the sub-arrays of the sensing pixels.

5. The display bottom light field sensor according to claim 1, characterized in that, Wherein each of the microlenses has an axis that is aligned with the geometric center of a corresponding one of the sub-arrays of the sensing pixels.

6. The display bottom light field sensor according to claim 1, characterized in that, Wherein the under-display light field sensor is implemented as a fingerprint sensor.

7. A device with an under-display light field sensor, characterized in that, Wherein an under-display light field sensor in the device is used to sense a fingerprint or a touch and the device includes: A display panel including an opposite display surface and a back surface, wherein the under-display light field sensor includes: A plurality of sub-arrays of sensing pixels facing the back surface; A plurality of microlenses located between the sub-arrays of the sensing pixels and the back surface, each of the microlenses corresponding to one of the sub-arrays of the sensing pixels, The sensing pixels in the same sub-array among the sub-arrays are used to capture multiple images of an object by sensing light in multiple different directions, and the object is located above or near the display panel; An imaging lens, located above the microlenses and disposed under the display panel, wherein the sensing pixels are used to capture the images of the object by sensing the light in the different directions penetrating the display panel and the imaging lens; Multiple light shielding structures, each of the light shielding structures being disposed between two adjacent microlenses among the microlenses; and A sensing circuit, configured to generate multiple sub-images based on multiple sensing signals of the sensing pixels and combine or superimpose multiple offset sub-images of the object to reconstruct a reconstructed image of the object.

8. The device with an under-display light field sensor according to claim 7, wherein, Wherein the sensing circuit includes: A readout circuit, configured to read out the sensing signals of the sensing pixels, wherein the sensing signals represent the images captured by a light field image sensor in the light field sensor under the display from the different directions; and An image processing circuit, configured to generate the sub-images based on the sensing signals of the sensing pixels and combine or superimpose the offset sub-images of the object to reconstruct the reconstructed image of the object.

9. The device with an under-display light field sensor according to claim 8, wherein, Wherein the image processing circuit includes: A sub-image data generator, configured to group the sensing signals of the corresponding sensing pixels in the sub-arrays to generate the sub-images, wherein the corresponding sensing pixels in the sub-arrays are defined as the sensing pixels located at the same position in the sub-arrays and are grouped into one of the sub-images; An offset module, configured to offset the sub-images by multiple displacement amounts; and An image combining module, configured to combine or superimpose the offset sub-images of the object to reconstruct an image data.

10. An image reconstruction method, characterized in that, A light field sensor under the display is configured to sense a fingerprint or a touch, the fingerprint or the touch being located above or near a display panel, the light field sensor under the display includes multiple sub-arrays of multiple sensing pixels, multiple microlenses, an imaging lens, and multiple light shielding structures, the sensing pixels being located under the display panel, the imaging lens being located above the microlenses and disposed under the display panel, each of the light shielding structures being disposed between two adjacent microlenses among the microlenses, wherein the sensing pixels in the same sub-array among the sub-arrays are used to capture multiple images of an object by sensing the light in the different directions penetrating the display panel and the imaging lens, the object being located above or near the display panel, the microlenses being located above the sub-arrays of the sensing pixels, each of the microlenses corresponding to one of the sub-arrays of the sensing pixels, and the method includes: Obtaining an initial image and generating multiple sub-images of the object based on the initial image; Obtaining multiple corresponding displacement amounts of the sub-images of the object; Correcting the sub-images of the object by using the corresponding displacement amounts to generate multiple corrected sub-images; and Reconstructing the corrected sub-images to generate a reconstructed image.

11. The image reconstruction method according to claim 10, characterized in that, Wherein the initial image includes a plurality of matrices, each of the matrices including n×m elements a ij , where n represents the columns of each of the matrices, m represents the rows of each of the matrices, i represents the column of the n×m elements a ij and is an integer from 1 to n, and j represents the row of the n×m elements a ij and is an integer from 1 to m. Generating the sub-images of the object from the initial image includes extracting the n×m elements a with the same i value and the same j value from each of the matrices ij to generate one of the sub-images, where the number of the sub-images of the object is the same as the product of n and m.

12. The image reconstruction method according to claim 10, wherein Further includes: Process at least one of the sub-images of the object before superimposing the corrected sub-images to generate the reconstructed image.

13. The image reconstruction method according to claim 12, wherein, Wherein processing at least one of the sub-images of the object includes reducing noise, enhancing contrast, adjusting brightness, filtering some frequency components in the signal, or any combination of the above.

14. The image reconstruction method according to claim 10, wherein, Wherein generating the sub-images of the object from the initial image includes grouping a plurality of sensing signals of corresponding sensing pixels in the sub-arrays to generate the sub-images of the object, wherein the corresponding sensing pixels in the sub-arrays are defined as the sensing pixels located at the same position in the sub-arrays and are grouped into one of the sub-images.

15. The image reconstruction method according to claim 14, wherein Wherein the sensing pixels of the sub-arrays are represented by (a, b), where a represents the sub-arrays and b represents a plurality of corresponding sensing pixels in the sub-arrays, wherein the corresponding sensing pixels having the same b value are defined as being at the same position in the sub-arrays, and wherein generating the sub-images of the object from the initial image includes grouping the sensing signals of the corresponding sensing pixels having the same b value to generate one of the sub-images of the object.

16. The image reconstruction method according to claim 10, wherein Wherein correcting the sub-images of the object using the corresponding displacement amounts to generate the corrected sub-images includes offsetting the sub-images of the object by the corresponding displacement amounts.

17. The image reconstruction method according to claim 16, wherein, Wherein reconstructing the corrected sub-images includes superimposing or combining the offset sub-images of the object.

Images