Under-screen camera system and method of operating the same

Abstract

The present application provides an under-screen camera system and an operating method thereof. The operating method of the under-screen camera system includes: providing raw data by a pixel array; generating color information according to a plurality of color filters and the raw data; generating first narrow-band information according to a plurality of first narrow-band filters and the raw data, wherein a spectral line width of the first narrow-band filters is between 5 nm and 70 nm; reconstructing edge information according to one of a plurality of diffraction patterns and the first narrow-band information, wherein the diffraction patterns are provided by a database unit of a point spread function; and combining the edge information and the color information to obtain an image. Since the combination of the first model unit, the first narrow-band filters, and the database unit of the point spread function can solve the diffraction problem, thereby obtaining an image with higher resolution, the efficiency of the under-screen camera system is improved.

Description

Technical Field

[0001] This invention relates to an under-display camera system and a method for operating the under-display camera system. Background Technology

[0002] Generally, under-display camera systems typically consist of a display and an image sensor, with the display positioned above the image sensor. When light passes through the display, the image generated by the image sensor suffers from diffraction. This diffraction severely blurs the image. Recently, deep neural networks combined with the point spread function (PSF) have been used to recover images blurred by diffraction. The PSF is defined by coherent light rays (such as narrow-band rays). However, natural light is non-coherent. Therefore, when all wavelengths of non-coherent light are added together, specific information from the PSF may be lost. This lost information makes image recovery more difficult and ultimately leads to a decrease in the resolution of the under-display camera system. Therefore, identifying this lost information can improve the performance of the under-display camera system to recover better images. Summary of the Invention

[0003] One technical aspect of the present invention is an under-display camera system.

[0004] According to one embodiment of the present invention, an under-display camera system includes a pixel array, a dot spread function database unit, a first model unit, and a second model unit. The pixel array is configured to provide raw data. The pixel array includes a plurality of color filters and a plurality of first narrowband filters. The color filters are respectively disposed on a plurality of first photodiodes and configured to generate color information based on the raw data. The plurality of first narrowband filters are respectively disposed on a plurality of second photodiodes and configured to generate first narrowband information based on the raw data. The first spectral linewidth of the first narrowband filters is between 5 nanometers and 70 nanometers. The dot spread function database unit is configured to store a plurality of diffraction patterns. The first model unit receives the first narrowband information from the pixel array and is electrically connected to the dot spread function database unit. The first model unit is configured to reconstruct edge information based on one of the diffraction patterns provided by the dot spread function database unit and the first narrowband information. The second model unit is electrically connected to the first model unit and the pixel array. The second model unit is configured to combine the edge information and color information to obtain an image.

[0005] In one embodiment of the present invention, the first narrowband filter corresponds to a first single wavelength, and the first single wavelength is between 400 nanometers and 1000 nanometers. The spectral linewidth of the color filter is between 150 nanometers and 200 nanometers. The area ratio of the first narrowband filter in the pixel array is between 6.25% and 25%.

[0006] In one embodiment of the present invention, two adjacent first narrowband filters are merged, and a color filter surrounds the first narrowband filter.

[0007] In one embodiment of the present invention, each of the first narrowband filters includes a first portion and a second portion disposed on the first portion. The first spectrum corresponding to the first portion is the same as a corresponding spectrum in a color filter, and the second spectrum corresponding to the second portion is different from the first spectrum.

[0008] In one embodiment of the present invention, the pixel array further includes a plurality of second narrowband filters. The second narrowband filters are configured to generate second narrowband information based on the original data, wherein the first model unit further reconstructs edge information based on another element in the diffraction pattern and the second narrowband information, and the second spectral linewidth of the second narrowband filters is between 5 nanometers and 70 nanometers, and the second spectral linewidth is different from the first spectral linewidth.

[0009] In one embodiment of the present invention, the pixel array further includes a plurality of second narrowband filters. The second narrowband filters are configured to generate second narrowband information based on the original data, wherein the first model unit further reconstructs edge information based on another of the diffraction patterns and the second narrowband information, and one of the color filters is located between the first narrowband filter and the second narrowband filter.

[0010] In one embodiment of the present invention, the pixel array further includes a plurality of second narrowband filters. The second narrowband filters are configured to generate second narrowband information based on the original data, wherein the first model unit further reconstructs edge information based on another element in the diffraction pattern and the second narrowband information, the second narrowband filters are adjacent to the first narrowband filters, and a color filter surrounds the second narrowband filters.

[0011] One technical aspect of the present invention is an operation method for an under-display camera system.

[0012] According to an embodiment of the present invention, an operation method of an under-display camera system includes: providing raw data from a pixel array; generating color information based on a plurality of color filters and the raw data, wherein the color filters are respectively disposed on a plurality of first photodiodes of the pixel array; generating first narrowband information based on a plurality of first narrowband filters and the raw data, wherein the first narrowband filters are respectively disposed on a plurality of second photodiodes of the pixel array, and the spectral linewidth of the first narrowband filters is between 5 nanometers and 70 nanometers; reconstructing edge information based on one of a plurality of diffraction patterns and the first narrowband information, wherein the diffraction patterns are provided by a database unit of point spread function; and combining the edge information and the color information to obtain an image.

[0013] In one embodiment of the present invention, the above method further includes generating second narrowband information through a plurality of second narrowband filters of a pixel array and raw data, wherein reconstructing edge information further includes reconstructing edge information based on another of the diffraction patterns and the second narrowband information.

[0014] In one embodiment of the present invention, the above method establishes a database unit of point spread function by means of the following steps: marking each of the diffraction patterns, wherein the diffraction patterns are generated by multiple single wavelengths of the display; and storing the diffraction patterns.

[0015] In one embodiment of the present invention, the above method reconstructs edge information through a first model unit and trains the first model unit through the following steps: providing a first image generated by a single wavelength passing through the display, a second image related to the formula of the point spread function, and a third image generated by a single wavelength not passing through the display; combining the first image and the second image to reconstruct a fourth image; comparing the third image and the fourth image; and adjusting multiple fitting parameters of the formula if the fourth image is not similar to the third image.

[0016] In one embodiment of the present invention, the above method obtains an image through a second model unit and trains the second model unit through the following steps: providing a fourth image, a fifth image generated by multiple wavelengths passing through the display, and a sixth image generated by wavelengths not passing through the display; combining the fourth image and the fifth image to reconstruct a seventh image; and comparing the sixth image and the seventh image.

[0017] In the above embodiments of the present invention, the pixel array includes a plurality of first narrowband filters configured to generate first narrowband information. A first model unit can reconstruct edge information based on one of a plurality of diffraction patterns provided by a database unit of point spread functions and the first narrowband information. Furthermore, a second model unit can combine the edge information with color information provided by a plurality of color filters to obtain an image. Since the combination of the first model unit, the plurality of first narrowband filters, and the database unit of point spread functions can solve the diffraction problem, thereby obtaining a higher resolution image, the performance of the under-display camera system is improved. Attached Figure Description

[0018] An embodiment of the invention can be best understood by reading in conjunction with the accompanying drawings and the following detailed description. It should be emphasized that, according to standard industry practice, the various features are not drawn to scale and are for illustrative purposes only. In fact, the dimensions of the various features may be arbitrarily increased or decreased for clarity of explanation.

[0019] Figure 1 A block diagram illustrating an under-display camera system according to an embodiment of the present invention is shown.

[0020] Figure 2AA top view of a pixel array according to an embodiment of the present invention is shown.

[0021] Figure 2B Draw Figure 2A A cross-sectional view of the pixel array along line segment 2B-2B.

[0022] Figure 2C A top view illustrating a pixel array according to another embodiment of the present invention is shown.

[0023] Figure 2D Draw Figure 2C A 2D-2D cross-sectional view of the pixel array along the line segment.

[0024] Figures 2E to 2H A top view illustrating a pixel array according to some embodiments of the present invention is shown.

[0025] Figures 3A to 3C A top view illustrating a pixel array according to another embodiment of the present invention is shown.

[0026] Figures 4A to 4L A top view illustrating a pixel array according to another embodiment of the present invention is shown.

[0027] Figure 5 A flowchart illustrating an operation method of an under-display camera system according to an embodiment of the present invention is shown.

[0028] Figure 6 The principle of diffraction patterns corresponding to different wavelengths according to an embodiment of the present invention is illustrated.

[0029] Figure 7 The diagram illustrates the difference in diffraction patterns between narrow-band and broadband rays according to an embodiment of the present invention.

[0030] Figure 8 A schematic diagram illustrating the operation of an under-screen camera system according to an embodiment of the present invention is shown.

[0031] Figure 9 A flowchart illustrating a method for creating a database unit according to an embodiment of the present invention is shown.

[0032] Figure 10 A flowchart illustrating a method for training a first model unit according to an embodiment of the present invention is shown.

[0033] Figure 11 A flowchart illustrating a method for training a second model unit according to an embodiment of the present invention is shown.

[0034] The symbols in the attached diagram are briefly explained as follows:

[0035] 10: Under-display camera system; 100: Pixel array; 100a, 100c, 100d, 100e, 100f: Pixel array; 110: Semiconductor substrate; 112: Second photodiode; 114: First photodiode; 121: First section; 122: First narrowband filter; 123: Second section; 124: Color filter; 125: Grille; 126: Second narrowband filter; 130: Microlens layer; 132: Top surface; 134: Microlens; 140: Protective layer; 150: Shielding element; 200: Database unit; 300: First model unit; 300a, 300b, 3 00c: Pixel array; 400: Second model unit; 400a, 400b, 400c, 400d, 400e, 400f, 400g, 400h, 400i, 400j, 400k, 400l: Pixel array; 510: Step; 520: Step; 530: Step; 540: Step; 550: Step; 910: Step; 920: Step; 1010: Step; 1020: Step; 1030: Step; 1040: Step; 1110: Step; 1120: Step; 1130: Step; 2B-2B: Line segment; 2D-2D: Line segment; H: Aperture; L: Ray. Detailed Implementation

[0036] The following description of embodiments provides many different implementations, or examples, for carrying out various features of the provided object. Specific examples of elements and arrangements are described below to simplify the subject matter. Of course, these examples are merely illustrative and are not intended to be limiting. Furthermore, element symbols and / or letters may be repeated in various examples. This repetition is for simplicity and clarity and does not in itself specify the relationship between the various embodiments and / or configurations discussed.

[0037] Spatial relative terms such as “below,” “under,” “lower,” “above,” and “upper” are used herein for descriptive purposes to describe the relationship between one element or feature and another, as shown in the accompanying drawings. Spatial relative terms are intended to cover different orientations of the apparatus in use or operation, other than those shown in the accompanying drawings. The apparatus may be oriented in other ways (rotated 90 degrees or otherwise), and the spatial relative descriptors used herein shall be interpreted accordingly.

[0038] Figure 1A block diagram illustrating an under-display camera system 10 according to an embodiment of the present invention is shown. The under-display camera system 10 includes a pixel array 100, a database unit 200, a first model unit 300, and a second model unit 400. The pixel array 100 is configured to provide raw data. The database unit 200 for a point spread function is configured to store multiple diffraction patterns. The database unit 200 for the point spread function may be wavelength-dependent. For example, the point spread function may include diffraction information of a single wavelength passing through the display. The database unit 200 for the point spread function can be trained in a laboratory setting. The first model unit 300 is electrically connected to the pixel array 100, the database unit 200, and the second model unit 400. The first model unit 300 may be a deep neural network (DNN). The deep neural network can be combined with various types of neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), or other neural networks, as well as various optimization algorithms. For example, the neural network needs to be trained using sufficient diffraction images and blurred images corresponding to the point spread function. The second model unit 400 is electrically connected to the pixel array 100 and the first model unit 300. The second model unit 400 may be another deep neural network with inputs different from those of the first model unit 300.

[0039] Figure 2A A top view of a pixel array 100 according to an embodiment of the present invention is shown. Figure 2B Draw Figure 2A A cross-sectional view of pixel array 100 along line segment 2B-2B. Please refer to... Figure 1 , Figure 2A as well as Figure 2B The pixel array 100 includes a semiconductor substrate 110, a plurality of first narrowband filters 122, a plurality of color filters 124, and a microlens layer 130. The color filters 124 are respectively disposed on a plurality of first photodiodes 114 and configured to generate color information based on raw data. For example, the color filters 124 may correspond to different wavelengths (e.g., wavelengths of red, green, and blue light). The first narrowband filters 122 are respectively disposed on a plurality of second photodiodes 112 and configured to generate first narrowband information based on raw data. The spectral linewidth of the color filters 124 is between 150 nanometers (nm) and 200 nanometers (nm). Notably, the first spectral linewidth of the first narrowband filters 122 is between 5 nm and 70 nm. The first narrowband filters 122 correspond to a first single wavelength, and the first single wavelength is between 400 nm and 1000 nm. The first single wavelength is the same as the wavelength corresponding to one of the plurality of diffraction patterns.

[0040] Furthermore, the first model unit 300 receives first narrowband information from the pixel array 100. The first model unit 300 is configured to reconstruct edge information based on one of the diffraction patterns provided by the database unit 200 of the point spread function and the first narrowband information. The second model unit 400 is configured to combine the edge information and color information to obtain an image.

[0041] Specifically, the pixel array 100 includes a plurality of first narrowband filters 122 configured to generate first narrowband information. The first model unit 300 can reconstruct edge information based on one of a plurality of diffraction patterns provided by the dot spread function database unit 200 and the first narrowband information. Furthermore, the second model unit 400 can combine the edge information with color information provided by a plurality of color filters 124 to obtain an image. Since the combination of the first model unit 300, the plurality of first narrowband filters 122, and the dot spread function database unit 200 can solve the diffraction problem, thereby obtaining a higher resolution image, the performance of the under-display camera system 10 is improved.

[0042] A microlens layer 130 is disposed on the pixel array 100 and configured to transmit light L to a first narrowband filter 122 and a color filter 124. The microlens layer 130 includes a plurality of microlenses 134. The microlenses 134 overlap the first narrowband filter 122 and the color filter 124. A grid 125 is located between one of the first narrowband filters 122 and one of the color filters 124. In some embodiments, the grid 125 may be located between two adjacent color filters 124. Each of the first narrowband filters 122 includes a first portion 121 and a second portion 123 disposed on the first portion 121. A first spectrum corresponding to the first portion 121 is the same as the spectrum of one of the color filters 124, and a second spectrum corresponding to the second portion 123 is different from the first spectrum.

[0043] Furthermore, if the first spectrum of the first portion 121 corresponds to the wavelength of green light and the second spectrum of the second portion 123 corresponds to the wavelength of red light, then the first spectral linewidth of the first narrowband filter 122 can be 42 nm. If the first spectrum of the first portion 121 corresponds to the wavelength of green light and the second spectrum of the second portion 123 corresponds to the wavelength of blue light, then the first spectral linewidth of the first narrowband filter 122 can be 52 nm. In some embodiments, the pixel array 100 further includes a protective layer 140 and a shielding element 150. The protective layer 140 covers the top surface 132 of the microlens layer 130. The shielding element 150 is configured to prevent the first photodiode 114 from interfering with the second photodiode 112.

[0044] It should be understood that the component connections and functions already described will not be repeated, but will be stated separately. Other forms of pixel arrays will be described in the following description.

[0045] Figure 2C A top view of a pixel array 100a according to another embodiment of the present invention is shown. Figure 2D Draw Figure 2C A 2D-2D cross-sectional view of the pixel array 100a along line segment. Also refer to... Figure 2C and Figure 2D Pixel array 100a and Figure 2B The pixel array 100 shown differs in that there is no grid 125 between adjacent first narrowband filters 122, which improves the light absorption efficiency of the second photodiode 112. Furthermore, adjacent first narrowband filters 122 are merged.

[0046] Figures 2E to 2H A top view illustrating a pixel array according to some embodiments of the present invention is shown. Please refer to... Figure 2E The pixel array 100c also includes a plurality of second narrowband filters 126. The second narrowband filters 126 are configured to generate second narrowband information based on the raw data. The first model unit 300 (see...) Figure 1 Further, based on the database cell 200 of the point spread function (see...), Figure 1 The edge information is reconstructed from the other of the diffraction patterns provided and the second narrowband information. The second narrowband filter 126 corresponds to a second single wavelength, and the second single wavelength is the same as the wavelength corresponding to the other of the diffraction patterns. It is worth noting that the second spectral linewidth of the second narrowband filter 126 is between 5 nm and 70 nm, and the second spectral linewidth is different from the first spectral linewidth. In this embodiment, one of the color filters 124 is located between the first narrowband filter 122 and the second narrowband filter 126. Please refer to... Figure 2F Pixel array 100d and Figure 2E The difference in the pixel array 100c shown is that the second narrowband filter 126 is adjacent to the first narrowband filter 122.

[0047] Please refer to Figure 2G The pixel array 100e is an 8×8 array, and the area ratio of the first narrowband filter 122 in the pixel array 100e is the same as the area ratio of the second narrowband filter 126 in the pixel array 100e. Please refer to... Figure 2H Pixel array 100f and Figure 2GThe pixel array 100e shown differs in that there is no grid 125 between adjacent pairs of the first narrowband filter 122, and there is no grid 125 between adjacent pairs of the second narrowband filter 126. Furthermore, adjacent pairs of the first narrowband filter 122 are merged, and adjacent pairs of the second narrowband filter 126 are merged.

[0048] Figures 3A to 3C A top view illustrating a pixel array according to another embodiment of the present invention is shown. Please refer to... Figure 3A The pixel array 300a is a 6×6 matrix, and the area ratio of the first narrowband filter 122 in the pixel array 300a is 25%. Please refer to... Figure 3B The pixel array 300b includes a first narrowband filter 122 and a color filter 124. In this embodiment, the color filter 124 surrounds the first narrowband filter 122. Please refer to... Figure 3C Pixel array 300c and Figure 3B The pixel array 300b shown differs in that there is no grid 125 between adjacent first narrowband filters 122. In this embodiment, the area ratio of the first narrowband filter 122 in pixel array 300c is 11%.

[0049] Figures 4A to 4L A top view illustrating a pixel array according to another embodiment of the present invention is shown. Please refer to... Figure 4A , Figure 4A The pixel array 400a shown is an 8×8 matrix, and the area ratio of the first narrowband filter 122 in pixel array 400a is 25%. Please refer to... Figure 4B The pixel array 400b includes a first narrowband filter 122 and a second narrowband filter 126. The area ratio of the first narrowband filter 122 in the pixel array 400b is the same as the area ratio of the second narrowband filter 126 in the pixel array 400b.

[0050] Please refer to Figure 4C The pixel array 400c is an 8×8 array, and the area ratio of the first narrowband filter 122 in the pixel array 400c is 6.25%. Please refer to... Figure 4D Pixel array 400d and Figure 4C The pixel array 400c shown differs in that there is no grid 125 between adjacent first narrowband filters 122.

[0051] Please refer to Figure 4E The pixel array 400e is an 8×8 array, and the color filter 124 surrounds the first narrowband filter 122. Please refer to... Figure 4F Pixel array 400f and Figure 4EThe pixel array 400e shown is different in that there is no grid 125 between adjacent pairs of the first narrowband filter 122.

[0052] Please refer to Figure 4G The pixel array 400g is an 8×8 array, including a first narrowband filter 122 and a second narrowband filter 126. In this embodiment, a color filter 124 surrounds the first narrowband filter 122 and the second narrowband filter 126. Please refer to... Figure 4H Pixel array 400h and Figure 4G The pixel array 400g shown differs in that there is no grid 125 between adjacent pairs of the first narrowband filter 122 and between adjacent pairs of the second narrowband filter 126.

[0053] Please refer to Figure 4I The pixel array 400i is an 8×8 array and includes a first narrowband filter 122. In this embodiment, a color filter 124 surrounds the first narrowband filter 122. Please refer to... Figure 4J Pixel array 400j and Figure 4I The pixel array 400i shown is different in that there is no grid 125 between adjacent pairs of the first narrowband filter 122.

[0054] Please refer to Figure 4K The pixel array 400k is an 8×8 array and includes a first narrowband filter 122 and a second narrowband filter 126. In this embodiment, a color filter 124 is located between the first narrowband filter 122 and the second narrowband filter 126. The color filter 124 surrounds the first narrowband filter 122 and the second narrowband filter 126. Please refer to... Figure 4L Pixel array 400l and Figure 4K The pixel array 400k shown differs in that there is no grid 125 between adjacent pairs of the first narrowband filter 122 and between adjacent pairs of the second narrowband filter 126.

[0055] The following description will explain how to operate the under-screen camera system.

[0056] Figure 5A flowchart illustrating an operation method of an under-display camera system according to an embodiment of the present invention is shown. The operation method of the under-display camera system includes the following steps. First, in step 510, raw data is provided by a pixel array. Next, in step 520, color information is generated based on a plurality of color filters and the raw data, wherein the color filters are respectively disposed on a plurality of first photodiodes of the pixel array. Then, in step 530, first narrowband information is generated based on a plurality of first narrowband filters and the raw data, wherein the first narrowband filters are respectively disposed on a plurality of second photodiodes of the pixel array. Next, in step 540, the spectral linewidth of the first narrowband filter is between 5 nanometers and 70 nanometers; edge information is reconstructed based on one of a plurality of diffraction patterns and the first narrowband information, wherein the diffraction pattern is provided by a database unit of point spread function. Next, in step 550, the edge information and color information are combined to obtain an image. The above steps will be described in detail in the following description.

[0057] Figure 6 The principle of diffraction patterns corresponding to different wavelengths according to an embodiment of the present invention is illustrated. Figure 7 The diagram illustrates the difference in diffraction patterns between narrow-band and broadband rays according to an embodiment of the present invention. (See also...) Figure 6 and Figure 7 Diffraction, which reduces image resolution, is a problem in image restoration. For example... Figure 6 As shown, even using the same diffraction structure, different wavelengths can produce different diffraction patterns. For example, when light ray L passes through aperture H, light ray L can produce different diffraction patterns corresponding to red light and blue light. Figure 7 Examples of narrow-band and wide-band diffraction patterns are shown. When all wavelength information is added together, a pattern like... Figure 7 The broadband diffraction pattern shown is different from the narrow-band diffraction pattern, which loses more detail. Because the narrow-band diffraction pattern contains more information than the broadband pattern, it can help recover higher-resolution images, thus improving overall performance.

[0058] Figure 8 A schematic diagram illustrating the operation of an under-display camera system according to an embodiment of the present invention is shown. Please refer to... Figure 1 , Figure 2B as well as Figure 8 The first model unit 300 can be a deep neural network (DNN). Deep neural networks can be combined with various types of neural networks, such as convolutional neural networks (CNN), recurrent neural networks (RNN), or other neural networks, as well as various optimization algorithms. The first model unit 300 is electrically connected to the database unit 200 of the point spread function. The second model unit 400 is electrically connected to the pixel array 100 and the first model unit 300.

[0059] The pixel array 100 includes a first narrowband filter 122 configured to generate first narrowband information. A first model unit 300 can reconstruct edge information based on one of the diffraction patterns provided by a dot spread function database unit 200 and the first narrowband information. In some embodiments, the wavelength corresponding to the first narrowband filter 122 is the same as the wavelength of one of the diffraction patterns. Furthermore, a second model unit 400 can combine the edge information with color information provided by a color filter 124 to obtain an image. Since the combination of the first model unit 300, the first narrowband filter 122, and the dot spread function database unit 200 can solve the diffraction problem, thereby obtaining a higher resolution image, the performance of the under-display camera system 10 is improved.

[0060] The method also includes through Figure 2E The second narrowband filter 126 of the pixel array 100c shown generates second narrowband information based on the original data. Furthermore, reconstructing edge information also includes reconstructing edge information based on another of a plurality of diffraction patterns and the second narrowband information. Different diffraction patterns can correspond to different wavelengths. The combination of the first narrowband information and the second narrowband information allows the first model unit 300 to generate edge information with higher resolution.

[0061] Simultaneously refer to Figure 8 and Figure 9 , Figure 9 A flowchart illustrating a method for establishing a database unit 200 according to an embodiment of the present invention is shown. The method for establishing the database unit 200 includes the following steps. In step 910, each of the diffraction patterns is marked, wherein the diffraction patterns are generated by a plurality of single wavelengths passing through the display. In step 920, the diffraction patterns are stored. For example, when a single wavelength passes through the display, the diffraction pattern generated by the single wavelength is stored, and each diffraction pattern corresponds to each single wavelength. Furthermore, a first narrowband filter 122 (see...) Figure 2A The first single wavelength corresponding to the pattern is the same as the wavelength of one of the multiple diffraction patterns.

[0062] Simultaneously refer to Figure 8 and Figure 10 , Figure 10A flowchart illustrating a method for training a first model unit 300 according to an embodiment of the present invention is shown. In some embodiments, the first model unit 300 performs edge information reconstruction. The method for training the first model unit 300 includes the following steps: In step 1010, a first image generated by a single wavelength passing through a display, a second image related to a formula for a point spread function, and a third image generated by a single wavelength not passing through the display are provided. In step 1020, the first image and the second image are combined to reconstruct a fourth image. In step 1030, the third image and the fourth image are compared. In step 1040, if the fourth image is not similar to the third image, multiple fitting parameters of the formula are adjusted. The first image combined with the second image can be considered as input to the first model unit 300, and the fourth image can be considered as output to the first model unit 300. The first model unit 300 can learn to reconstruct the fourth image by combining the first image and the second image and adjusting the fitting parameters of the formula if the fourth image is not similar to the third image.

[0063] Please refer to Figure 2A , Figure 8 as well as Figure 11 , Figure 11 A flowchart illustrating a method for training a second model unit 400 according to an embodiment of the present invention is shown. In some embodiments, image acquisition is performed by the second model unit 400. The method for training the second model unit 400 includes the following steps. In step 1110, a fourth image, a fifth image generated by multiple wavelengths passing through a display, and a sixth image generated by wavelengths not passing through the display are provided. In step 1120, the fourth image and the fifth image are combined to reconstruct a seventh image. In step 1130, the sixth image and the seventh image are compared. The second model unit 400 has different inputs and outputs than the first model unit 300. For example, the inputs of the first model unit 300 are a diffraction pattern provided by the database unit 200 and first narrowband information provided by the first narrowband filter 122. The inputs of the second model unit 400 are edge information provided by the first model unit 300 and color information provided by the color filter 124. The fifth image may include color information, and the fourth image may include edge information. The second model unit 400 can reconstruct the seventh image by combining the edge information and color information, and learn how to recover the image by comparing the sixth image and the seventh image.

[0064] In summary, the pixel array includes a first narrowband filter configured to generate first narrowband information. The spectral linewidth of the first narrowband filter is between 5 nm and 70 nm. The first model unit can reconstruct edge information based on one of the diffraction patterns provided by the database unit of the dot spread function and the first narrowband information. Furthermore, the second model unit can combine the edge information with color information provided by the color filter to obtain an image. Since the combination of the first model unit, the first narrowband filter, and the database unit of the dot spread function can solve the diffraction problem and thus obtain a higher resolution image, the performance of the under-display camera system is improved.

[0065] The above description is only a preferred embodiment of the present invention, but it is not intended to limit the scope of the present invention. Any person skilled in the art can make further improvements and changes on this basis without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope defined in the claims of this application.

Claims

1. An under-display camera system, characterized in that, include: Pixel array, configured to provide raw data, wherein the pixel array includes: Multiple color filters are respectively disposed on multiple first photodiodes and configured to generate color information based on the original data; and Multiple first narrowband filters are respectively disposed on multiple second photodiodes and configured to generate first narrowband information based on the original data, wherein the first spectral linewidth of the multiple first narrowband filters is between 5 nanometers and 70 nanometers; The under-display camera system also includes: A database cell for the point spread function, configured to store multiple diffraction patterns; A first model unit receives the first narrowband information from the pixel array and is electrically connected to the database unit of the point spread function, wherein the first model unit is configured to reconstruct edge information based on one of the plurality of diffraction patterns provided by the database unit of the point spread function and the first narrowband information; and A second model unit is electrically connected to the first model unit and the pixel array, wherein the second model unit is configured to combine the edge information and the color information to obtain an image.

2. The under-display camera system according to claim 1, wherein the plurality of first narrowband filters correspond to a first single wavelength, and the first single wavelength is between 400 nm and 1000 nm, the spectral linewidth of the plurality of color filters is between 150 nm and 200 nm, and the area ratio of the plurality of first narrowband filters in the pixel array is between 6.25% and 25%.

3. The under-display camera system according to claim 1, wherein two adjacent of the plurality of first narrowband filters are merged, and the plurality of color filters surround the plurality of first narrowband filters.

4. The under-display camera system according to claim 1, wherein each of the plurality of first narrowband filters includes a first portion and a second portion disposed on the first portion, the first spectrum corresponding to the first portion is the same as the spectrum of one of the plurality of color filters, and the second spectrum corresponding to the second portion is different from the first spectrum.

5. The under-display camera system according to claim 1, wherein the pixel array further comprises: A plurality of second narrowband filters are configured to generate second narrowband information based on the original data, wherein the first model unit further reconstructs the edge information based on another of the plurality of diffraction patterns and the second narrowband information, wherein the second spectral linewidth of the plurality of second narrowband filters is between 5 nanometers and 70 nanometers, and the second spectral linewidth is different from the first spectral linewidth.

6. The under-display camera system according to claim 1, wherein the pixel array further comprises: A plurality of second narrowband filters are configured to generate second narrowband information based on the original data, wherein the first model unit further reconstructs the edge information based on another of the plurality of diffraction patterns and the second narrowband information, and one of the plurality of color filters is located between the plurality of first narrowband filters and the plurality of second narrowband filters.

7. The under-display camera system according to claim 1, wherein the pixel array further comprises: A plurality of second narrowband filters are configured to generate second narrowband information based on the original data, wherein the first model unit further reconstructs the edge information based on another of the plurality of diffraction patterns and the second narrowband information, the plurality of second narrowband filters being adjacent to the plurality of first narrowband filters, and the plurality of color filters surrounding the plurality of second narrowband filters.

8. A method for operating an under-display camera system, characterized in that, include: The raw data is provided by the pixel array; Color information is generated based on multiple color filters and the original data, wherein the multiple color filters are respectively disposed on multiple first photodiodes of the pixel array; First narrowband information is generated based on multiple first narrowband filters and the original data, wherein the multiple first narrowband filters are respectively disposed on multiple second photodiodes of the pixel array, and the spectral linewidth of the multiple first narrowband filters is between 5 nanometers and 70 nanometers. Edge information is reconstructed based on one of a plurality of diffraction patterns and the first narrow-frequency information, wherein the plurality of diffraction patterns are provided by a database unit of a point spread function; as well as The image is obtained by combining the edge information and the color information.

9. The method according to claim 8, further comprising: The second narrowband information is generated by a plurality of second narrowband filters of the pixel array and the original data, wherein reconstructing the edge information further includes reconstructing the edge information based on another of the plurality of diffraction patterns and the second narrowband information.

10. The method of claim 8, wherein the database cell of the point spread function is established by the following steps: Mark each of the plurality of diffraction patterns, wherein the plurality of diffraction patterns are generated respectively by a plurality of single wavelengths of a display; and Store the multiple diffraction patterns.

11. The method of claim 8, wherein the edge information is reconstructed by a first model unit, and the first model unit is trained by the following steps: Provides a first image generated by a single wavelength passing through the display, a second image using the formula of the point spread function, and a third image generated by the single wavelength not passing through the display, the second image including diffraction information of the single wavelength passing through the display; The fourth image is reconstructed by combining the first image and the second image; Compare the third image with the fourth image; and If the fourth image is not similar to the third image, adjust multiple fitting parameters of the formula.

12. The method of claim 11, wherein the image is obtained by a second model unit, and the second model unit is trained by the following steps: Provide the fourth image, a fifth image generated by multiple wavelengths passing through the display, and a sixth image generated by the multiple wavelengths not passing through the display; The seventh image is reconstructed by combining the fourth image and the fifth image; and Compare the sixth image with the seventh image.

Images