Under-screen camera device based on incoherent imaging synthesis, imaging method, display panel and electronic equipment

Abstract

The invention discloses an under-screen camera device based on incoherent imaging synthesis, an imaging method, a display panel and electronic equipment. The device comprises a camera lens group, an image sensor, a control device and an optical coherence regulation and control assembly, the optical coherence regulation and control assembly enables a plurality of incoherent sub-apertures to be formed in a pupil of the camera lens group; the image sensor receives imaging data of the incoherent sub-aperture, and then the control device outputs an image. The coherence of light entering a pupil is destroyed through an optical coherence regulation and control assembly, the intensity of an image on an image sensor is the algebraic sum of the intensity of an image corresponding to each incoherent sub-aperture, and a system comprehensive OTF is the algebraic sum of an OTF corresponding to each sub-aperture, so that the pixel structure corresponding to each incoherent sub-aperture is independently designed, and the coherence of the light entering the pupil is reduced. When the control device processes the imaging data, limitation of partial frequency information loss on a recovery algorithm can be broken through, mutual exclusion of display and imaging functions on a pixel structure is reduced, and the problem of image blurring caused by diffraction of a periodic light-transmitting structure is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an under-screen camera device based on incoherent imaging synthesis, an imaging method, a display panel and an electronic device. BACKGROUND

[0002] In order to pursue a high screen ratio of a mobile phone and even a full-screen mobile phone, a front camera needs to be placed under a display panel to form an under-screen camera. In order to simultaneously realize normal display and imaging functions, the display panel above the under-screen camera is modified into a semi-transparent structure. Specifically, each pixel of the display panel simultaneously includes a light-emitting area, an opaque area (hereinafter referred to as a "light-blocking area") such as a metal trace, and a transparent aperture area (hereinafter referred to as a "light-transmitting area"). The light-blocking area is used for display function, and the light-transmitting area is used for imaging function. The periodic light-transmitting-light-blocking structure is in the imaging light path of the front camera, causing a significant diffraction effect, which blurs the imaging of the front camera. At present, there are mainly two means to improve the imaging blur problem of the under-screen camera caused by diffraction.

[0003] The first is to optimize the light-transmitting area structure to reduce optical diffraction. However, the shape, arrangement, and proportion of the pixel area (hereinafter referred to as "aperture ratio") of the light-transmitting area are constrained by the display function. For example, the display panel needs a high pixel density, which means that the period of the light-transmitting area should be small, and the diffraction effect is stronger. For another example, the area of the light-blocking area needs to accommodate the optoelectronic devices (including OLED light-emitting materials, thin film transistors, etc.) required for normal display, so the aperture ratio of the light-transmitting area should be small, which also means stronger diffraction. Therefore, the optimization of the light-transmitting area structure is constrained by the display function, and it is impossible to design a pixel structure that effectively reduces the diffraction effect.

[0004] The second means to improve image blur is algorithm post-processing, and the basic idea is to deconvolve the blurred image according to the known OTF (Optical Transfer Function). The OTF can be obtained through the exit pupil structure including the display panel structure. The display panel determines the exit pupil of the imaging system as a periodic light-transmitting structure, so its OTF (i.e. the autocorrelation of the exit pupil) contains a large number of zero values. When deconvolution is performed, the OTF in the denominator causes divergence, and the deconvolution fails, while the introduction of Wiener filtering and other algorithms to suppress divergence will reduce the quality of the restored image. Compared with the classic deconvolution algorithm, the image restoration algorithm based on deep learning can improve the image quality to a certain extent, but for the frequency components near the zero point in the OTF, the restoration effect is still not good, which is limited by the generalization problem inherent in deep learning. SUMMARY

[0005] The present application proposes an under-screen camera device based on incoherent imaging synthesis, which is used to effectively solve the technical problem of image blur caused by diffraction of periodic light-transmitting structures in related technologies, and can minimize the occurrence of mutual exclusion of pixel structures between display and imaging functions.

[0006] In a first aspect, an embodiment of the present application provides an under-screen camera device based on incoherent imaging synthesis, which is arranged on the backlight side of a display panel, including: a camera lens group, an image sensor, a control device, and an optical coherence control component;

[0007] The optical coherence regulating component is used to form at least two incoherent sub-apertures in the pupil of the camera lens group;

[0008] The image sensor is disposed on the image side of the camera lens assembly, and the image sensor is configured to receive imaging data of each of the incoherent sub-apertures;

[0009] The control device is configured to receive and process the signal from the image sensor to output an image.

[0010] Furthermore, the optical coherence regulation component includes a first polarizer and a second polarizer, and the polarization directions of the first polarizer and the second polarizer are perpendicular to each other, so that the light between the two incoherent sub-apertures has orthogonal vibration directions.

[0011] Furthermore, the optical coherence regulation component includes at least two color filters, each of which has a non-overlapping transmission spectrum, so that the light between each of the incoherent sub-apertures has a different frequency.

[0012] Furthermore, the optical coherence regulation component includes a plurality of time domain masks, each of which is configured to be turned on at a preset time and allow light to pass through, so that the light between each of the incoherent sub-apertures has an uncertain phase relationship.

[0013] Furthermore, the camera lens group and the image sensor are provided in a matching manner, and the number of the camera lens group and the image sensor is 1;

[0014] And / or, the optical coherence regulating component is arranged on the object side of the camera lens assembly, and one or more optical coherence regulating components are arranged for superposition;

[0015] And / or, the under-screen camera device based on incoherent imaging synthesis also includes an under-screen pixel area, which is arranged on the display panel and corresponds to the camera lens group, and the under-screen pixel area is divided into pixel areas corresponding to each of the incoherent sub-apertures, and each pixel structure of each pixel area has a preset pixel density and / or a preset aperture ratio.

[0016] From the above technical solutions, the present application embodiments have at least the following beneficial effects: the light coherence regulation component destroys the coherence of the light entering the camera lens group pupil, so that the camera lens group pupil is divided into multiple incoherent sub-apertures, and when the images formed by each incoherent sub-aperture are superimposed on the image sensor, only the intensity of each image is superimposed, and the complex amplitude of each image containing phase information is not superimposed. The intensity of the image on the image sensor is the algebraic sum of the intensities of the images corresponding to each incoherent sub-aperture, and the system comprehensive optical transfer function is the algebraic sum of the optical transfer functions corresponding to each incoherent sub-aperture. Therefore, by independently designing the pixel structure of the display panel above each incoherent sub-aperture, the synthesized optical transfer function no longer contains a large number of zeros, so that the control device can break through the limitation of the loss of partial frequency information on the restoration algorithm such as deconvolution or deep learning when processing the imaging data. In this way, the pixel structure corresponding to each sub-aperture does not have to sacrifice the aperture ratio, ppi, periodicity, and other specifications, and the imaging function will not conflict with the display performance to a great extent. The situation of mutual exclusion of display and imaging functions is reduced, and the problem of image blurring caused by the diffraction of the periodic light transmission structure is effectively solved.

[0017] The second aspect embodiment of the present application provides an imaging method of a screen-under camera device based on incoherent imaging synthesis. The screen-under camera device based on incoherent imaging synthesis is arranged on the backlight side of a display panel and includes a camera lens group, an image sensor, and a light coherence regulation component. The light coherence regulation component is used to form at least two incoherent sub-apertures in the pupil of the camera lens group. The image sensor is arranged on the image side of the camera lens group and is configured to receive imaging data of each incoherent sub-aperture.

[0018] The imaging method includes the following steps:

[0019] According to the imaging data, the optical transfer functions of each incoherent sub-aperture are obtained.

[0020] The optical transfer functions of each incoherent sub-aperture are superimposed to obtain a total optical transfer function.

[0021] An image restoration algorithm is constructed according to the total optical transfer function, and an image is output based on the image restoration algorithm.

[0022] Further, the construction of the image restoration algorithm according to the total optical transfer function includes constructing the image restoration algorithm by at least one of deconvolution or deep learning.

[0023] The third aspect embodiment of the present application provides a display panel, which includes the screen-under camera device based on incoherent imaging synthesis according to the first aspect embodiment of the present application.

[0024] A fourth embodiment of the present application provides an electronic device, including: a display panel as described in the third embodiment of the present application.

[0025] The storage medium of the fifth embodiment of the present application stores a computer program thereon, and when the computer program is executed by a processor, it implements the imaging method of the second embodiment of the present application.

[0026] It is not difficult to understand that the imaging method of the under-screen camera device based on incoherent imaging synthesis in the embodiment of the second aspect of this application, the electronic device in the embodiment of the third aspect of this application, the electronic device in the embodiment of the fourth aspect of this application, and the storage medium in the embodiment of the fifth aspect of this application all have the technical effects of the under-screen camera device based on incoherent imaging synthesis in the embodiment of the first aspect, and therefore will not be repeated.

[0027] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0029] Figure 1 A schematic diagram of an under-screen camera device and a display panel based on incoherent imaging synthesis according to an embodiment of the present application;

[0030] Figure 2 Schematic diagram of the pupil of a camera lens assembly provided in one embodiment of the present application, wherein (a) shows an embodiment with two incoherent sub-apertures, (b) shows an embodiment with four incoherent sub-apertures, and (c) shows the original pupil without sub-aperture division.

[0031] Figure 3 Schematic diagrams of three methods for implementing incoherent sub-apertures according to an embodiment of the present application, wherein (a) is an embodiment using a first polarizer and a second polarizer as optical coherence control components, (b) is an embodiment using two different color filters as optical coherence control components, and (c) is an embodiment using four time-domain masks as optical coherence control components;

[0032] Figure 4Schematic diagrams of one sub-aperture when divided into two incoherent sub-apertures according to an embodiment of the present application, wherein from left to right are a schematic diagram of the pixel structure corresponding to the sub-aperture, a horizontal OTF diagram, and a vertical OTF diagram;

[0033] Figure 5 for Figure 4 Schematic diagrams of another sub-aperture under the embodiment, wherein from left to right are a schematic diagram of the pixel structure corresponding to the sub-aperture, a horizontal OTF diagram, and a vertical OTF diagram;

[0034] Figure 6 for Figure 4 Schematic diagram of the horizontal OTF and vertical OTF of two sub-apertures after superposition on the image sensor in the embodiment, where from left to right are the total horizontal OTF diagram and the total vertical OTF diagram;

[0035] Figure 7 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application.

[0036] Reference numerals:

[0037] 101, display panel; 1011, under-screen pixel area; 1012, ordinary pixel area; 102, camera lens group; 1021, incoherent sub-aperture; 103, image sensor; 104, optical coherence control component; 1041, first polarizer; 1042, second polarizer; 1043, color filter; 1044, time domain mask;

[0038] 201. Processor; 202. Memory. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0040] Understandably, to address the diffraction blur issue in under-display cameras, the main approaches used in related technologies fall into two categories: optimizing the structure of the pixel's light-transmitting area and image restoration algorithms. However, the limitations of pixel structure optimization lie in the fact that the shape, arrangement, and aperture ratio of the light-transmitting area must simultaneously meet display requirements, making it impossible to design a pixel structure with low diffraction intensity. In particular, the aperture ratio, which directly affects the intensity of diffraction fringes, cannot be excessively large. The aperture ratio of OLED pixels typically does not exceed 30%, while that of micro-LED pixels typically does not exceed 70%. Furthermore, the higher the pixel density (ppi, or pixels per inch), the greater the proportion of area occupied by inherently non-transparent components (luminescent materials, driver circuits, etc.), resulting in a smaller aperture ratio of the light-transmitting area. This makes pixel structure optimization difficult to implement in under-display cameras on high-end mobile phones. Image restoration algorithms can be further divided into traditional deconvolution algorithms and deep learning algorithms that contain prior knowledge. The former is affected by the large number of zero points in the OTF (this is an inevitable result of the OTF being a periodic light-transmitting area), and restoration quality and computational convergence cannot be taken into account at the same time. The latter is restricted by the generalization of deep learning, especially the frequency components near the OTF zero points, and the restoration quality depends on the specific training set.

[0041] Based on this, an embodiment of the present application discloses an under-screen camera device based on incoherent imaging synthesis, which attempts to solve the image blur problem caused by diffraction of periodic transparent structures. Specifically, it can overcome the limitations of the two means of optimizing pixel structure and image restoration algorithm as much as possible.

[0042] See also Figures 1 to 7 As shown, the embodiment of the first aspect of the present application discloses an under-screen camera device based on incoherent imaging synthesis, which is arranged on the backlight side of the display panel 101, including: a camera lens group 102, an image sensor 103, a control device and an optical coherence control component 104;

[0043] The optical coherence control component 104 is used to form at least two incoherent sub-apertures 1021 in the pupil of the camera lens group 102; the image sensor 103 is arranged on the image side of the camera lens group 102, and the image sensor 103 is configured to receive imaging data of each incoherent sub-aperture 1021; the control device is configured to receive and process the signal of the image sensor 103 to output an image.

[0044] In the embodiment of the present application, the coherence of light entering the pupil of the camera lens group 102 is destroyed by the optical coherence control component 104, so that the pupil of the camera lens group 102 is divided into multiple incoherent sub-apertures 1021. When the images formed by the incoherent sub-apertures 1021 are superimposed on the image sensor 103, only the intensity of each image is superimposed, and the complex amplitude containing phase information of each image is not superimposed. The intensity of the image on the image sensor 103 is the algebraic sum of the intensities of the corresponding images of each incoherent sub-aperture 1021. The comprehensive optical transfer function of the system is the algebraic sum of the optical transfer functions corresponding to each incoherent sub-aperture 1021. Therefore, By independently designing the pixel structure of the display panel 101 above each incoherent sub-aperture 1021, the synthesized optical transfer function no longer contains a large number of zero points, so that the control device can break through the limitations of partial frequency information loss on recovery algorithms such as deconvolution or deep learning when processing imaging data. In this way, the pixel structure corresponding to each incoherent sub-aperture 1021 does not have to sacrifice specifications such as aperture ratio, ppi, and periodicity. The imaging function will not conflict with the display performance to a large extent, reducing the mutual exclusion of pixel structures between display and imaging functions, and effectively solving the problem of image blur caused by diffraction of periodic transparent structures.

[0045] It is understood that the incoherent subaperture 1021 is a concept that combines subaperture technology with incoherent imaging properties. In the incoherent imaging system of the present application, after the large aperture is divided into multiple subapertures by the optical coherence control component 104, the signals of each subaperture are processed and synthesized in an incoherent manner (light intensity superposition). Specifically, when the optical signals between subapertures lack a stable phase correlation, imaging relies on the direct superposition of light intensities rather than the superposition of complex amplitudes (coherent superposition). Therefore, the incoherent subaperture 1021 achieves subaperture signal synthesis through light intensity superposition, eliminating the original coherent relationship between the complex amplitudes of the subapertures, thereby constructing an incoherent imaging system.

[0046] It should be noted that the core characteristic of an incoherent imaging system is that the light source and the light field during the imaging process are incoherent, meaning that at least one of the following three conditions must be met: the light has different vibration directions, different frequencies, and no stable phase relationship. For example, the light emitted by this type of light source consists of a large number of independent light points, each of which has a randomly varying phase and no stable phase relationship. Therefore, the light waves from each light point cannot interfere with each other on the image plane. Unlike coherent imaging systems, which form images through complex amplitude superposition (involving phase), in incoherent imaging systems, the light intensity on the image plane is the direct superposition (algebraic sum) of the light intensities generated on the image plane by each point light source on the object plane. Incoherent imaging systems form images using incoherent light sources and the principle of light intensity superposition. In some embodiments, incoherent imaging systems use a point spread function to describe the dispersion of point light source imaging, and evaluate imaging quality using the normalized spectrum of the point spread function - the optical transfer function.

[0047] For example, in some embodiments, the frequency domain description of an incoherent imaging system is achieved through the optical transfer function (OTF). The OTF can be used to analyze the system's transfer characteristics for different spatial frequencies (i.e., the fineness of object detail, with high frequencies corresponding to fine structures and low frequencies corresponding to macroscopic contours). This involves both amplitude and phase transfer. Mathematically, the OTF is the normalized Fourier transform of the system's point spread function (PSF). Therefore, in image restoration algorithms, if the same object is imaged using systems with different OTFs, the frequency information of multiple images can be fused through the algebraic sum of the OTFs to improve the restoration effect (e.g., combining the OTFs of lenses with different focal lengths to compensate for high-frequency loss in a single system). Furthermore, the total OTF of the incoherent sub-aperture 1021 system is the weighted superposition of the OTFs of each incoherent sub-aperture 1021. This relationship is determined by the "intensity superposition" property of incoherent imaging and the linear properties of the Fourier transform. This superposition can complement the frequency response of the incoherent sub-aperture 1021, optimize the system's ability to transmit spatial frequencies (such as filling the zero point of a single incoherent sub-aperture 1021 OTF and expanding the effective frequency band), and improve the overall imaging quality.

[0048] In other embodiments, after dividing the under-screen pixel area 1011 into multiple incoherent sub-apertures 1021, in addition to frequency-domain imaging algorithms based on the optical transfer function (OTF), spatial signal fusion, physical property modeling, and multimodal information complementation can be used to acquire images. Image restoration algorithms can then be constructed by superimposing sub-aperture functions (such as point spread functions, response functions, and noise models). The core concept is to leverage the "independent superposition" of light intensities within incoherent sub-apertures 1021, compensating for imaging defects (such as blur, noise, and interference) inherent in a single sub-aperture through spatial processing, physical constraints, or multi-dimensional information fusion, ultimately achieving image restoration. Therefore, in addition to the OTF frequency domain method, the "independence of light intensity and complementary functions" characteristics of incoherent sub-apertures 1021 can be exploited to acquire images through spatial PSF stacking deconvolution, complementary fusion of response functions, physical constraint interference suppression, and deep learning feature stacking. A restoration algorithm is constructed based on the stacking of sub-aperture functions (PSF, response function, interference function, and characteristic function), which can also achieve the effect of controlling the device to receive and process signals from image sensor 103 to output an image. These methods all utilize the logic of "segmentation-independent sampling-complementary stacking" to compensate for the imaging shortcomings of a single sub-aperture. They are particularly suitable for complex scenarios where under-display cameras must simultaneously cope with display interference, low light transmittance, and diffraction blur.

[0049] The following will be combined Figures 1 to 7 The under-screen camera device based on incoherent imaging synthesis disclosed in the embodiments of the present application is specifically explained and illustrated.

[0050] It should be understood that, in order to realize the non-coherent sub-aperture 1021 and improve the performance of the under-screen camera device and the image quality of the display panel 101 by the non-coherent imaging system, a device for destroying the coherence of light can be added between the under-screen pixel area 1011 of the display panel 101 and the camera lens group 102, and the structural design of the light coherence control assembly 104 for destroying the light correlation is the key to realizing the above functions.

[0051] For example, in some embodiments, referring to Figure 3 (a), the light coherence control assembly 104 includes a first polarizer 1041 and a second polarizer 1042, and the polarization directions of the first polarizer 1041 and the second polarizer 1042 are perpendicular to each other, so that the light between the two non-coherent sub-apertures 1021 has orthogonal vibration directions. It can be understood that by making the light between the non-coherent sub-apertures 1021 have orthogonal vibration directions through the first polarizer 1041 and the second polarizer 1042, specifically, by inserting vertical and horizontal linear polarizers, two non-coherent sub-apertures 1021 are formed by making the light between the non-coherent sub-apertures 1021 have orthogonal vibration directions.

[0052] In some embodiments, the first polarizer 1041 and the second polarizer 1042 can be polarizers with vertical and horizontal polarization directions. In other embodiments, they can also be polarizers with other angle polarization directions, and the two polarizers are arranged in a perpendicular manner to make the light between the sub-apertures have orthogonal vibration directions to form two non-coherent sub-apertures 1021.

[0053] For example, in some embodiments, referring to Figure 3 (b), the light coherence control assembly 104 includes at least two color filters 1043, each color filter 1043 has a non-overlapping transmission spectrum, so that the light between each non-coherent sub-aperture 1021 has a different frequency. It can be understood that by using multiple color filters 1043 with non-overlapping transmission spectra, the light between the sub-apertures can have different frequencies. Specifically, color filters 1043 with transmission spectra T1(λ) and T2(λ) are inserted (λ: wavelength of light), and T1(λ) and T2(λ) have no overlapping area, such as T1(λ) only allows short-wave visible light to pass through, and T2(λ) only allows long-wave visible light to pass through. In this way, two non-coherent sub-apertures 1021 are formed by making the light between the sub-apertures have different frequencies.

[0054] In some embodiments, using this method, the number of non-coherent sub-apertures 1021 is not limited to two, and more non-coherent sub-apertures 1021 can be constructed by using more color filters 1043 with non-overlapping transmission spectra.

[0055] For example, in some embodiments, referring to Figure 3(c) The optical coherence control component 104 includes multiple time domain masks 1044, each of which is configured to be turned on at a preset time and allow light to pass through, so that the light between each incoherent sub-aperture 1021 has an uncertain phase relationship. It can be understood that by turning on multiple time domain masks 1044 at preset times, the light between the incoherent sub-apertures 1021 can have no definite phase relationship. Specifically, in the embodiment of the present application, four time domain masks 1044 are inserted, and the four masks allow light to pass through in sequence. No two time domain masks 1044 will be turned on at the same time, ensuring that the light waves received on the camera sensor are necessarily different wave trains corresponding to different masks, without a definite phase relationship, and forming incoherent sub-apertures 1021.

[0056] In some embodiments, the multiple time domain masks 1044 may be turned on at preset timings based on an uncertain correlation between the light, and may be specifically designed to be turned on at specific timings or at random timings.

[0057] In some embodiments of the present application, reference is made to Figure 1 The optical coherence control component 104 is disposed on the object side of the camera lens assembly 102. One or more optical coherence control components 104 may be provided. It is understood that the embodiments provided herein list three approaches (vibration direction, frequency, and phase relationship) and cite three corresponding specific means (orthogonal polarizers, complementary color filters 1043, and time-domain masks 1044) for destroying the coherence of light between sub-apertures using the optical coherence control component 104. In specific implementations, the means for destroying coherence may be a combination of the above three means.

[0058] In some embodiments, the number of incoherent sub-apertures 1021 formed by the optical coherence modulation component 104 is greater than or equal to two, so that the composite OTF is different from the independent OTFs. Furthermore, the regions formed by the incoherent sub-apertures 1021 can have any geometric shape that facilitates the implementation of the mechanical structure, because the composite OTF does not have any requirements on the aperture shape.

[0059] In some embodiments of the present application, reference is made to Figure 1 , the camera lens group 102 and the image sensor 103 are arranged in combination, and the number of the camera lens group 102 and the image sensor 103 is 1; it can be understood that the under-screen camera device based on incoherent imaging synthesis in the embodiment of the present application has only one image sensor 103 and one lens group, so as to distinguish it from a multi-camera solution with significantly higher complexity and cost.

[0060] In some embodiments of the present application, reference is made to Figure 1 and Figure 2The under-screen camera device based on incoherent imaging synthesis also includes an under-screen pixel area 1011, which is disposed on the display panel 101 and corresponds to the camera lens assembly 102. The under-screen pixel area 1011 is divided into pixel regions corresponding to respective incoherent sub-apertures 1021. Each pixel structure in each of these pixel regions has a predetermined pixel density (typically measured in pixels per inch (ppi)) and / or a predetermined aperture ratio. It can be understood that the incoherent sub-apertures 1021, through spatial segmentation and signal synthesis, overcome the limitation of a single aperture where the performance of a single pixel determines the overall performance, thereby enabling the use of pixel structures with higher ppi and lower aperture ratios.

[0061] The following describes in detail the under-screen camera device based on incoherent imaging synthesis according to an embodiment of the present application using a specific embodiment. It should be noted that the following embodiment is merely an illustrative description and should not be construed as limiting the embodiments of the present application.

[0062] See also Figures 1 to 7 As shown, the under-screen camera device based on incoherent imaging synthesis of this embodiment is as follows Figure 1 The display panel 101 has a first pixel area (a normal pixel area 1012 that only performs display functions) and a second pixel area (an under-screen pixel area 1011 above the under-screen camera). The camera lens group 102 is closely attached to the bottom of the under-screen pixel area 1011. The camera has an image sensor 103 located on the image plane of the lens group.

[0063] like Figure 2 The camera's only pupil has multiple incoherent sub-apertures 1021 for the imaging process. The incoherent sub-apertures 1021 divide the pupil into multiple geometric regions. Incoherence here means that the light waves between the sub-apertures do not have a definite phase relationship. When the images formed by the incoherent sub-apertures 1021 are superimposed on the only image sensor 103, only the intensities of the images can be superimposed, but not the complex amplitudes containing phase information of the images. The number of incoherent sub-apertures 1021 is greater than or equal to 2, and the geometric division method may be: Figure 2 The shown ones are 2 semicircles, 4 90° sectors, etc., and other reasonable and easy-to-implement geometric shapes are also possible. The shape of the geometric area does not affect the beneficial effects of this patent in terms of OTF and image restoration. For comparison, Figure 2 The rightmost image shows the original pupil without sub-aperture division, and the light waves within a single aperture are coherent.

[0064] like Figures 4 to 6 Taking two incoherent sub-apertures 1021 as an example, the pixel structure in the first incoherent sub-aperture 1021 can be as follows: Figure 4 The rectangular structure shown on the left has the corresponding horizontal and vertical OTFs as follows: Figure 4 As shown in the middle and right; the pixel structure in the second sub-aperture can be represented by Figure 4 The structure on the left is rotated 90° to obtain, as Figure 5 shown on the left, its corresponding horizontal and vertical OTFs are as Figure 5 shown in the middle and right. It can be seen that the pixel structure of the two incoherent sub-apertures 1021 causes the OTFs in the vertical and horizontal directions to have multiple zeros that are detrimental to image restoration algorithms. The two incoherent sub-apertures 1021 cooperate to implement an incoherent synthetic aperture, and then the overall OTF on the image sensor 103 is as Figure 3 shown on the left. It can be seen that the OTF values in both the horizontal direction (left) and the vertical direction (right) are significantly greater than zero. Figure 6

[0065] It can be understood that the single pupil corresponding to the single image sensor 103 has multiple incoherent sub-apertures 1021, which constitute an incoherent synthetic aperture. Thus, the intensity of the image on the image sensor 103 is the algebraic sum of the intensities of the images corresponding to the incoherent sub-apertures 1021. Further, by the linear nature of Fourier transform, the overall OTF of the system is the algebraic sum of the OTFs corresponding to the incoherent sub-apertures 1021. By independently designing the pixel structure of the display panel 101 on each incoherent sub-aperture 1021, the synthetic OTF can no longer contain a large number of zeros, breaking the limitation of the loss of partial frequency information on the deconvolution or deep learning restoration algorithm. Moreover, each incoherent sub-aperture 1021 does not have to independently implement an OTF that is easy for image restoration, but only needs the synthetic OTF to meet the requirements of the restoration algorithm, and thus the pixel structure corresponding to each incoherent sub-aperture 1021 does not have to sacrifice the specifications such as aperture ratio, ppi, periodicity, and does not conflict with the display performance.

[0066] An embodiment of the second aspect of the present application discloses an imaging method of a screen-under camera device based on incoherent imaging synthesis. The imaging method of the screen-under camera device based on incoherent imaging synthesis can be the imaging method of the screen-under camera device based on incoherent imaging synthesis of the first aspect of the present application. The screen-under camera device based on incoherent imaging synthesis is arranged on the backlight side of the display panel 101 and includes a camera lens group 102, an image sensor 103, and a light coherence control assembly 104. The light coherence control assembly 104 is configured to form at least two incoherent sub-apertures 1021 in the pupil of the camera lens group 102. The image sensor 103 is arranged on the image side of the camera lens group 102 and is configured to receive imaging data of each incoherent sub-aperture 1021.

[0067] The imaging method includes the following steps:

[0068] Obtaining the OTF of each incoherent sub-aperture 1021 according to the imaging data;

[0069] Superimposing the OTFs of the incoherent sub-apertures 1021 to obtain the total OTF;​

[0070] An image restoration algorithm is constructed according to the total OTF, and an image is output based on the image restoration algorithm.

[0071] It can be understood that by dividing the under-screen pixel area 1011 into multiple regions and optimizing the pixel structures corresponding to each region, the OTFs have complementary zero points, thereby eliminating the large number of zero points in the resulting OTF. Specifically, by inserting orthogonal polarizers, complementary color filters 1043, and phase retarders with different optical path differences, the coherence of light between different regions is destroyed, and incoherent sub-apertures 1021 are constructed. The OTFs of the incoherent sub-apertures 1021 are then superimposed to obtain an OTF. Based on this OTF, an image restoration algorithm is constructed to improve image quality.

[0072] It's understood that in the optical transfer function (OTF), the zero point refers to the spatial frequency at which the modulus of the OTF (i.e., the modulation transfer function (MTF)) is zero. At this spatial frequency, the system is completely unable to transmit the contrast information of the object, and the details of the object at that frequency are lost in the image. The zero point of the OTF can be understood as a sign that the system has completely lost its ability to transmit contrast at a specific spatial frequency.

[0073] In this application, the total OTF is the weighted sum of the OTFs of each incoherent sub-aperture 1021. This relationship stems from the intensity superposition characteristics of incoherent imaging and the linear nature of the Fourier transform. Through the differentiated design of the incoherent sub-apertures 1021 (such as pupil shape and pixel structure), the OTFs of each incoherent sub-aperture 1021 can be made complementary in the frequency domain, achieving the effect of "filling zero points and improving MTF (especially at frequencies where the MTF value is originally zero)", ultimately improving the system's imaging quality and image restoration capabilities.

[0074] It should be understood that, illustratively, the pixel structures above each incoherent sub-aperture 1021 are different, so that the superimposed OTF is more conducive to diffraction image recovery than the OTF of a sub-aperture used independently. When each sub-aperture is used independently, it faces the same problem as previous technologies, that is, it is impossible to balance the display performance and the mutually exclusive requirements of the diffraction image recovery algorithm for the pixel structure. The novel incoherent sub-aperture 1021 feature of the present application brings beneficial effects: the pixel structure corresponding to each incoherent sub-aperture 1021 can use a higher ppi and a lower aperture ratio - high ppi is conducive to delicate and sharp images, and low aperture ratio ensures that the opaque thin-film transistor has sufficient device space, both of which are conducive to high-quality display. Although the OTF of such a designed sub-aperture still contains a large number of zero points when used alone, which is not conducive to image recovery, the incoherent synthetic aperture reduces or even eliminates the zero points in the synthetic OTF, which is beneficial to image recovery algorithms such as deconvolution and deep learning, and breaks through the mutually exclusive requirements of display and imaging functions for pixel structure.

[0075] For example, in some embodiments, constructing an image restoration algorithm based on the total OTF includes constructing the image restoration algorithm through at least one of deconvolution or deep learning. It is understood that any feasible algorithm can be used in combination with this patent to obtain an image restoration algorithm, and this application does not further limit this. It should be noted that the fewer zero points in the OTF, the less information loss. Regardless of the restoration algorithm used, the OTF provided by this patent is more advantageous than the OTF containing a large number of zero points in the related art, and thus can achieve the beneficial effects of the above-mentioned embodiments.

[0076] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0077] The display panel 101 of the third embodiment of the present application can be a display panel 101 of a mobile phone, a computer or an electronic product with a camera function, etc. The display panel 101 includes: the under-screen camera device based on incoherent imaging synthesis of the first embodiment of the present application.

[0078] The electronic device of the fourth embodiment of the present application may be a mobile phone, a computer, or an electronic product with a display panel 101 and a camera function, etc. The electronic device includes: the display panel 101 of the third embodiment of the present application.

[0079] It can be understood that the contents of the above-mentioned device embodiments are applicable to the embodiments of this product and equipment. The functions specifically implemented by the embodiments of this product and equipment are the same as those of the above-mentioned device embodiments, and the beneficial effects achieved are also the same as those achieved by the above-mentioned device embodiments.

[0080] Reference Figure 7 The electronic device of an embodiment of the present application includes: a memory 202, a processor 201, and a computer program stored in the memory 202 and executable on the processor 201. When the processor 201 executes the computer program, the above-mentioned method embodiment is implemented.

[0081] Similarly, it can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0082] The storage medium of the fifth embodiment of the present application stores a computer program thereon, and when the computer program is executed by a processor, the imaging method of the embodiment of the first aspect of the present application is implemented.

[0083] Similarly, the contents in the above method embodiments are applicable to the computer readable storage medium embodiments, the computer readable storage medium embodiments specifically implement the functions same as the above method embodiments, and achieve the beneficial effects same as the above method embodiments.

[0084] In the description of the present application, it should be understood that the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only intended to facilitate the description of the present application and simplify the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0085] The terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more. It should be noted that the term "and / or" used herein is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. Similarly, at least one of A or B can also mean that A exists alone, A and B exist together, and B exists alone.

[0086] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connection" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integrally connected; it can be directly connected, or indirectly connected through an intermediate medium, or the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0087] In the above description of the present specification, the description of the terms "one embodiment", "another embodiment" or "some embodiments" means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the description of the present specification, the specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0088] It should be understood that although the steps in the flowcharts of the drawings are shown in a sequential order, such steps are not necessarily performed in the order shown in the arrows. Unless otherwise specifically noted, the steps can be performed in other orders. Moreover, at least some of the steps in the flowcharts of the drawings can include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but can be performed at different times, and the order of the execution of the sub-steps or stages is not necessarily sequential, but can be round-robin or alternating with at least some of the other steps or sub-steps or stages of other steps.

[0089] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules, unless otherwise specifically noted. It is also to be understood that detailed discussion of the actual implementation of each module is unnecessary to an understanding of the present application. Rather, the properties, functions and internal relationships of the various functional modules disclosed in the devices herein are considered to be within the ordinary skill in the art having the benefit of this disclosure. Accordingly, those skilled in the art with access to the present disclosure need not spend undue time and effort in implementing the present application as set forth in the claims. It is also to be understood that the disclosed specific concepts are merely illustrative and not intended to limit the scope of the present application, which is defined by the full scope of the appended claims and their equivalents.

[0090] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0091] The logic and / or steps represented in the flow diagrams and / or described herein, for example, can be considered as a sequence of executable instructions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. Just by way of example, a computer-readable medium can be any device or apparatus that can store and convey instructions for execution by the instruction execution system, apparatus, or device. With respect to the present description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0092] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or other suitable material upon which the program is printed, as the program can be electronically captured, for example, via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that can be later executed by a computer.

[0093] It should be understood that aspects of the present application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, can be used: a combination of logic gates, discrete logic, application specific integrated circuits (ASICs), programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and / or the like.

Claims

1. An under-screen camera device based on incoherent imaging synthesis, characterized in that: It is arranged on the backlight side of the display panel and includes: a camera lens group, an image sensor, a control device and an optical coherence control component; The optical coherence regulating component is used to form at least two incoherent sub-apertures in the pupil of the camera lens group; The image sensor is disposed on the image side of the camera lens assembly, and the image sensor is configured to receive imaging data of each of the incoherent sub-apertures; The control device is configured to receive and process the signal from the image sensor to output an image.

2. The under-screen camera device based on incoherent imaging synthesis according to claim 1, characterized in that: The optical coherence regulating component includes a first polarizer and a second polarizer, and the polarization directions of the first polarizer and the second polarizer are perpendicular to each other, so that the light between the two incoherent sub-apertures has orthogonal vibration directions.

3. The under-screen camera device based on incoherent imaging synthesis according to claim 1, characterized in that: The optical coherence regulation component includes at least two color filters, each of which has a non-overlapping transmission spectrum, so that the light between each of the incoherent sub-apertures has a different frequency.

4. The under-screen camera device based on incoherent imaging synthesis according to claim 1, characterized in that: The optical coherence regulation component includes a plurality of time domain masks, each of which is configured to be turned on at a preset time and allow light to pass through, so that the light between each of the incoherent sub-apertures has an uncertain phase relationship.

5. The under-screen camera device based on incoherent imaging synthesis according to claim 1, characterized in that: The camera lens group and the image sensor are arranged in a matching manner, and the number of the camera lens group and the image sensor is 1; And / or, the optical coherence regulating component is arranged on the object side of the camera lens assembly, and one or more optical coherence regulating components are arranged for superposition; And / or, the under-screen camera device based on incoherent imaging synthesis also includes an under-screen pixel area, which is arranged on the display panel and corresponds to the camera lens group, and the under-screen pixel area is divided into pixel areas corresponding to each of the incoherent sub-apertures, and each pixel structure of each pixel area has a preset pixel density and / or a preset aperture ratio.

6. An imaging method for an under-screen camera device based on incoherent imaging synthesis, wherein the under-screen camera device based on incoherent imaging synthesis is disposed on the backlight side of a display panel, comprising: Camera lens assembly, image sensor and optical coherence control components; The optical coherence regulating component is used to form at least two incoherent sub-apertures in the pupil of the camera lens group; The image sensor is disposed on the image side of the camera lens assembly, and the image sensor is configured to receive imaging data of each of the incoherent sub-apertures; The imaging method includes the following steps: Obtaining an optical transfer function of each of the incoherent sub-apertures according to the imaging data; superimposing the optical transfer functions of the incoherent sub-apertures to obtain a total optical transfer function; An image restoration algorithm is constructed according to the total optical transfer function, and an image is output based on the image restoration algorithm.

7. The imaging method of an under-screen camera device based on incoherent imaging synthesis according to claim 6, characterized in that: The constructing of the image restoration algorithm according to the total optical transfer function includes: constructing the image restoration algorithm by at least one of deconvolution and deep learning.

8. A display panel, characterized in that: include: An under-screen camera device based on incoherent imaging synthesis as claimed in any one of claims 1 to 5.

9. An electronic device, characterized in that: include: The display panel as claimed in claim 8.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the imaging method according to claim 6 or 7 is implemented.

Images