Processing images captured by a camera behind the display

Abstract

A method includes capturing an original image through a semi-transparent pixel area of ​​a display panel via a camera arranged behind a display panel of an electronic device. The original image includes one or more color components. The method also includes determining a point spread function (PSF) for each of the one or more color components for multiple pixel areas of the original image. The method also includes, for multiple pixel areas of the original image, performing deconvolution of each of the one or more color components of the original image based at least in part on their respective PSFs. Therefore, the method includes generating a reconstructed image corresponding to the original image based on the deconvolution of the one or more color components of the multiple pixel areas of the original image.

Description

Technical Field

[0001] The present invention relates generally to electronic displays and, more particularly, to processing images captured by a camera behind an electronic display. Background Art

[0002] Electronic displays such as active matrix liquid crystal displays (AMLCDs), active matrix organic light emitting displays (AMOLEDs), and micro-LED displays are often the type of displays deployed for personal electronic devices such as mobile phones, tablet computers, smart watches, etc. Such personal electronic devices often may include a front-facing camera, which may be disposed near the display and may be most commonly used by the user to capture self-portraits (e.g., “selfies”).

[0003] However, as the complexity of the front-facing camera system increases (e.g., a depth camera), more and more of the area designated for the display of the electronic device may be exchanged to expand the area designated for the camera system. This may result in a reduction in the resolution and viewable area of ​​the display. Summary of the invention

[0004] [Technical solution]

[0005] In order to address the current deficiencies, a method is provided, which includes capturing an original image through a semi-transparent pixel area of ​​a display panel by a camera arranged behind a display panel of an electronic device. The original image contains one or more color components. The method also includes determining a point spread function (PSF) of each of the one or more color components for multiple pixel areas of the original image. The method also includes, for multiple pixel areas of the original image, deconvolving each of the one or more color components of the original image based at least in part on their respective PSFs. Therefore, the method includes generating a reconstructed image corresponding to the original image based on the deconvolution of one or more color components of multiple pixel areas of the original image.

[0006] [Beneficial Effects]

[0007] The provided method and apparatus can capture an original image through a camera arranged behind a display panel of an electronic device via semi-transparent pixels to reconstruct the original image and increase the visible area and resolution of the display of the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 An exemplary electronic device is shown.

[0009] Figure 2 A diagram showing the path of light as it passes through each individual layer of a display of an electronic device.

[0010] Figure 3A diagram showing a pixel array, RGB unit pixels, alpha mask unit pixels, and a photomask.

[0011] Figure 4A An example system and workflow diagram for reconstructing an image captured by a camera disposed behind a display of an electronic device is shown.

[0012] Figure 4B A flow chart is shown for specifically reconstructing a color image captured by a camera arranged behind a display of an electronic device.

[0013] Figure 5A A reference coordinate system is shown, from which one or more point spread functions (PSFs) may be derived based on the reference coordinate system.

[0014] Figure 5B An example system and workflow diagram for measuring and storing the point spread function (PSF) of an electronic device is shown.

[0015] Figure 6 A flow chart of a method for reconstructing an image captured by a camera arranged behind a display of an electronic device is shown.

[0016] Figure 7 An example computer system is shown. DETAILED DESCRIPTION

[0017] The present embodiment is intended to reconstruct an image captured by a camera disposed behind a display of an electronic device. In a particular embodiment, the electronic device may capture an original image through a semi-transparent (or transparent, partially transparent, etc.) area of ​​the display panel through a camera disposed behind the display panel of the electronic device. In a particular embodiment, the original image may include multiple color components, more specifically, (R) red component, (G) green component, and (B) blue component. In some implementations, the camera determines the bit depth. In a particular embodiment, the original image may include a bit depth N=10, which includes various color components. In a particular embodiment, other values ​​of the bit depth are possible (e.g., N=2, N=8, or N=12, etc.). In a particular embodiment, for multiple pixel areas of the original image, the electronic device may also determine a point spread function (PSF) for each color component. In a particular embodiment, the electronic device may determine the PSF of each color component by selecting a plurality of predicted PSFs corresponding to the number of color components from a memory of the electronic device. In a particular embodiment, each PSF may be predicted based on a predetermined calibration process performed for the camera and the display panel.

[0018] In a particular embodiment, the electronic device may then deconvolve each color component of the original image based on its respective PSF for the number of pixel regions of the original image. In a particular embodiment, the electronic device may perform deconvolution of each color component by performing Richardson-Lucy deconvolution of each color component. In a particular embodiment, the electronic device may perform deconvolution of each color component by performing Tikhonov regularized inverse filter deconvolution deconvolution on each color component. In a particular embodiment, the electronic device may then generate a reconstructed image corresponding to the original image based on the deconvolution of the number of color components of the number of pixel regions of the original image. In a particular embodiment, the electronic device may generate a reconstructed image corresponding to the original image by removing the blurring effect of the original image. In a particular embodiment, the electronic device may also generate a reconstructed image corresponding to the original image by color correcting each color component of the number of pixel regions of the original image.

[0019] Invention method

[0020] To promote an understanding of the principles of the present invention, reference will now be made to the embodiments shown in the accompanying drawings, and specific language will be used to describe them. However, it should be understood that no limitation of the scope of the present invention is intended thereby, and such changes and further modifications in the systems shown, as well as further applications of the principles of the present invention as described herein, are contemplated as would normally occur to one skilled in the art to which the present invention relates. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. The systems, methods, and examples provided herein are illustrative only and are not intended to be limiting.

[0021] As used herein, the term "some" is defined as "none, or one, or more than one, or all." Thus, the terms "none," "one," "more than one," "more than one but not all," or "all" fall within the definition of "some." The term "some embodiments" may refer to no embodiments, one embodiment, more than one embodiment, or all embodiments. Thus, the term "some embodiments" is defined to mean "none, or one, or more than one, or all embodiments."

[0022] The terms and structures used herein are used to describe, teach, and illustrate some embodiments and specific features and elements thereof, but not to limit, restrict, or narrow the scope of the claims or their equivalents.

[0023] More specifically, any terms used in this document, such as but not limited to "includes", "comprising", "having", "consisting of" and their grammatical variations, do not specify exact limitations or restrictions, and certainly do not exclude the possibility of adding one or more features or elements unless otherwise specified. In addition, it does not exclude the possibility of removing one or more listed features and elements, unless there is another restrictive language "must include" or "need to include".

[0024] Regardless of whether a feature or element is limited to only one use, it can still be referred to as "one or more features" or "one or more elements" or "at least one feature" or "at least one element". In addition, the use of the term "one or more" features or elements or "at least one" feature or element does not exclude the absence of the feature or element, unless the limiting language specifies otherwise, such as "requires one or more..." or "requires one or more elements".

[0025] Unless otherwise defined, all terms used herein, especially any technical and / or scientific terms, may be considered to have the meanings commonly understood by one of ordinary skill in the art.

[0026] Reference is made herein to some "embodiments". It should be understood that embodiments are examples of possible implementations of any features and / or elements presented in the appended claims. Some embodiments have been described in order to illustrate one or more potential ways that specific features and / or elements of the appended claims satisfy the requirements of uniqueness, utility, and non-obviousness.

[0027] The use of phrases and / or terms such as, but not limited to, "a first embodiment," "another embodiment," "an alternative embodiment," "an embodiment," "an embodiment," "embodiments," "some embodiments," "other embodiments," "further embodiments," "further embodiments," "additional embodiments," "additional embodiments," or variations thereof, do not necessarily refer to the same embodiment. Unless otherwise specified, one or more particular features and / or elements described in conjunction with one or more embodiments may be found in one embodiment, or may be found in multiple embodiments, or may be found in all embodiments, or may not be found in any embodiment. Although one or more features and / or elements may be described herein only in the context of a single embodiment, or alternatively in the context of multiple embodiments, or further alternatively in the context of all embodiments, these features and / or elements may be provided separately, or in any appropriate combination, or not provided at all. Conversely, any features and / or elements described in the context of separate embodiments may alternatively be implemented as present together in the context of a single embodiment.

[0028] Any and all details described herein are used in the context of some embodiments and therefore should not be considered as limiting factors of the appended claims. The appended claims and their legal equivalents can be implemented in the context of embodiments other than those used as illustrative examples in the following description.

[0029] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0030] In this way, the present embodiment can increase the visible area and resolution of the display of the electronic device by arranging the forward-pointing camera of one or more electronic devices behind the display. For example, due to the increase in the display area (e.g., eliminating the display area usually designated for one or more forward-pointing cameras), the electronic device can further provide an improved graphical user interface (GUI), which has a full-screen view as a whole, rather than only displaying battery status, cellular signal strength data, Wi-Fi status, time information, etc., in accordance with the notch design or punch hole design. The present technology can further improve the aesthetic quality of the electronic device and allow the user of the electronic device to display a higher resolution image on the display of the electronic device. Further, since one or more forward-pointing cameras can be placed behind the display, the present technology can allow one or more forward-pointing cameras to be placed in any position (e.g., in the center area of ​​the display), rather than in the corner or edge of the electronic device display. This can provide an improved user experience and / or GUI, such as by guiding the user taking a selfie to look at the center area of ​​the display, and further by giving the impression of making eye contact with another user when the user participates in a video conference, videophone exchange or other video streaming services.

[0031] Figure 1 An exemplary electronic device 100 is shown. In certain embodiments, the electronic device 100 may include, for example, any of a variety of personal electronic devices 102, such as a mobile phone electronic device, a tablet electronic device, a laptop electronic device, etc. In certain embodiments, as shown in FIG. Figure 1 As further shown, the personal electronic device 102 may include one or more processors 104, memory 106, sensors 108, cameras 110, displays 112, input structures 114, network interfaces 116, power supplies 118, and input / output (I / O) interfaces 120. It should be noted that Figure 1 This is merely one example of a particular implementation and is intended to illustrate the types of components that may be included as part of electronic device 100 .

[0032] In certain embodiments, one or more processors 104 are operably coupled to memory 106 to execute various algorithms for providing interactive music conducting and composing activities through an intelligence-based learning process. Such programs or instructions executed by processor 104 may be stored in any suitable article of manufacture that includes one or more tangible computer-readable media that store at least instructions or routines together, such as memory 106. Memory 106 may include any suitable article of manufacture for storing data and executable instructions, such as random access memory (RAM), read-only memory (ROM), rewritable flash memory, hard drive, etc. In addition, programs (e.g., operating systems) encoded on such computer program products may also include instructions that can be executed by processor 104 to enable electronic device 100 to provide various functions.

[0033] In certain embodiments, the sensor 108 may include, for example, one or more cameras (e.g., depth cameras), touch sensors, microphones, motion detection sensors, heat detection sensors, light detection sensors, time of flight (ToF) sensors, ultrasonic sensors, infrared sensors, or other similar sensors that can be used to detect various user inputs (e.g., user voice input, user gesture input, user touch input, user instrument input, user motion input, etc.). The camera 110 may include any number of cameras (e.g., wide cameras, narrow cameras, telephoto cameras, ultra-wide cameras, depth cameras, etc.) that can be used to capture various 2D and 3D images. The display 112 may include any display architecture (e.g., AMLCD, AMOLED, micro LED, etc.) that may provide further means by which a user can interact and engage with the electronic device 100. In certain embodiments, as Figure 1 As further shown, one or more cameras 110 may be disposed behind or below the display 112 (e.g., as shown by the dashed lines of the electronic device 100) (e.g., one or more cameras 110 may be completely hidden by the display 112), and thus, the display 112 may include transparent pixel areas and / or semi-transparent pixel areas through which the one or more hidden cameras 110 may detect light and thereby capture images. It should be understood that the one or more cameras 110 may be disposed at any position behind or below the display 112, such as a central area behind the display 112, an upper area behind the display 112, or a lower area behind the display 112.

[0034] In particular embodiments, the input structure 114 may include any physical structure for controlling one or more global functions of the electronic device 100 (e.g., pressing a button to "turn on" or "turn off" the electronic device 100). The network interface 116 may include, for example, any number of network interfaces suitable for allowing the electronic device 100 to access and receive data through one or more cloud-based networks (e.g., cloud-based services that can serve hundreds or thousands of electronic devices 100 and their corresponding associated users) and / or distributed networks. The power supply 118 may include any suitable power source, such as a rechargeable lithium polymer (Li-poly) battery and / or an alternating current (AC) power converter that can be used to power and / or charge the electronic device 100 for operation. Similarly, an I / O interface 120 may be provided to allow the electronic device 100 to interface with various other electronic or computing devices (such as one or more auxiliary electronic devices).

[0035] Figure 2 Graph 200 shows the path of light 202 as it passes through each individual layer of display 112. Figure 2 As shown, in certain embodiments, the display 112 and / or display panel 112 may include, among other layers, a cover glass 204, one or more polarizers 206, a quarter wave plate 208, an encapsulation layer 210, an OLED layer 212, a thin film transistor (TFT) layer 214, and a reflective film layer 215. In certain embodiments, for example, when a portion of the reflective layer 215 is removed, the light path 202 may be allowed to pass through the display 112. In certain embodiments, when the light path 202 passes through the layers 204-215 of the display 112, the light path 202 may be periodically modulated by the sub-pixels of the display 112 (e.g., the RGB color regions of the display 112), and a phase delay of the light path 202 may also be introduced due to the spacing between the sub-pixels of the display 112. Such periodic modulation of the light path 202 may cause diffraction blur and color dispersion for the light path 202 to pass through the camera lens 216 and be detected by the image sensor 218. Diffraction and dispersion effects may degrade images captured by a camera 110 disposed below and / or behind the display 112 .

[0036] Figure 3Diagram 300 shows a pixel array 302, RGB unit pixels 304, and alpha mask unit pixels 306, as well as a photomask 308. The photomask 308 shows one example of light modulation performed by the semi-transparent display panel 112. In certain embodiments, the pixel array 302 may include, for example, a 50 micrometer (μm) AMOLED pixel array, including a plurality of RGB unit pixels 304 and alpha mask unit pixels 306 (e.g., illustrated as highlight squares within the pixel array 302 and highlight squares within the photomask 308). In certain embodiments, the alpha mask unit pixels 306 may correspond to semi-transparent pixel areas within the pixel array 302 and the photomask 308, through which one or more cameras 110 disposed behind the display 112 may detect light and thereby capture an image. In certain embodiments, the photomask 308 may facilitate the light path 202 through the display 112 (e.g., through the photomask 308 and the alpha mask unit pixels 306). In certain embodiments, photomask 308 may include, for example, a fill factor of approximately 50%, wherein illustrated white areas of photomask 308 may correspond to transparent areas of photomask 308, and illustrated black areas may correspond to opaque areas of photomask 308. In certain embodiments, alpha mask unit pixels 306 and photomask 308 may be used for testing, simulation, development, and / or calibration purposes.

[0037] Figure 4AAn example system and workflow diagram 400A for reconstructing an image captured by a camera disposed behind a display of an electronic device according to an embodiment of the present disclosure is shown. In certain embodiments, the electronic device 100 may capture an image of a real-world scene 402 through an image sensor 218 disposed behind a display panel 112 of the electronic device 100. In certain embodiments, the image of the real-world scene 402 captured by the image sensor 218 may correspond to an original image 404. In certain embodiments, the original image 404 may be degraded (e.g., blurred or distorted) based on the image of the real-world scene 402 captured by the image sensor 218 through the display panel 112. In certain embodiments, after performing (at functional block 406) the capture of the original image 404, the electronic device 100 may retrieve a PSF (e.g., a function of a three-dimensional diffraction pattern of light emitted from an imperceptible small point light source and captured by one or more image sensors 218) for each RGB color component of the original image 404 for one or more pixel regions of the original image 404. In certain embodiments, the PSF may be stored on the electronic device 100. In certain embodiments, the electronic device 100 may determine the respective PSF for each RGB color component by selecting (at functional block 408) the predicted PSF for each RGB color component from the memory 106 of the electronic device 100. In certain embodiments, for example, the electronic device 100 may determine multiple PSFs in various pixel regions of the real-world scene 402 to capture the variation of the PSF with angle of incidence to the optical axis of the display panel 112.

[0038] In certain embodiments, for the number of pixel regions of the original image 404, the electronic device 100 may then perform deconvolution of each RGB color component of the original image 404 based on its respective PSF (at functional block 410). In certain embodiments, the electronic device 100 may perform deconvolution of each RGB color component by performing a Richardson-Lucy deconvolution of each RGB color component or by performing a Tikhonov regularized inverse filter deconvolution of each RGB color component. In certain embodiments, other deconvolution techniques may be used. In certain embodiments, the electronic device 100 may then generate a reconstructed image 414 corresponding to the original image 404 based on the deconvolution of each RGB color component (at functional block 412). As shown by comparing the original image 404 with the reconstructed image 414, the electronic device 100 may generate the reconstructed image 414 by removing the blurring effect of the original image 404.

[0039] Thus, the present embodiment can increase the viewable area and resolution of the display 112 of the electronic device 100 by arranging one or more front-facing cameras 110 of the electronic device 100 behind the display 112. For example, due to the increase in display area (e.g., eliminating the display area typically designated for the one or more front-facing cameras 110), the electronic device 100 can also provide an improved (GUI) with a complete full-screen view, as opposed to only displaying battery status, cellular signal strength data, Wi-Fi status, time information, etc., in accordance with a notch design or a punch hole design. The present technology can also improve the aesthetic quality of the electronic device 100 and allow the user of the electronic device 100 to display a higher resolution image on the display 112 of the electronic device 100. In addition, since the one or more front-facing cameras 110 can be placed behind the display 112, the present technology can allow the one or more front-facing cameras 110 to be placed in any position, such as the center area of ​​the display 112 (e.g., as opposed to a corner or edge of the display 112). This may provide an improved user experience and / or GUI, such as by directing a user taking a selfie to gaze at a central area of ​​display 112, and further by giving the impression of making eye contact with another user when the user is participating in a video conference, videophone exchange, or other video streaming service.

[0040] Figure 4B A flowchart 400B is shown for specifically reconstructing a color image captured by a camera disposed behind a display of an electronic device according to the presently disclosed embodiments. In a particular embodiment, the electronic device 100 (at block 416) may segment the captured degraded color image into separate (R) red color components 418, (G) green color components 420, and (B) blue color components 422. In a particular embodiment, the electronic device 100 may deconvolve and filter (e.g., Richardson-Lucy deconvolution, Tikhonov regularized inverse filter deconvolution) the (R) red color component 418, the (G) green color component 420, and the (B) blue color component 422 based on, for example, their respective PSFs. In a particular embodiment, the respective PSFs may be pre-measured (e.g., determined experimentally) at, for example, the time of calibration and / or manufacturing of the electronic device 100. In a particular embodiment, the electronic device 100 may then perform (at block 428) color correction of each of the (R) red color component 418, the (G) green color component 420, and the (B) blue color component 422 to generate (at block 430) a reconstructed image based at least in part thereon.

[0041] Figure 5AA reference coordinate system 500B is shown, through which one or more point spread functions (PSFs) can be derived based on the reference coordinate system. As shown, the reference coordinate system 500B may include an object plane 502, a photomask 504, a lens plane 506, and a sensor plane 508. In certain embodiments, since the object light field may be polychromatic and spatially incoherent, the incoherent image may use information transfer theory to find the image intensity distribution at each wavelength and integrate over the Bayer filter transmission spectrum to obtain the measured image intensity in the (R) red color component, the (G) green color component, and the (B) blue color component. In certain embodiments, the image intensity distribution of the incoherent image system may be viewed as the incoherent addition of the spatial distribution of the image intensities of the constituent object points. Specifically, the image intensity distribution of an object point emitted at a unit radiation emittance is called a point spread function (PSF), which can be approximately viewed as the spatial impulse response of the optical system. If the optical system is linear and spatially invariant, the image intensity distribution of the extended object will be the incoherent superposition of these PSFs multiplied by the corresponding object point intensities.

[0042] For example, assuming that the wavelength is λ and the light field emitted from a point in the object plane 502 is O(0,0,λ): Since the wavelength and phase are known, the light field from this point light source is coherent. The light field after being modulated by the photomask can be expressed as:

[0043]

[0044] Among them, M(x m ,y m ) is the mask distribution at the photomask plane 504, j is an imaginary unit The light field can propagate a distance Z m and passes through a lens (e.g., lens 216) at lens plane 506. The light field after the lens (e.g., lens 216) can be expressed as:

[0045]

[0046] In equation (2), P(x l ,y l , λ) represents the phase aberration introduced by the lens (e.g., lens 216), and f represents the focal length of the lens (e.g., lens 216). The light field at the sensor plane 508 can be expressed as:

[0047]

[0048]

[0049] Figure 5BAn example system and workflow diagram 500B is shown for measuring and determining one or more pre-measured point spread functions (PSFs) of an electronic device (e.g., for each RGB color component and / or one or more specific monochromatic color components, respectively) according to an embodiment of the present disclosure. For example, reference coordinate system 500B can be used as a basis for measuring and determining one or more PSFs. In certain embodiments, in order to reconstruct a degraded original image, electronic device 100 can pre-measure (e.g., experimentally determined during a calibration process and / or a manufacturing process of electronic device 100) and store the PSF of electronic device 100. In certain embodiments, as Figure 5B As shown, a point light source 510 (e.g., a white LED or an array of white LEDs) may emit light waves through, for example, a pinhole or other imperceptibly small aperture in the direction of the electronic device 100. In certain embodiments, the light waves may pass through, for example, the photomask 308, the camera lens 216, and may ultimately be detected by the image sensor 218. In certain embodiments, the electronic device 100 may then pre-measure one or more PSFs 512 for each RGB color component and / or one or more specific monochromatic color components based on, for example, sampling of a transfer function corresponding to the effect of the photomask 308 in response to the point light source 510.

[0050] For example, in certain embodiments, one or more PSFs of the electronic device 100 may represent the intensity response of the point light source 510. In certain embodiments, the one or more PSFs may be represented as:

[0051] h(x s ,y s ,λ)=|O s (x s ,y s ,λ)| 2 …(Equation 4).

[0052] In certain embodiments, the image intensity distribution may include a convolution of one or more PSFs (e.g., for each RGB color component and / or one or more specific monochromatic color components) and the intensity distribution of the real scene 402 scaled by magnification. Thus, for example, the image intensity distribution incident on the image sensor 218 may be represented as:

[0053]

[0054] Among them I O (x s ,y s ,λ)=|O(x s ,y s ,λ)| 2 …(Equation 5).

[0055] In equation (5), (x s ,y s ) may represent a spatial coordinate at the plane of the image sensor 218, λ may represent a wavelength, and The convolution operation may be represented. In a particular embodiment, before the light waves generated by the point light source 510 are detected by the image sensor 218, the light waves may pass through a Bayer filter for each RGB color component and / or one or more specific monochrome color components. The final image intensity frequency at each pixel of each RGB channel may be represented as:

[0056]

[0057] In equation (6), K is R, G or B, and F K (λ) may represent the spectral transmittance of the Bayer filter, the display panel 112, and the camera lens 216 at the K channel. Therefore, assuming that the intensity distribution within each RGB channel does not vary by more than the wavelength, equation (6) may be simplified and expressed as:

[0058]

[0059] In equation (7), I O,K It can represent the average intensity distribution on the spectrum at channel K. In addition, in equation (7), h k may represent the PSF integrated over the spectrum at channel K. Equation (7) shows how the image intensity distribution at channel K is related to one or more PSFs, which are primarily degraded by, for example, the effects of the photomask 308 and are based in part on one or more potential aberrations with respect to the camera lens 216. In certain embodiments, the point light source 510 detection process may introduce noise, such as Poisson noise, dark current noise, readout noise, or quantization noise, in some instances. Therefore, in certain embodiments, it is noted that N K is the spatial distribution of the noise at channel K, and the actual image intensity distribution corresponding to one or more PSFs of the electronic device 100 can be expressed as:

[0060]

[0061] In certain embodiments, the electronic device 100 may then store (at a database 514) one or more pre-measured PSFs 512 (e.g., for each RGB color component and / or one or more specific monochromatic color components) in, for example, the memory 106 for later use in reconstructing an image captured by a camera 110 disposed behind a display 112 of the electronic device 100. In certain embodiments, for example, multiple PSFs may be pre-measured in different regions of the image field to capture the variation of the PSF with the angle of the optical axis incident on the display panel 112. The multiple PSFs (e.g., for each RGB color component and / or one or more specific monochromatic color components) may be stored in, for example, the memory 106 for later use in reconstructing pixel regions of an image captured by a camera 110 disposed behind a display 112 of the electronic device 100, and these reconstructed pixel regions may then be combined into a complete reconstructed image.

[0062] Figure 6 A flow chart of a method 600 for reconstructing an image captured by a camera disposed behind a display of an electronic device according to an embodiment of the present disclosure is shown. The method 600 may be performed using one or more processing devices (e.g., one or more processors 104), which may include hardware (e.g., a general purpose processor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a system on a chip (SoC), a microcontroller, a field programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a visual processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), or any other processing device suitable for processing image data), software (e.g., instructions running / executing on one or more processors), firmware (e.g., microcode), or some combination thereof. In certain embodiments, one or more blocks of the method 600 may be executed only once or as needed (e.g., per request / instruction), such as when a photo is taken. In certain embodiments, one or more blocks of the method 600 may be executed continuously and / or iteratively (e.g., automatically run multiple times over a period of time), such as when recording a video, or when a camera application is executing a viewfinder / continuously displaying a camera preview.

[0063] Method 600 may start from block 602, where one or more processing devices (e.g., electronic device 100) captures an original image through a semi-transparent pixel area of ​​a display panel via a camera arranged behind a display panel of the electronic device, wherein the original image includes one or more color components. For example, in a particular embodiment, the electronic device may capture an original image through a semi-transparent pixel area of ​​a display panel of the electronic device by one or more cameras arranged behind the display panel. The original image includes a red color component, a green color component, and a blue color component. In a particular embodiment, the original image may include a bit depth N=10, wherein various color components are synthesized. In a particular embodiment, other values ​​of the bit depth are possible (e.g., N=2, N=8, or N=12, etc.). Then, method 600 may continue at block 604, where one or more processing devices (e.g., electronic device 100) determine a point spread function (PSF) of each of the one or more color components for multiple pixel areas of the original image. For example, in a particular embodiment, the electronic device may determine the PSF of each of the one or more color components by selecting a plurality of predicted PSFs corresponding to the one or more color components from a memory of the electronic device, respectively. In particular embodiments, the various PSFs may be pre-measured based at least in part on a predetermined calibration process performed for the camera and display panel.

[0064] Then, the method 600 may continue at block 606, where one or more processing devices (e.g., the electronic device 100) perform deconvolution of each of the one or more color components of the original image for a plurality of pixel regions of the original image based at least in part on the respective PSFs of the one or more color components of the original image. In a particular embodiment, performing deconvolution of each color component may include performing Richardson-Lucy deconvolution of each color component. In a particular embodiment, performing deconvolution of each color component may include performing Tikhonov regularized inverse filter deconvolution of each color component.

[0065] Then, the method 600 may end at block 608, where one or more processing devices (e.g., the electronic device 100) generate a reconstructed image corresponding to the original image based on the deconvolution of one or more color components of the plurality of pixel regions of the original image. In certain embodiments, generating the reconstructed image corresponding to the original image may include removing a blur effect of the original image. In certain embodiments, generating the reconstructed image corresponding to the original image may also include performing color correction for each color component of the plurality of pixel regions of the original image.

[0066] For example, in certain embodiments, the technique for generating a reconstructed image may be represented by the following equations (9) and (10). S,Kcan be degraded by photomasks, optical aberrations, and detection noise. In order to recover the basic ideal object intensity I O,K , equation (9) can be inverted based on deconvolution techniques. To perform deconvolution, the PSF of each RGB channel can be measured experimentally or generated by wave optics software, for example, as described above with reference to Figure 5A and 5B The inverse expression can be expressed as:

[0067]

[0068] The solution of this optimization expression is An estimate of the true object intensity can be included. The second term can include a regularization function that depends on the total variation, thereby enforcing the sparsity of the gradient in the reconstructed image. In simulations and experiments, the simulated PSFh can be used K To solve equation (9). Richardson-Lucy deconvolution (or Tikhonov regularized inverse filter deconvolution) can be used. The regularization term is temporarily not applicable. Richardson Lucy may include a deconvolution technique based on the assumption that the main noise source is Poisson noise, which can be expressed as:

[0069]

[0070] In equation (10), channel K may be omitted for simplicity, and n may be the number of iterations, and the asterisk is the adjoint operation.

[0071] In this way, the present embodiment can increase the visible area and resolution of the display of the electronic device by arranging the forward-pointing camera of one or more electronic devices behind the display. For example, due to the increase in the display area (e.g., eliminating the display area usually designated for one or more forward-pointing cameras), the electronic device can further provide an improved graphical user interface (GUI), which has a full-screen view as a whole, rather than only displaying battery status, cellular signal strength data, Wi-Fi status, time information, etc., in accordance with the notch design or punch hole design. The present technology can further improve the aesthetic quality of the electronic device and allow the user of the electronic device to display a higher resolution image on the display of the electronic device. Further, since one or more forward-pointing cameras can be placed behind the display, the present technology can allow one or more forward-pointing cameras to be placed in any position (e.g., in the center area of ​​the display), rather than in the corner or edge of the electronic device display. This can provide an improved user experience and / or GUI, such as by guiding the user taking a selfie to look at the center area of ​​the display, and further by giving the impression of eye contact with another user when the user participates in a video conference, videophone exchange or other video streaming services.

[0072] Figure 7 An exemplary computer system 700 according to an embodiment of the present disclosure is shown, which can be used to reconstruct an image captured by a camera arranged behind a display of an electronic device. In a specific embodiment, one or more computer systems 700 perform one or more steps of one or more methods described or illustrated herein. In a specific embodiment, one or more computer systems 700 provide the functionality described or illustrated herein. In a specific embodiment, software running on one or more computer systems 700 performs one or more steps of one or more methods described or illustrated herein, or provides the functionality described or illustrated herein. A specific embodiment includes one or more parts of one or more computer systems 700. Here, where appropriate, references to computer systems may include computing devices, and vice versa. In addition, where appropriate, references to computer systems may include one or more computer systems.

[0073] The present invention relates to any suitable number of computer systems 700. The present invention contemplates that computer system 700 takes any suitable physical form. By way of example and not limitation, computer system 700 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (e.g., a computer on a module (COM) or a system on a module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a computer system grid, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more of these devices. Where appropriate, computer system 700 may include one or more computer systems 700; single or distributed; across multiple locations; across multiple machines; across multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks.

[0074] Where appropriate, one or more computer systems 700 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example and not limitation, one or more computer systems 700 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. Where appropriate, one or more computer systems 700 may perform one or more steps of one or more methods described or illustrated herein at different times or at different locations.

[0075] In a particular embodiment, computer system 700 includes a processor 702, memory 704, storage 706, input / output (I / O) interface 708, communication interface 710, and bus 712. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

[0076] In a particular embodiment, the processor 702 includes hardware for executing instructions, such as instructions that constitute a computer program. By way of example and not limitation, to execute instructions, the processor 702 may retrieve (or fetch) instructions from an internal register, an internal cache, memory 704, or storage 706; decode and execute them; and then write one or more results to an internal register, an internal cache, memory 704, or storage 706. In a particular embodiment, the processor 702 may include one or more internal caches for data, instructions, or addresses. The present invention contemplates that the processor 702 includes any appropriate number of any appropriate internal caches, as applicable. By way of example and not limitation, the processor 702 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of instructions in the memory 704 or storage 706, and the instruction cache may speed up the retrieval of these instructions by the processor 702.

[0077] The data in the data cache may be a copy of data in the memory 704 or storage 706 for instructions executed at the processor 702 to operate; the results of previous instructions executed at the processor 702 for access by subsequent instructions executed at the processor 702 or for writing to the memory 704 or storage 706; or other suitable data. The data cache may speed up the read or write operations of the processor 702. The TLB may speed up the virtual address translation of the processor 702. In a particular embodiment, the processor 702 may include one or more internal registers for data, instructions, or addresses. The present invention contemplates that the processor 702 includes any suitable number of any suitable internal registers (as applicable). Where appropriate, the processor 702 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 702. Although the present invention describes and illustrates a particular processor, the present invention contemplates any suitable processor.

[0078] In a particular embodiment, the memory 704 includes a main memory for storing instructions to be executed by the processor 702 or data to be operated by the processor 702. By way of example and not limitation, the computer system 700 may load instructions from the storage 706 or another source (such as, for example, another computer system 700) to the memory 704. The processor 702 may then load the instructions from the memory 704 to an internal register or an internal cache. To execute the instructions, the processor 702 may retrieve the instructions from the internal register or the internal cache and decode them. During or after the execution of the instructions, the processor 702 may write one or more results (which may be intermediate or final results) to the internal register or the internal cache. The processor 702 may then write one or more of these results to the memory 704. In a particular embodiment, the processor 702 only executes instructions in one or more internal registers or internal caches or memory 704 (relative to or otherwise in the storage 706) and only operates on data in one or more internal registers or internal caches or memory 704 (relative to or otherwise in the storage 706).

[0079] One or more memory buses (each of which may include an address bus and a data bus) may couple the processor 702 to the memory 704. The bus 712 may include one or more memory buses, as described below. In a particular embodiment, one or more memory management units (MMUs) reside between the processor 702 and the memory 704 and facilitate access to the memory 704 requested by the processor 702. In a particular embodiment, the memory 704 includes a random access memory (RAM). Where appropriate, the RAM may be a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port or multi-port RAM. The present invention relates to any suitable RAM. Where appropriate, the memory 704 may include one or more memories 704. Although the present invention describes and illustrates a specific memory, the present invention contemplates any suitable memory.

[0080] In a particular embodiment, storage 706 includes mass storage for data or instructions. By way of example and not limitation, storage 706 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more thereof. Where appropriate, storage 706 may include removable or non-removable (or fixed) media. Where appropriate, storage 706 may be internal or external to computer system 700. In a particular embodiment, storage 706 is a non-volatile solid-state memory. In a particular embodiment, storage 706 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically variable ROM (EAROM), or a flash memory, or a combination of two or more thereof. The present invention contemplates mass storage 706 in any suitable physical form. Storage 706 may include one or more storage control units, where appropriate, to facilitate communication between processor 702 and storage 706. Where appropriate, storage 706 may include one or more storage 706. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0081] In a particular embodiment, the I / O interface 708 includes hardware, software, or both, providing one or more interfaces for communication between the computer system 700 and one or more I / O devices. Where appropriate, the computer system 700 may include one or more of these I / O devices. One or more of these I / O devices can enable communication between a person and the computer system 700. As an example and not limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a trackball, a camera, another suitable I / O device, or a combination of two or more thereof. The I / O device may include one or more sensors. The present invention contemplates any suitable I / O device and any suitable I / O interface 708 for them. Where appropriate, the I / O interface 708 may include one or more device or software drivers so that the processor 702 can drive one or more of these I / O devices. Where appropriate, the I / O interface 708 may include one or more I / O interfaces 708. Although the present invention describes and illustrates a specific I / O interface, the present invention contemplates any suitable I / O interface.

[0082] In certain embodiments, the communication interface 710 includes hardware, software, or both, providing one or more interfaces for communication (such as, for example, packet-based communication) between the computer system 700 and one or more other computer systems 700 or one or more networks. By way of example and not limitation, the communication interface 710 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network (such as a WI-FI network). The present invention contemplates any suitable network and any suitable communication interface 710 thereof.

[0083] As an example and not limitation, the computer system 700 can communicate with one or more parts of a self-organizing network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more thereof. One or more parts of one or more of these networks can be wired or wireless. For example, the computer system 700 can communicate with a wireless PAN (WPAN) (such as, for example, a Bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular phone network (such as, for example, a global system for mobile communications (GSM) network) or other suitable wireless networks, or a combination of two or more thereof. Where appropriate, the computer system 700 may include any suitable communication interface 710 for any of these networks. Where appropriate, the communication interface 710 may include one or more communication interfaces 710. Although the present invention describes and illustrates a specific communication interface, the present invention contemplates any suitable communication interface.

[0084] In certain embodiments, bus 712 includes hardware, software, or two mutually coupled components of computer system 700. By way of example and not limitation, bus 712 may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front side bus (FSB), a HyperTransport (HT) interconnect, an industry standard architecture (ISA) bus, an INFINIBAND interconnect, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI Express (PCIe) bus, a serial advanced technology connection (SATA) bus, a video electronics standard association local (VLB) bus, or other suitable busses or a combination of two or more of the busses. Where appropriate, bus 712 may include one or more buses 712. Although the present invention describes and illustrates a particular bus, the present invention contemplates any suitable bus or interconnect.

[0085] Here, the computer-readable non-transitory storage medium may include one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more thereof (as applicable). The computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile (as appropriate).

[0086] Unless expressly stated otherwise or the context indicates otherwise, the word "or" in this Agreement is inclusive and not exclusive. Thus, herein, "A or B" means "A, B, or both," unless expressly stated otherwise or the context indicates otherwise. In addition, "and" is joint and contingent, unless expressly stated otherwise or the context indicates otherwise. Thus, herein, "A and B" means "A and B, jointly or severally," unless expressly stated otherwise or the context indicates otherwise.

[0087] As used herein, “automatically” and its derivatives mean “without human intervention” unless expressly stated otherwise or the context indicates otherwise.

[0088] The embodiments disclosed herein are only examples, and the scope of the present disclosure is not limited to these embodiments. Embodiments according to the present invention are disclosed in particular in the attached claims for methods, storage media, systems, and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or references in the attached claims are selected only for formal reasons. However, any subject matter resulting from intentional reference to any previous claim (especially multiple dependencies) may also be claimed, thereby disclosing any combination of claims and their features, and may be claimed, regardless of the dependencies selected in the attached claims. The subject matter that may be claimed includes not only the feature combinations described in the attached claims, but also any other combination of the features described in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. In addition, any embodiments and features described or depicted herein may be claimed in separate claims and / or in any combination of any embodiments or features described or depicted herein or any features of the attached claims.

[0089] The scope of the present invention includes all changes, substitutions, changes, alterations and modifications to the exemplary embodiments described or illustrated herein, which will be understood by those of ordinary skill in the art. The scope of the present invention is not limited to the exemplary embodiments described or illustrated herein. In addition, although the present disclosure describes and illustrates various embodiments herein, including specific components, elements, features, functions, operations or steps, any of these embodiments may include any combination or arrangement of any components, elements, features, functions, operations, or steps described or illustrated anywhere herein as will be understood by those of ordinary skill in the art. In addition, in the appended claims, references to devices or systems or parts of devices or systems are adapted to, arranged to, capable of, configured to, capable of, operable to, or operable to perform a specific function, including the device, system, component, whether or not it or the specific function is activated, turned on or unlocked, as long as the device, system or component is so adapted, arranged, capable of, configured, enabled, operable or operable. In addition, although the present invention describes or illustrates specific embodiments as providing specific advantages, specific embodiments may not provide, some or all of these advantages.

Claims

1. A method for image processing, the method comprising: Capturing, via a camera disposed behind a display panel of the electronic device, a raw image through a semi-transparent pixel area of ​​the display panel, wherein the raw image includes one or more color components, wherein the raw image is subjected to modulation of a light path; For each of a plurality of pixel regions of the original image, determining a point spread function (PSF) of a corresponding pixel region of each of the one or more color components; for the plurality of pixel regions of the original image, performing a deconvolution of each of the one or more color components of the original image based at least in part on their respective PSFs; as well as Based on the deconvolution of the one or more color components of the plurality of pixel regions of the original image, a reconstructed image corresponding to the original image is generated, wherein the reconstructed image corrects for modulation of a light path passing through a semi-transparent pixel region of a display panel.

2. The method according to claim 1, wherein: The one or more color components include: Red component; Green component; and Blue component.

3. The method according to claim 1, wherein: Determining the PSF of the corresponding pixel region for each of the one or more color components includes selecting, from a memory of the electronic device, a plurality of predicted PSFs corresponding to the one or more color components, respectively.

4. The method according to claim 1, wherein: The respective PSFs are pre-measured based at least in part on a predetermined calibration process performed with respect to the camera and the display panel.

5. The method according to claim 1, wherein: For the plurality of pixel regions of the original image, performing the deconvolution of each of the one or more color components includes performing a Richardson-Lucy deconvolution of each of the one or more color components.

6. The method according to claim 1, wherein: For the plurality of pixel regions of the original image, performing the deconvolution of each of the one or more color components includes performing a Tikhonov regularized inverse filter deconvolution of each of the one or more color components.

7. The method according to claim 1, wherein: Generating the reconstructed image corresponding to the original image includes removing blurring effects of the original image.

8. The method according to claim 1, wherein: Generating the reconstructed image corresponding to the original image includes performing color correction of each of the one or more color components of the plurality of pixel regions of the original image.

9. An electronic device, comprising: Display panel; a camera, arranged behind the display panel; one or more non-transitory computer-readable storage media comprising instructions; and one or more processors coupled to the storage medium and the camera, the one or more processors configured to execute instructions to: capturing, via the camera, a raw image through a semi-transparent pixel region of the display panel, wherein the raw image includes one or more color components, wherein the raw image is subjected to modulation of a light path; For a plurality of pixel regions of the original image, determining a point spread function (PSF) for each of the one or more color components; for the plurality of pixel regions of the original image, performing a deconvolution of each of the one or more color components of the original image based at least in part on their respective PSFs; as well as Based on the deconvolution of the one or more color components of the plurality of pixel regions of the original image, a reconstructed image corresponding to the original image is generated, wherein the reconstructed image corrects for modulation of a light path passing through a semi-transparent pixel region of a display panel.

10. The electronic device according to claim 9, wherein: The instructions for determining the PSF for each of the one or more color components include instructions for selecting, from a memory of the electronic device, a plurality of predicted PSFs corresponding to the one or more color components, respectively.

11. The electronic device according to claim 9, wherein: The respective PSFs are pre-measured based at least in part on a predetermined calibration process performed with respect to the camera and the display panel.

12. The electronic device according to claim 9, wherein: The instructions to perform the deconvolution of each of the one or more color components for the plurality of pixel regions of the original image include instructions to perform Richardson-Lucy deconvolution of each of the one or more color components.

13. The electronic device according to claim 9, wherein: The instructions to perform the deconvolution of each of the one or more color components for the plurality of pixel regions of the original image include instructions to perform a Tikhonov regularized inverse filter deconvolution of each of the one or more color components.

14. The electronic device according to claim 9, wherein: The instructions for generating the reconstructed image corresponding to the original image include instructions for removing blurring effects of the original image.

15. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of an electronic device, cause the one or more processors to: capturing, via a camera disposed behind a display panel of the electronic device, a raw image through a semi-transparent pixel area of ​​the display panel, wherein the raw image includes one or more color components, wherein the raw image is subjected to modulation of a light path; For a plurality of pixel regions of the original image, determining a point spread function (PSF) for each of the one or more color components; for the plurality of pixel regions of the original image, performing deconvolution of each of the one or more color components of the original image based at least in part on their respective PSFs; as well as Based on the deconvolution of the one or more color components of the plurality of pixel regions of the original image, a reconstructed image corresponding to the original image is generated, wherein the reconstructed image corrects for modulation of a light path passing through a semi-transparent pixel region of a display panel.

Images