Under-display camera system and method
By using adjustment matrix in image capture equipment to compensate the shadow effect of camera sensors under the display, the problem of low intensity and accuracy of camera sensors receiving image information under the display is solved, image quality is improved, and image capture is optimized in low light environments performance.
Patent Information
- Application Number
- CN202411311867.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2020-06-01
- Filing Date
- 2020-10-16
- Publication Date
- 2024-11-12
AI Technical Summary
In an image capturing device, the camera sensor under the display has low intensity and accuracy of received image information due to the shadow effect of the display layer, and there are problems with light scattering and shadowing, which affects the image quality.
By storing subpixel-related parameters and aging factors in the image capture device in the image capture device, an adjustment matrix is created to compensate for image data, improve display transmittance, and manage user interface modes in low-light environments to optimize image capture.
It effectively compensates for the shadow effect of the camera sensor under the display, improves the intensity and accuracy of image reception, improves image quality, and improves the performance of image capture devices in low-light environments.
Smart Images

Figure CN118945492A_ABST
Abstract
Description
[0001] This application is a divisional application of the invention patent application with application date of October 16, 2020, application number 202080076498.3, and name “Under-display camera system and method”.
[0002] CROSS-REFERENCE TO RELATED APPLICATIONS
[0003] This application claims priority to PCT application PCT / CN2019 / 118746 filed on November 15, 2019 and PCT application PCT / CN2020 / 093726 filed on June 1, 2020, the entire contents of which are incorporated by reference. Technical Field
[0004] The present disclosure relates generally to image and camera processing. Background Art
[0005] Image capture devices (e.g., digital cameras) are often integrated into a wide variety of devices. In the present disclosure, an image capture device refers to any device that can capture one or more digital images, including devices that can capture still images and devices that can capture image sequences to record video. By way of example, an image capture device may include a stand-alone digital camera or digital video camera, a wireless communication device cell phone equipped with a camera (e.g., a mobile phone (including a cellular or satellite radio phone)), a tablet computer or personal digital assistant (PDA) equipped with a camera, a computer device including a camera (e.g., a so-called "web camera"), or any device with digital imaging or video capabilities.
[0006] The image capture device can produce images under various lighting conditions (e.g., light sources). For example, the image capture device can operate in environments that include a lot of reflected light or saturated light, and in environments that include high contrast. Some example image capture devices include adjustment modules for automatic exposure control, automatic white balance, and automatic focus, as well as other modules for adjusting the processing performed by the imaging signal processor hardware (e.g., a hue adjustment module). Summary of the invention
[0007] In general, the present disclosure describes a technique for image processing that includes performing display shading compensation for camera sensors under a display (e.g., those used for a front-facing camera). One way to maximize the size of a display on an image capture device is to place one or more camera sensors under the display. When a camera sensor is placed under a display (e.g., a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, an active matrix organic light emitting diode (AMOLED, which may be a specific example of an OLED display), or other display), the layers of the display cause a shadow to the camera sensor, and therefore, the intensity and accuracy of the image information received by the camera sensor is lower than when the camera sensor is not under the display. For example, the layers of the display may attenuate ambient light reaching the camera sensor, and sub-pixels above the camera sensor may cause shadows, such as spatial strides and shadows.
[0008] Sub-pixels include elements that make up pixels, for example, red elements, blue elements, and green elements of RGB pixels. In addition, if the pixel above the camera sensor is displaying content, the camera sensor can capture light scattered by the displayed content. In addition, the transparency of the display (e.g., OLED transparency) within the sub-pixel area may be affected by the sub-pixel value or current intensity. Natural light travels through the display to the camera sensor. The transparency of the display (e.g., OLED transparency) and the transparency of the sub-pixel physical area may be affected by the pixel value of the displayed content. The pixel value (which becomes the driving current of the sub-pixel) will affect the absorption of natural light and the amount of light passing through the display. The area of the display through which natural light passes may have areas with physical pixel elements and areas without physical pixel elements. For areas with physical pixel elements and areas without physical pixel elements, the transparency (or the amount of light passing through the display) may be different. While the display is actively displaying content, the transparency of the pixel may be affected by the pixel value. In addition, as the display ages, the transparency of the display may decrease due to the current and time of driving the pixel.
[0009] In some examples, positioning a camera sensor partially below at least a portion of a display or adjacent to a display may result in display shadows, and the camera sensor may also capture light scattered by displayed content. The present disclosure describes a technique for addressing and / or compensating for these issues with under-display cameras, e.g., camera sensors disposed below, partially below, or adjacent to a display such that light passes through display layers before being received by the camera sensor.
[0010] Additionally or alternatively, in some examples, preset parameters related to the size, shape, and position of sub-pixels in the display can be stored in the image capture device. The image capture device can also determine an aging factor related to the aging state of the pixels in the display above the camera sensor. A portion or all of the displayed content can be captured, and an adjustment matrix, for example, a two-dimensional gain matrix, can be created based on the aging factor, the preset parameters, and / or at least a portion of the content to be displayed. As used herein, a portion of the content to be displayed refers to a portion of one or more frames, one or more entire frames, or a combination of both. A portion of a frame refers to a portion of a frame or an entire frame. The adjustment matrix can be applied to an image captured by the camera sensor to compensate for display shadows.
[0011] The present disclosure also describes a technique for improving transmittance in a display. These techniques for improving transmittance are a supplement or alternative to the image processing techniques of the present disclosure, and can be used together with the image processing techniques disclosed herein, or can be used alone. In one example, a first user interface (UI) mode for use with an image capture application can be automatically selected, and when the image capture application is used in low ambient light conditions, the first UI mode can improve display transparency compared to a second UI mode that can display content, wherein part or all of the content is displayed above the camera sensor. In some examples, for example, in medium ambient light conditions, a third UI mode can be used. In some examples, the selection of a UI mode can be based at least in part on a sensor signal.
[0012] In one example of the present disclosure, an image processing method includes: receiving, at an image capture device, first image data captured by a camera sensor, wherein the camera sensor is configured to receive light through at least a portion of a display on the image capture device; receiving, at the image capture device, at least a portion of displayed content; determining, at the image capture device, an adjustment matrix based on at least a portion of the displayed content; applying the adjustment matrix to the first image data to create second image data; and outputting the second image data.
[0013] In another example, the present disclosure describes an image capture device comprising: a memory; and one or more processors coupled to a camera sensor and the memory and configured to: receive first image data from the camera sensor, the camera sensor being configured to receive light through at least a portion of a display; receive at least a portion of display content; determine an adjustment matrix based on at least a portion of the display content; apply the adjustment matrix to the first image data to create second image data; and output the second image data.
[0014] In another example, the present disclosure describes an image capture device comprising: a unit for receiving first image data from a camera sensor, the camera sensor being configured to receive light through at least a portion of a display; a unit for receiving at least a portion of displayed content; a unit for determining an adjustment matrix based on at least a portion of the displayed content; a unit for applying the adjustment matrix to the first image data to create second image data; and a unit for outputting the second image data.
[0015] In another example, the present disclosure describes a non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: receive first image data from a camera sensor configured to receive light through at least a portion of a display; receive at least a portion of display content; determine an adjustment matrix based on at least a portion of the display content; apply the adjustment matrix to the first image data to create second image data; and output the second image data.
[0016] In one example of the present disclosure, a method includes: receiving, by an image capture device, a signal from a sensor; determining, by the image capture device and at least partially based on the signal, a user interface mode, the user interface mode comprising a first mode or a second mode, wherein the first mode comprises a first number of black pixels in an area of a display, and the second mode comprises a second number of black pixels in the area of the display, the first number being greater than the second number; and receiving, by the image capture device, image data from a camera sensor, wherein the camera sensor is configured to receive light through at least a portion of the display.
[0017] In another example, the present disclosure describes an image capture device comprising: a display configured to display a captured image; a camera sensor under the display, the camera sensor under the display being configured to receive light through at least a portion of the display; a memory configured to store the captured image; and one or more processors coupled to the camera sensor, the display, and the memory and configured to: receive a signal from the sensor; determine a user interface mode based at least in part on the signal, the user interface mode comprising a first mode or a second mode, wherein the first mode comprises a first number of black pixels in an area of the display and the second mode comprises a second number of black pixels in the area of the display, the first number being greater than the second number; and receive image data from the camera sensor.
[0018] In another example, the present disclosure describes an image capture device comprising: a unit for receiving a signal from a sensor; a unit for determining a user interface mode based at least in part on the signal, the user interface mode comprising a first mode or a second mode, wherein the first mode comprises a first number of black pixels in an area of a display and the second mode comprises a second number of black pixels in the area of the display, the first number being greater than the second number; and a unit for receiving image data from a camera sensor, wherein the camera sensor is configured to receive light through at least a portion of the display.
[0019] In another example, the present disclosure describes a non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: receive a signal from a sensor; determine a user interface mode based at least in part on the signal, the user interface mode comprising a first mode or a second mode, wherein the first mode comprises a first number of black pixels in an area of a display and the second mode comprises a second number of black pixels in the area of the display, the first number being greater than the second number; and receive image data from a camera sensor, wherein the camera sensor is configured to receive light through at least a portion of the display.
[0020] The details of one or more aspects of the present disclosure are set forth in the accompanying drawings and the following description. Other features, objectives, and advantages of the techniques described in the present disclosure will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 is a block diagram of an example image capture device that is configured to implement the techniques of this disclosure.
[0022] Figures 2A-2D is a block diagram illustrating an example of an image capture device with a front-facing camera sensor and a display.
[0023] Figure 3 is a block diagram illustrating a cross-sectional view of an example image capture device configured to implement the techniques of this disclosure.
[0024] Figure 4A-4B is a block diagram illustrating properties of an OLED display according to different examples of the technology of this disclosure.
[0025] Figure 5 is a conceptual diagram showing an example of zoning control that can be used with the technology of the present disclosure.
[0026] Figures 6A-6Cis a conceptual diagram showing a possible physical layout of RGB subpixels in a display.
[0027] Figure 7 is a block diagram of an example image capture device that may implement the techniques of this disclosure.
[0028] Figures 8A-8C is a conceptual diagram illustrating different example UI modes in an area of a display above an under-display camera sensor, in accordance with techniques of this disclosure.
[0029] Figures 9A-9C is a conceptual diagram showing a further example of the first mode of the technology according to the present disclosure.
[0030] Fig.10 is a block diagram illustrating an example image signal processing (ISP) pipeline in accordance with techniques of this disclosure.
[0031] Figures 11A-11C is a conceptual diagram representing determination of preset parameters according to the technology of the present disclosure.
[0032] Fig.12 is a conceptual diagram illustrating a technique for synchronous display shading compensation.
[0033] Fig.13 is a flow chart illustrating an example display shading compensation technique according to the present disclosure.
[0034] Fig.14 is a flow chart illustrating an example display shading compensation technique according to the present disclosure.
[0035] Fig.15 is a flow diagram illustrating an example synchronization technique according to the present disclosure.
[0036] Fig.16 is a flow chart illustrating an example of determining a user interface according to the techniques of this disclosure.
[0037] Fig.17 is a flow chart illustrating another example of determining a user interface according to the technology of this disclosure. DETAILED DESCRIPTION
[0038] The present disclosure describes an image processing technique that is responsible for and / or compensates for display shadows caused by a camera sensor being disposed below at least a portion of a display. The display may use a transparent material whose pixel structure is designed so that light can penetrate the display to reach the camera sensor. The camera sensor used in this manner may be larger than other front-facing "selfie" cameras and may have a wider aperture lens. For example, the camera sensor size does not need to be limited or constrained by a bezel or border space around the display. By positioning the camera sensor under the display on the device so that the camera sensor can receive light through at least a portion of the display, the size of the available display space may be expanded compared to a similarly sized device having a space dedicated to the camera sensor on the front of the device. Alternatively, a smaller form factor may be used to provide the same available display size. In addition, by positioning the camera sensor under the display, the camera sensor can be placed anywhere under the display. For example, the camera sensor can be located where the user's eyes may be pointed when taking a "selfie". In this way, the gaze of the eyes in the image captured by the camera sensor may appear to be looking toward the camera rather than below the camera, as may occur when the camera sensor is located above the display or near the top of the image capture device.
[0039] In many image capture devices, it may be desirable to maximize the size of the display on the image capture device. This is particularly true for smaller image capture devices (e.g., mobile phones and other mobile devices). Many image capture devices (e.g., mobile devices) include a front-facing camera (a "selfie" camera) that faces the user of the mobile device. Maximizing the size of the display on an image capture device with a front-facing camera is not without limitations. The front-facing camera has been located on the front of the image capture device, between the edge of the device and the edge of the display. In order to maximize the size of the display on an image capture device with a front-facing camera, some manufacturers have enlarged the display and introduced a notch on the display to avoid the display covering the camera sensor. Other manufacturers have enlarged the display so that it substantially covers the front of the image capture device and added a pop-up camera instead of placing the camera sensor on the body of the image capture device.
[0040] One way to maximize the size of a display is to position the camera sensor under the display. However, by positioning the camera sensor under the display, the display may cause attenuation, spatial stepping and shadowing, light scattering or diffusion, and / or other undesirable effects in the image signal captured by the camera sensor. For example, haze, glare, and / or color cast may affect the quality of the captured image. These problems may worsen as the display ages. For example, the current used to drive the sub-pixel may cause the sub-pixel to lose brightness (e.g., dim). In the case where the area of the display above the camera sensor is used to display content, the impact on image quality may be more severe. For example, some or all of the sub-pixels directly above the camera sensor may be actively displaying content, and the camera sensor may capture light scattered from the displayed content when capturing the image. Generally, the aforementioned undesirable effects of using a camera under a display can be referred to as display shadows.
[0041] The present disclosure describes a technique for display shading compensation. The display shading compensation technique of the present disclosure can use characteristics of the physical layout of the transparent display, the characteristics of the content displayed above the camera sensor, and the aging state of the pixels above the camera sensor to compensate for display shading experienced in any captured image.
[0042] The present disclosure also describes a UI technique for managing a display in low light situations. For example, in low light situations, less light may pass through the display to a camera sensor under the display than in high light situations. The present disclosure describes techniques for improving light transmittance through the display so that the camera sensor can receive enough light to capture an aesthetically pleasing image.
[0043] Figure 1 is a block diagram illustrating a device 2 that may be configured to perform the techniques of the present disclosure. Device 2 may constitute part of an image capture device, or may constitute part of a digital video device capable of encoding and transmitting and / or receiving still images and / or video sequences. By way of example, device 2 may constitute part of a wireless mobile communications device such as a cellular or satellite radio telephone, a smartphone, a stand-alone digital camera or video camera, a personal digital assistant (PDA), a tablet computer, a laptop computer, or any device having imaging or video capabilities where image processing is desired.
[0044] like Figure 1As shown in FIG. 1 , the apparatus 2 includes an image processing device 4 for storing raw image data and performing various processing techniques on such data. The image processing device 4 may include one or more integrated circuits including a digital signal processor (DSP), on-chip memory, and possible hardware logic or circuitry. More generally, the image processing device 4 may include any combination of a processor, hardware, software, or firmware, and the various components of the image processing device 4 may be implemented as such. Furthermore, if desired, the image processing device 4 may include a single integrated chip or encoder / decoder (CODEC).
[0045] exist Figure 1 In the example shown, the image processing device 4 includes a local memory 8, a memory controller 10, and an image signal processor 6. The image signal processor 6 may be a general purpose processing unit, or may be a processor specifically designed for imaging applications (e.g., for handheld electronic devices). As shown, the image signal processor 6 is coupled to the local memory 8 and the external memory 14 via the memory controller 10. In some examples, the local memory 8 may be incorporated into the image signal processor 6, for example, as a cache memory.
[0046] like Figure 1, the image signal processor 6 may be configured to perform an automatic exposure control (AEC) process 20, an automatic white balance (AWB) process 22, an automatic focus (AF) process 24, a display shading compensation (DSC) process 26, a lens shading compensation (LSC) process 28, and / or a fixed pattern noise compensation (FPNC) process 30. In some examples, the image signal processor 6 may include hardware-specific circuitry (e.g., an application-specific integrated circuit (ASIC)) configured to perform the AEC process 20, the AWB process 22, the AF process 24, the DSC process 26, the LSC process 28, and / or the FPNC process 30. In other examples, the image signal processor 6 may be configured to execute software and / or firmware to perform the AEC process 20, the AWB process 22, the AF process 24, the DSC process 26, the LSC process 28, and / or the FPNC process 30. When configured in software, code for the AEC process 20, AWB process 22, AF process 24, DSC process 26, LSC process 28, and / or FPNC process 30 may be stored in the local memory 8 and / or the external memory 14. In other examples, the image signal processor 6 may use a combination of hardware, firmware, and / or software to perform the AEC process 20, AWB process 22, AF process 24, DSC process 26, LSC process 28, and / or FPNC process 30. When configured as software, the AEC process 20, AWB process 22, AF process 24, DSC process 26, LSC process 28, and / or FPNC process 30 may include instructions that configure the image signal processor 6 to perform various image processing and device management tasks, including the DSC techniques of the present disclosure.
[0047] AEC process 20 may include instructions for configuring, calculating, storing, and / or applying exposure settings for camera module 12. Exposure settings may include shutter speed and aperture settings for capturing images. According to the techniques of the present disclosure, image signal processor 6 may use depth information captured by camera module 12 to better identify the subject of the image and make exposure settings based on the identified subject. AF process 24 may include instructions for configuring, calculating, storing, and / or applying autofocus settings for camera module 12.
[0048] The AWB process 22 may include instructions for configuring, calculating, storing, and / or applying AWB settings (e.g., AWB gains) that may be applied to one or more images captured by the camera module 12. In some examples, the AWB gain determined by the AWB process 22 may be applied to the image from which the AWB gain is determined. In other examples, the AWB gain determined by the AWB process 22 may be applied to one or more images captured after the image from which the AWB gain is determined. Thus, the AWB gain may be applied to a second image captured after the first image from which the AWB gain is determined. In one example, the second image may be an image captured immediately after the first image from which the AWB gain is determined. That is, if the first image is frame N, the second image to which the AWB gain is applied is frame N+1. In other examples, the second image may be an image captured two images after the first image from which the AWB gain is determined. That is, if the first image is frame N, the second image to which the AWB gain is applied is frame N+2. In other examples, the AWB gain may be applied to an image captured later in time than the first image (eg, frame N+3, frame N+4, etc.) In other examples, the AWB gain may be applied to the first image from which the AWB gain was determined.
[0049] The DSC process 26 may include instructions for configuring, calculating, storing, and / or applying display shading compensation gains. For example, the DSC process 26 may receive first image data captured by a camera sensor. The camera sensor may be disposed below at least a portion of the display. The DSC process 26 may perform the following operations: receiving at least a portion of the display content; determining an adjustment matrix, such as a two-dimensional gain matrix, based on preset parameters, at least a portion of the displayed content, and / or an aging state of pixels above the camera sensor; and applying the adjustment matrix to the first image data to create second image data; and outputting the second image data. The preset parameters may indicate the shape, size, or position of sub-pixels in the display, and may represent compensation parameters for compensating for problems such as shading or coloring caused by the display sub-pixel layout (e.g., the size, shape, and position of sub-pixels in the display). In some examples, the preset parameters may also indicate the location of active pixels in the display, and may represent compensation based on partition control (e.g., where a subset of all pixels in a given area may actively display content). In some examples, the preset parameters may be used as described below with respect to Figures 8A-8CThe grayscale color check card described above is used to determine to generate multiple adjustment matrices, and the DSC process 26 can compensate at least a portion of the displayed content (e.g., those above the camera sensor) by calculating the average of the sum of the sub-pixel values (e.g., (R+G+B) / number of pixels*3), and use the average as an index to select or adjust one of the adjustment matrices. As used herein, an average is an example of a statistical measurement.
[0050] The LSC process 28 may include instructions for configuring, calculating, storing, and / or applying lens shading compensation gains. For example, the LSC process 28 may compensate for light falloff toward the edges of an image due to a camera lens.
[0051] The FPNC program 30 may include instructions for configuring, calculating, storing, and / or applying a FPN compensation process. For example, the FPNC process 30 may subtract a primary dark frame from a captured image to compensate for FPN.
[0052] The local memory 8 may store raw image data, and may also store processed image data after any processing performed by the image signal processor 6. The local memory 8 may be formed of any of a variety of non-transitory memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The memory controller 10 may control the memory organization within the local memory 8. The memory controller 10 may also control memory loading from the local memory 8 to the image signal processor 6 and writing back from the image signal processor 6 to the local memory 8. Images to be processed by the image signal processor 6 may be loaded directly into the image signal processor 6 from the camera module 12 after image capture, or may be stored in the local memory 8 during image processing.
[0053] As previously noted, the device 2 may include a camera module 12 to capture images to be processed, although the present disclosure is not necessarily limited in this regard. The camera module 12 may include an array of solid-state sensor elements, such as complementary metal oxide semiconductor (CMOS) sensor elements, charge coupled device (CCD) sensor elements, etc. Alternatively or in addition, the camera module 12 may include a set of image sensors that include color filter arrays (CFA) arranged on corresponding sensor surfaces. The camera module 12 may be directly coupled to the image signal processor 6 to avoid delays in image processing. The camera module 12 may be configured to capture still images or full-motion video sequences, in which case image processing may be performed on one or more image frames of the video sequence.
[0054] The camera module 12 may send pixel values (e.g., in Bayer or RGB format) and / or raw statistical information describing the captured image to the image signal processor 6. The information obtained from the camera module 12 may be used in the DSC process 26, which will be described in detail below. In general, the image signal processor 6 may be configured to analyze the raw statistical information and the depth information to calculate and / or determine imaging parameters, such as sensor gain, R / G / B gain, AWB gain, shutter speed, aperture size, etc. The calculated and / or determined imaging parameters may be applied to the captured image, applied to one or more subsequently captured images, and / or sent back to the camera module 12 to adjust exposure and / or focus settings.
[0055] Device 2 may include a display 16 that displays the image after the image processing described in the present disclosure. After such image processing, the image may be written to local memory 8 or external memory 14. The processed image may then be sent to display 16 for presentation to a user. Display 16 may display other information, including files stored in memory locations (e.g., external memory 14), software applications installed in image signal processor 6, user interfaces, network-accessible content objects, and visual representations of other information.
[0056] In some examples, device 2 may include multiple memories. For example, device 2 may include external memory 14, which typically includes a relatively large memory space. For example, external memory 14 may include DRAM or flash memory. In other examples, external memory 14 may include non-volatile memory or any other type of data storage unit. Compared with external memory 14, local memory 8 may include smaller and faster memory space, although the present disclosure is not necessarily limited in this regard. By way of example, local memory 8 may include SDRAM. In any case, external memory 14 and local memory 8 are merely exemplary and may be combined into the same memory portion, or may be implemented in any number of other configurations. In some examples, external memory 14 may store a first UI and a second UI. For example, the first UI may display black pixels in an area above a camera sensor under a display, and the second UI may display non-black pixels in an area above a camera sensor under a display.
[0057] The device 2 may also include a transmitter (not shown) to transmit the processed image or the encoded image sequence to another device. In fact, the disclosed techniques may be used in handheld wireless communication devices (e.g., smart phones) that include digital camera functionality or digital video capabilities. In this case, the device will also include a modulator-demodulator (MODEM) to facilitate wireless modulation of the baseband signal onto a carrier waveform, thereby facilitating wireless communication of the modulated information.
[0058] The local memory 8, display 16 and external memory 14 (and other components if necessary) can be coupled via a communication bus 15. Several other elements may also be included in the device 2, but for simplicity and ease of description, Figure 1 Not specifically shown. Figure 1 The architecture shown in is merely exemplary, as the techniques described herein may be implemented using a variety of other architectures.
[0059] Figures 2A-2D 1 is a block diagram illustrating examples of image capture devices (e.g., smart phones). Each image capture device is depicted as having a display and a front-facing camera sensor. In this context, the front-facing camera sensor is the camera sensor that faces the user of the image capture device in typical operation. For example, the front-facing camera sensor is typically on the same side of the device as the main display. Each front-facing camera sensor can be part of a camera module (e.g., camera module 12). For example, in Figure 2A , the image capture device 200 includes a display 202, a camera sensor 204, and a button 206. The button 206 can serve a variety of purposes, such as waking up the image capture device 200, changing the content displayed on the display 202, etc. It can be seen that the button 206 and the camera sensor 204 occupy space in front of the image capture device 200. By positioning the camera sensor 204 and the button 206 in front of the image capture device 200, a smaller area can be used for the display 202.
[0060] exist Figure 2B In the example of , on the other hand, the image capture device 210 does not have buttons on the front. In this case, the buttons can be on the side, or the functionality of the buttons can be included in the display 212 (e.g., via a touch display interface). The image capture device 210 is depicted with a camera sensor 214 and a notch 216. The notch 216 can be an area that is removed from the display before assembling the image capture device 210. In this example, the area covered by the notch 216 is therefore not part of the display 212 and does not display content. The notch 216 can be employed so as to be compatible with Figure 2A Compared with image capturing device 200, the ratio of the front side of image capturing device 210 occupied by display 212 is increased.
[0061] exist Figure 2C In the example of the image capture device 220, the image capture device 220 has a display 222 and a pop-up camera 226. The camera sensor 224 may be included in the pop-up camera 226. In the example of the image capture device 220, the entire display 222 may display content and there may be no such as Figure 2B A notch in the image capture device 210 is provided.
[0062] exist Figure 2D In the example of , image capture device 230 has display 232 and camera sensor 234. In some examples, image capture device 230 may have more than one camera sensor. For example, image capture device 230 may have camera sensor 234 and camera sensor 238. Image capture device 230 may include device 2 or be an example of device 2, and display 232 may be an example of display 16. Figure 2D In the example of the image capture device 230, Figures 2A-2C Unlike the example of , camera sensor 234 and camera sensor 238 are disposed below display 232. In some examples, a portion of camera sensor 234 or camera sensor 238, but not all, may be disposed below display 232. Display 232 may include a transparent layer. Region 232A of display 232, region 232B of display 232, and region 232C of display 232 may be combined. Figure 5 , Figure 7 9. Although the techniques of this disclosure are generally described with reference to an image capture device having a camera sensor disposed below a display (e.g., image capture device 230), the techniques of this disclosure may be used with other image capture devices (e.g., image capture devices 200, 210, and 220 or image capture devices having a camera sensor disposed partially below a display).
[0063] Reference now Figures 2A-2DIn each of the embodiments, image capture device 200 has a larger form factor than image capture devices 210, 220, and 230, but has a display 202 that is the same size as display 222 of image capture device 220 and display 232 of image capture device 230, and its display 202 is slightly larger than display 212 of image capture device 210 due to notch 216. Image capture device 210 has the same form factor as image capture devices 220 and 230, but image capture device 210 has less available display space on display 212 due to notch 216. In addition, notch 216 may be distracting to some users. Image capture device 220 has the same form factor and available display space as image capture device 230, but image capture device 220 has movable parts in pop-up camera 226. These movable parts may become damaged or stuck under repeated use or if a user drops image capture device 220 on a hard surface. Therefore, it may be desirable to position the camera sensor below the display because positioning the camera sensor under the display can maximize the display space while avoiding notches and moving mechanical parts.
[0064] In addition, for Figure 2D In the case of image capture device 230, camera sensor 234 can be positioned anywhere below display 232. In this example, the camera sensor is shown as being located in the middle of display 232. Such a position may be more desirable than the position of the front-facing camera sensors in image capture device 200, image capture device 210, and image capture device 220. For example, a user attempting to take a "selfie" may look toward a real-time image of themselves on the display of the image capture device. The farther the camera sensor is from where the user's eyes are pointed, the more likely the captured image will depict the eyes looking away from the camera sensor. This eye-gazing phenomenon may result in aesthetically unpleasing images in which the user's gaze appears to be below (or above) where they might be expected to be looking (e.g., not toward the eyes of the viewer of the image, but downward (or upward) from the eyes of the viewer of the image, e.g., below or above the head of the viewer of the image).
[0065] Figure 3 yes Figure 2D 230. For simplicity, the camera sensor 238 is not shown in FIG. Figure 3 or the remaining figures, and may function similarly to the camera sensor 234 described herein. Figure 3In the example of , image capture device 230 includes display 232, camera sensor (CS) 234, and housing 236. Housing 236 may include electronic circuit boards, processors, memory, batteries, radio frequency circuits, antennas, and other components. As shown, display 232 is disposed above camera sensor 234, and camera sensor 234 is disposed below or below display 232. In this example, as shown in FIG. Figure 2D , the camera sensor 234 is a front-facing camera. The camera sensor 234 is configured to capture images by capturing ambient light passing through the display 232. That is, the camera sensor 234 can receive ambient light that passes through at least a portion of the display 232 before being incident on the camera sensor 234. As used herein, the camera sensor is under, below or below the display or the display is above the camera sensor, which is intended to describe that the camera sensor is configured and positioned so that an image is captured by capturing ambient light passing through a display (such as the display 232). The display 232, when displaying content, can emit light toward the user and generally away from the camera sensor 234. The camera sensor 234 can capture images actively when the display 232 is actively displaying content or in this case. In other words, the camera sensor 234 can receive ambient light passing through at least a portion of the display 232 while the display 232 can emit light toward the user.
[0066] Figure 4A and Figure 4B is a simplified diagram of an example OLED display that can be used in accordance with the techniques of this disclosure. Figure 4A and Figure 4B The display of is described as an OLED display, but the techniques of this disclosure can be used with any display (e.g., LCD, LED, AMOLED, or other display) that is configured to allow light to pass through the display to a camera sensor located below the display. Figure 4A and Figure 4B Although the example OLED display is depicted as having three layers in FIG. 1 , the OLED display may be composed of more layers.
[0067] exist Figure 4A In the example of , OLED display 250 includes cathode layer 252, organic light emitting material layer 254, and anode layer 256. For example, when current flows between cathode layer 252 and anode layer 256 through organic light emitting material layer 254, OLED display 250 can emit light, thereby causing (multiple) images to appear on OLED display 250. In this way, organic light emitting material layer 254 can emit light toward a user through cathode layer 252. In some examples, camera sensor 234 can receive ambient light while organic light emitting material layer 254 can emit light. Figure 4AIn the example of , ambient light can impinge on the surface of cathode layer 252. A portion of the ambient light can pass through cathode layer 252, organic light emitting material layer 254, and anode layer 256. In this example, cathode layer 252 and anode layer 256 may not be transparent. In addition, organic light emitting material layer 254 may have RGB, RGBW, WRGB (where W is white), RGBG, or other sub-pixels that can block, attenuate, or distort ambient light from passing through organic light emitting material layer 254. Therefore, the amount of ambient light passing through OLED display 250 may be relatively small (shown as transmitted light). Therefore, the camera sensor 234 for receiving transmitted light below OLED display 250 may not receive very much transmitted light, as indicated by the thin arrow. This may result in poor image quality of the image captured by the camera sensor.
[0068] exist Figure 4B In the example of , OLED display 260 includes a transparent cathode layer 262, an organic light emitting material layer 264 and a transparent anode layer 266. Figure 4A In the example of , when current flows between transparent cathode layer 262 and transparent anode layer 266 through organic light emitting material layer 264, OLED display 260 can emit light, thereby causing (multiple) images to appear on OLED display 250. In this way, organic light emitting material layer 264 can emit light toward a user through transparent cathode layer 262. Figure 4B In the example of , because both the transparent cathode layer 262 and the transparent anode layer 266 are transparent, more ambient light can be transmitted through the OLED display 260 to the camera sensor 234. The camera sensor 234 can receive ambient light (shown as transmitted light, as indicated by the thick arrow) that passes through at least a portion of the OLED display 260 before being incident on the camera sensor 234. In some examples, the camera sensor 234 can receive ambient light while the organic light emitting material layer 264 can emit light. Figure 4B In the example, Figure 4A As in the example of , the organic light emitting material layer 264 may include RGB, RGBW or WRGB sub-pixels that may block, attenuate or distort ambient light passing through the organic light emitting material layer 264. In summary, Figure 4B The attenuation or distortion in the example may be greater than Figure 4A The attenuation or distortion is smaller, for example, because the transparent cathode layer 262 and the transparent anode layer 266 are transparent. However, by allowing current to flow between the cathode layer 252 and the anode layer 256, the OLED display 250 may age, which may cause the transparency to decrease over time.
[0069] Figure 5is a conceptual diagram depicting an example of zoning control that may be used in accordance with the techniques of the present disclosure. With zoning control, one region or zone of a display actively uses all of the pixels of the one region or zone to display content, while another region or zone of the display may only actively use some of the pixels of the other region or zone to display content. Figure 5 Two different views of an image that may be displayed are depicted, where each box in each view represents a pixel in the display. View 270 depicts an image that may be displayed in an area of a display (e.g., display 232) that may not be located above a camera sensor (e.g., camera sensor 234). Figure 2D 270 , this area of the display is shown as area 232A. All pixels in area 232A can be used to display content, such as that shown in view 270 .
[0070] View 272 depicts an area of the display that may be above the camera sensor (eg, Figure 2D In view 272, a subset of pixels actively displays content. In some examples, region 232B may have the same size as camera sensor 234. In other examples, region 232B may have a different size from camera sensor 234. In some examples, region 232C may have the same size as camera sensor 238. In other examples, region 232C may have a different size from camera sensor 238. In some examples, region 232B or region 232C may be determined in a laboratory. In some examples, region 232B or region 232C may be determined by an optical path that affects a camera sensor (e.g., camera sensor 234 or camera sensor 238). For example, a scientist or engineer may test various layouts of region 232B or region 232C and select a layout that balances the aesthetics of the display and the ability of light to pass through the display to camera sensor 234 or camera sensor 238.
[0071] In the example of view 272, the position of the active pixel utilized by display 232 is the top left pixel (e.g., pixel 271) in every group of four pixels (e.g., group 274 separated from other groups of four pixels by a white dashed line) to display content, and other pixels in every group of four pixels are not utilized. This particular configuration is a non-limiting example. Other configurations may be utilized in accordance with the techniques of this disclosure.
[0072] In some examples, view 272 may be displayed in region 232B above camera sensor 234 only when camera sensor 234 is actively used to capture (multiple) images, and view 270 (e.g., using all pixels) may be displayed in region 232A. In some examples, view 272 may be displayed in region 232C above camera sensor 238 only when camera sensor 238 is actively used to capture (multiple) images, and view 270 (e.g., using all pixels) may be displayed in region 232A. In some examples, when camera sensor 234 is not actively used to capture (multiple) images, view 270 (e.g., using all pixels) may be used in regions 232A, 232B, and 232C. For example, when device 2 is executing an image capture application (e.g., a camera application or a video application), view 272 may be displayed in region 232B.
[0073] When camera sensor 234 is actively used to capture image(s), it may be desirable for view 272 to be displayed in region 232B to reduce light scattering and distortion of ambient light that may be caused by displayed content and captured by camera sensor 234. Thus, a subset of pixels in region 232B may be active when camera sensor 234 is actively capturing image(s), or a subset of pixels in region 232C may be active when camera sensor 238 is actively capturing image(s). By reducing the number of pixels above camera sensor 234 (or camera sensor 238) that display content during image capture, light scattering and distortion in captured images may be reduced.
[0074] In some examples, view 272 may not be displayed in region 232B when camera sensor 234 is actively used to capture image(s), or view 272 may not be displayed in region 232C when camera sensor 238 is actively used to capture image(s). For example, it may be more desirable to display all pixels (view 270) across the entire display 232 (regions 232A, 232B, and 232C).
[0075] Figures 6A-6C is a conceptual diagram depicting an example layout of sub-pixels in a display according to the techniques of the present disclosure. As mentioned above, a sub-pixel is an element of a pixel, for example, a red element, a blue element, or a green element of an RGB pixel. Figures 6A-6C The example of shows RGB sub-pixels, but the display may contain RGBW, WRGB or other sub-pixel layouts. Fig. 6AIn the example of FIG. 2 , layout 276 includes three columns of blue (B) sub-pixels, as shown. Between each column of blue sub-pixels there are alternating green (G) sub-pixels and red (R) sub-pixels. There are spaces 277 between these sub-pixels. Figure 6B In the example of , the layout 278 includes two columns of green (G) sub-pixels, as shown. On either side of the green sub-pixel columns are alternating columns of blue (B) sub-pixels and red (R) sub-pixels. There are spaces 279 between these sub-pixels. Figure 6C In the example of FIG. 2 , layout 280 includes several rows of circular sub-pixels. Each of these sub-pixels can be a green (G) sub-pixel. Between the rows of green sub-pixels there are alternating red (R) sub-pixels and blue (B) sub-pixels in a diamond shape. Between these sub-pixels there are spaces 281. Figures 6A-6C These are just a few examples of potential layouts of sub-pixels in a display. The shape, size, and location of the sub-pixels is a matter of design choice for the display manufacturer. Thus, the layout of the sub-pixels may vary from one manufacturer to another or from one type of display to another. Information related to the layout (e.g., shape, size, and location) of the sub-pixels may be obtained from the manufacturer of the display.
[0076] When a camera sensor (e.g., camera sensor 234) is located below a display (e.g., display 232), camera sensor 234 may receive ambient light through spaces (e.g., spaces 277, 279, or 281) between sub-pixels. For example, one or more camera sensors may be disposed below or below at least a portion of a display layer such that light passes through the display layer before being received by the one or more sensors. Although light may pass through the display layer before being received by one or more camera sensors (e.g., camera sensor 234), the sub-pixels may shadow portions of camera sensor 234 and may cause problems such as haze, glare, and / or color cast in the captured image(s). In addition, as discussed above, displaying content above a camera sensor (e.g., camera sensor 234) may also affect image quality due to light scattering of the display content that may be captured by camera sensor 234. The display shading compensation techniques of the present disclosure may compensate for shadows caused by sub-pixels and light scattering caused by display content, thereby improving the image quality of images captured by an image capture device having a camera sensor below a display.
[0077] Figure 7 is a block diagram of an example image capture device that may implement the techniques of the present disclosure. Image capture device 102 may be Figure 1By way of example, the image capture device 102 may include a wireless mobile communication device, such as a cellular or satellite radio telephone, a smart phone, a stand-alone digital camera or video camera, a personal digital assistant (PDA), a tablet computer, a laptop computer, or any device with imaging or video capabilities in which image processing is desired.
[0078] Image capture device 102 may include one or more processors 110, camera sensor 112, image signal processor 106, memory 114, display 116, communication circuitry 118, and ambient light sensor 122. Display 116 may include area 120 (which may be located above camera sensor 112) Figure 2D In some examples, image signal processor 106 may be one of one or more processors 110. In some examples, image signal processor 106 may be more than one of one or more processors. In some examples, image signal processor 106 may be separate from one or more processors 110, as depicted.
[0079] The memory 114 may include an image capture application 104. The image capture application 104 may be an application used by a user to turn on the camera function of the image capture device 102. The memory 114 may also store a first camera UI (first UI 105), a second camera UI (second UI 108), and a predetermined threshold (threshold 107). In some examples, the memory 114 may also be a third camera UI (third UI 111). The memory 114 may also be configured to store pixel values associated with an image captured by the camera sensor 112. The memory 114 may also store instructions for causing one or more processors 110 to perform the techniques of the present disclosure.
[0080] The camera sensor 112 can capture pixel values of an image when activated. For example, the camera sensor 112 can capture pixel values when the one or more processors 110 are executing the image capture application 104. The image signal processor 106 can process the pixel values captured by the camera sensor 112.
[0081] The one or more processors 110 may obtain pixel values from the image signal processor 106 and may provide the pixel values to a memory 114 for storage, to a communication circuit 118 for transmission to another device, or to a display 116 for display to a user. When the camera is off (e.g., when the one or more processors 110 are not executing the image capture application 104), the one or more processors 110 may obtain pixel values from the memory 114 and provide the pixel values to the display 116 for display, for example.
[0082] In some examples, according to the technology of the present disclosure, the image capture device 102 includes a display 116 configured to display a captured image. The image capture device 102 includes a camera sensor 112. The camera sensor 112 is configured to receive light through at least a portion of the display. The image capture device 102 includes a memory 114 configured to store the captured image. The image capture device 102 includes one or more processors 110, which are coupled to the camera sensor 112, the display 116 and the memory 114 under the display. The one or more processors 110 are configured to receive a signal from the sensor. The one or more processors 110 are configured to determine a user interface mode based at least in part on the signal, the user interface mode including a first mode (e.g., a first UI 105) and a second mode (e.g., a second UI 108), wherein the first mode includes a first number of black pixels in an area of the display, and the second mode includes a second number of black pixels in an area of the display, the first number being greater than the second number. The one or more processors 110 are also configured to receive image data from the camera sensor 112.
[0083] Ambient illumination (e.g., ambient light) may be important to an under-display camera sensor (e.g., camera sensor 112) because the ambient illumination may affect automatic exposure control, autofocus, and auto white balance in the image capture device 102. For example, in a dark, low lux environment, the image captured by the camera sensor 112 may be relatively poor. For example, if the ambient lux is 8000 lux, the camera sensor 112 may function well with proper calibration. If the ambient lux is 20 lux, the image captured by the camera sensor 234 may be poor, and complex adjustments may be required to improve the image. If the ambient lux is 0.2 lux, and no flash is used, the image captured by the camera sensor 234 may be so poor that the image may not be repairable.
[0084] According to the technology of the present disclosure, the image capture device 102 can display black pixels in the area 120 above the camera sensor 112 in a first mode, for example, in a low lux environment. As used herein, "black pixels" include pixels with a grayscale value of 0, blank pixels, or unaddressed pixels. By displaying black pixels in the area 120 above the camera sensor 112, the transmittance of the display 116 in the area 120 can be improved, thereby facilitating automatic exposure control, autofocus, and automatic white balance (which can be part of the image signal processor 106) of the image capture device 102. However, always displaying black pixels in the area 120 may interfere with the normal display UI and may not be aesthetically pleasing to the user of the image capture device 102. Therefore, it may be desirable to provide a technology for displaying black pixels in the area 120 above the camera sensor 112 only during a low lux environment.
[0085] One technique for controlling the selection of the UI may be a technique based on automatic exposure control (AEC). AEC may use histogram statistics of consecutive frames and input pixel dynamic range analysis to control the gain and exposure time of the camera sensor. AEC may utilize 5-15 frames of data to determine the appropriate gain-exposure time balance setting. In addition, the AEC process is also affected by the pixel values in the area of the display above the camera sensor.
[0086] In some examples, once the image capture application 104 is launched, the one or more processors 110, through, for example, the image capture application 104, the system camera service, or the camera hardware abstraction layer (HAL), can use the ambient light sensor 122 to check the ambient lux value. If the ambient lux value is greater than a predetermined threshold, the ambient light can be considered high. In this case, the display transmittance is not necessarily critical. Therefore, the one or more processors 110 can launch the second UI 108.
[0087] If the ambient lux value is less than a predetermined threshold, the ambient light may be considered low. In this case, the display transmittance may be more important for proper image acquisition. Therefore, the one or more processors 110 may initiate the first UI 105. This means that the pixels in the area of the display above the camera sensor may have a 0 pixel value to increase the transmittance of the area of the display above the camera sensor.
[0088] For example, when the camera is turned on (e.g., when the one or more processors 110 are executing the image capture application 104), the one or more processors 110 can determine the ambient light level. For example, the one or more processors can query the ambient light sensor 122 to determine the ambient light level. The ambient light sensor 122 can be configured to sense the ambient light level. The one or more processors 110 can determine whether the ambient light level is below a threshold. In some examples, the threshold can be predetermined. In other examples, the threshold can be dynamic and based on factors other than the ambient light level, such as whether the flash is in automatic mode or turned on, sensor signals, etc. For example, the one or more processors 110 can compare the determined ambient light level with the threshold 107. Based on the ambient light level being below the predetermined threshold, the one or more processors 110 can control the display to display the first UI 105 in the first mode. When the one or more processors 110 control the display to display the first UI 105, the pixels in the area 120 above the camera sensor (e.g., camera sensor 112) under the display are set to black (e.g., the pixel value is set to 0). In this way, image capture device 102 may improve transmittance in region 120 when ambient light is low.
[0089] In some examples, one or more processors 110 may determine a second ambient light level. One or more processors 110 may determine whether the second ambient light level is below a threshold. Based on the second ambient light level not being below the threshold, one or more processors 110 may control the display to display the second UI 108 in the second mode. When one or more processors 110 control the display to display the second UI 108, the pixels in the area 120 above the camera sensor (e.g., camera sensor 112) under the display may be non-black (e.g., the pixel value is not set to 0). In this way, when the ambient light is not low, area 120 may display content (e.g., an image, a portion of an image, an icon, a portion of an icon, or other content), thereby providing a more pleasing appearance than the first UI 105.
[0090] In some examples, the user can select a mode of the UI. For example, when the image capture application 104 is launched, an icon can be displayed that can switch between different modes of the UI, or a separate icon for each mode can be displayed. When the user clicks the icon, the touch sensor 109 can send a signal to the one or more processors 110, and based at least in part on the signal, the one or more processors 110 can determine the mode of the UI.
[0091] In some examples, one or more processors may perform scene analysis on the image being captured by the camera sensor 112, and may select a UI mode based on the scene analysis. For example, if the face of the subject of the image is well lit, but the background is dark, the one or more processors may select the second mode instead of the first mode. In some examples, one or more processors 110 may determine that the UI mode may be based at least in part on whether the flash is in automatic mode, set to on, or set to off. For example, if the flash is off, the one or more processors 110 may determine that the UI mode is the first mode. If the flash is on or set to automatic, the one or more processors 110 may determine that the UI mode is the second mode. In some examples, the one or more processors may further determine the UI mode based on other sensor signals (e.g., camera sensor signals, depth sensor signals, etc.).
[0092] In some examples, there may be a third UI 111. The third UI 111 may have a greater number of black pixels on the camera sensor 112 than the second UI 108, but a smaller number of black pixels on the camera sensor 112 than the first UI 105. In some examples, the one or more processors 110 may determine that the mode of the UI is the third UI mode based on the ambient light level being above a first threshold but below a second threshold (e.g., on a cloudy day). In some examples, the one or more processors 110 may determine that the UI is in the third mode (third UI 111) based at least in part on other sensor signals (e.g., touch sensor 109, camera sensor 112, etc.).
[0093] In some examples, for the first UI 105, the device 2 may optionally fade in and / or fade out black pixels in the area of the display (e.g., area 232B) above the camera sensor (e.g., camera sensor 234) to provide a more aesthetically pleasing visual effect. For example, when displaying the first UI, the image capture device 102 may fade in black pixels in the area above the camera under the display. In other words, the image capture device 102 may convert the pixel values of the pixels in the area 120 above the camera sensor 112 from an existing non-zero value to a zero value over a period of time. For example, the image capture device may convert from an existing non-zero pixel value to a zero pixel value by reducing the value from an existing non-zero pixel value to a zero pixel value over a period of time. In some examples, for example, when switching from displaying the first camera user interface to displaying something else, the image capture device 102 may fade out black pixels in the area above the camera sensor under the display. In other words, the image capture device 102 may convert the pixel values from an existing zero pixel value to a non-zero pixel value over a period of time. For example, the image capture device may transition from an existing zero pixel value to a non-zero pixel value by increasing the value from the existing zero pixel value to the non-zero pixel value over a period of time. For example, image capture device 102 may fade out black pixels based on an image capture application being closed. In some examples, image capture device 102 may fade out black pixels based on a new ambient light level sensed by ambient light sensor 122 being not less than a predetermined threshold.
[0094] Figures 8A-8C is a conceptual diagram illustrating different example UI modes in an area of a display above an under-display camera sensor, in accordance with techniques of this disclosure. Fig. 8A An example of a first mode (first UI 105) is depicted. Image capture device 502 displays a scene captured by image capture device 502. Image capture device 502 also displays shutter control 506. A camera sensor is shown in a white dotted line below the display of image capture device 502. Area 504 displays a first number of black pixels above the camera sensor. In some examples, all pixels in area 504 are black in the first mode. In this way, image capture device 502 can improve the transmittance of area 504 above the camera sensor. For example, image capture device 502 can display black pixels above the camera sensor in low ambient light conditions. Although area 504 is shown as a square, area 504 can be any shape.
[0095] Figure 8BAn example of a second mode is depicted. For example, image capture device 512 displays a scene captured by image capture device 512. Image capture device 512 also displays shutter control 516. A camera sensor is shown in a dotted line below the display of image capture device 512. Area 514 can display a second number of black pixels above the camera sensor. In some examples, the second number is smaller than the first number used in the first mode. In some examples, the second number is zero. In this way, image capture device 502 can display image content in area 514 above the camera sensor. For example, image capture device 502 may not display black pixels above the camera sensor in high ambient light situations. Although area 514 is shown as a square, area 514 can be any shape.
[0096] Figure 8C An example of a third mode is depicted. For example, image capture device 522 displays a scene captured by image capture device 522. Image capture device 522 also displays shutter control 526. A camera sensor is shown in a dotted line below the display of image capture device 522. Area 524 displays a third number of black pixels above the camera sensor. The number of black pixels can be greater than the second number used in the second mode and less than the third number used in the third mode. In this way, image capture device 522 can display some image content in area 524 above the camera sensor. This third mode can be used in situations where ambient light is less than a first predetermined threshold but less than a second predetermined threshold. For example, image capture device 502 can display some black pixels above the camera sensor in situations where ambient light is moderate.
[0097] Figures 9A-9C is a conceptual diagram showing a further example of the first mode of the technology according to the present disclosure. Fig. 9A , image capture device 532 is depicted as having an oval region 534 showing black pixels above the camera sensor (shown in white dashed lines). Fig. 9B , the image capture device 542 is depicted as displaying a circular area above the camera sensor (shown by a white dashed line). Fig. 9C , image capture device 552 is depicted as displaying a generally rectangular area above the camera sensor (shown in white dashed lines). These are just a few of the shapes of the area above the camera sensor that can display black pixels above the camera sensor when the image capture device is in the first mode. As discussed above, the area displaying black pixels in the first mode can be any shape.
[0098] Fig.10ISP pipeline 50 may be included in image signal processor 6 and may be described above with respect to Figure 1 The ISP pipeline 50 may be included in Figure 2D 230. Although the operations of the ISP pipeline 50 are shown in a specific order, the specific order is exemplary, and the order of operations may be changed according to the technology of the present disclosure.
[0099] The FPN compensator (FPN comp) 126 can receive data from the camera sensor 234. In some examples, at least a portion of the sub-pixels are disposed above the camera sensor 234. In other examples, the sub-pixels of the display are disposed above and adjacent to the camera sensor 234 without the sub-pixels of the display obstructing the camera sensor. For example, a small camera sensor can be disposed below the display between the sub-pixels. The FPN compensator 126 can compensate for fixed pattern noise, such as dark or black pattern noise. The output of the FPN compensator 126 can be provided to an autofocus statistics unit 138, which can determine statistics about the autofocus operation, which statistics can be used, for example, to improve the autofocus function for subsequently captured images. The output of the FPN compensator 126 can also be provided to a display shading compensator 128.
[0100] Display capturer 140 can capture, for example, content displayed on display 232. In some examples, display capturer 140 can capture at least a portion of the displayed content. In some examples, display capturer 140 can capture a portion of the displayed content displayed in area 232B of display 232. In some examples, display capturer 140 can capture a portion of the content displayed in an area located at least partially above camera sensor 234. In some examples, display capturer 140 can capture a portion of the content displayed in an area directly above camera sensor 234. In other examples, display capturer 140 can capture all content displayed on display 232. Display capturer 140 can be implemented as a cache buffer that captures display content at runtime. In some examples, display capturer 140 can be located outside ISP pipeline 50. In some examples, display capturer 140 can be implemented in local memory 8, in external memory 14, in display processing pipeline, or elsewhere within image capture device 2.
[0101] The ISP pipeline 50 may access a stored set of preset parameters 144. In some examples, the preset parameters 144 may be stored in the local memory 8 of the image processing device 4. In other examples, the preset parameters 144 may be stored elsewhere, for example, in the external memory 14. The preset parameters may represent a set of preset parameters used to compensate for, for example, the display sub-pixel layout (e.g., the pixel layout of the display). Figures 6A-6C In some examples, the preset parameters may include a plurality of stored adjustment matrices. Since information about the size, shape, and position of sub-pixels in a given manufacturer's display is known to and readily available from the manufacturer, this information may be used to determine initial values for the preset parameters. In some examples, the preset parameters may also include partition control information, such as which specific pixels may be active in an area (e.g., area 232B) when the camera sensor 234 is actively capturing (multiple) images, such as described above with respect to Figure 5 In some examples, the preset parameters can then be fine-tuned in a lab or factory using sample camera sensors and sample displays with different display contents before being input to the image capture device. In other examples, the preset parameters can be updated wirelessly to the image capture device based on the fine-tuning in the lab.
[0102] Figures 11A-11C It is a conceptual diagram showing determination of preset parameters. Fig.11A represents the layout of sub-pixels in a display, e.g. Figure 6C The preset parameters can be determined by placing a grayscale color check card in the camera tuning box and capturing an image of the grayscale color check card using a camera sensor below the display. In some examples, the image can be captured without any image being displayed on the display. In other examples, the image can be repeatedly captured while images of different grayscale levels are displayed on the display. Since the camera sensor is located below the display, Fig.11A Sub-pixels may block portions of the image being captured. When capturing an image of a grayscale color checker card, it may look more like Fig. 11B The preset parameters may be selected so that after applying the adjustment matrix including the preset parameters, the output image of the display shading compensator 128 may be similar to a pure gray image, for example, Fig. 11C 402 is shown in FIG.
[0103] For example, an engineer or technician can find the maximum pixel value in a captured image. They can then divide the maximum value by each pixel to determine the gain to be applied to each pixel. An adjustment matrix (e.g., a two-dimensional gain matrix) can be generated so that after the gain is applied, the output of each pixel is equal to the output of every other pixel in order to generate Fig. 11C A pure gray image is obtained as an output of the application of the adjustment matrix. The preset parameters can be fine-tuned, for example, by capturing an image with a camera sensor while displaying different content with different colors and brightness levels on a display. The preset parameters can then be adjusted so that the image captured after applying the adjustment matrix closely resembles the image itself. In some examples, the preset parameters can include at least one adjustment matrix.
[0104] In some examples, an engineer or technician can use the display content of a 17-level grayscale test pattern (or other grayscale test pattern) and different brightness levels to determine an adjustment matrix for preset parameters. The size of the adjustment matrix can be m times n, which can be determined based on capturing an 18% grayscale level test pattern (or other grayscale level test pattern). Fig.11A and 11B As shown in , the sub-pixels can have a repeating shape. The m by n rectangular shape can be determined in lab testing. For example, if the repeating shape can all be captured in a 2 pixel by 2 pixel block, the adjustment matrix can be 2 pixels by 2 pixels. In some examples, the m by n shape can match the shape and size of the camera sensor. In other examples, the m by n shape may not match the shape and size of the camera sensor.
[0105] In the m by n region, the display shading compensator 128 may apply different gains to each sub-pixel pattern (eg, R sub-pixels, G sub-pixels, and B sub-pixels). In other words, in some examples, each color channel has its own set of adjustment matrices.
[0106] After applying the gain, all sub-pixels may be equal, e.g., Fig. 11C . For each display gray level and for each color channel, the adjustment matrix of size m by n can be stored in, for example, preset parameters 144. For example, 17 or 33 or other numbers of different gray levels (0-N) can be stored. This process can be repeated for different brightness levels and used to determine different adjustment matrices, for example, a two-dimensional gain matrix. These different adjustment matrices can be stored in preset parameters 144, as shown in Table 1 below. Although specific brightness levels are shown in Table 1, any brightness level can be used according to the technology of the present disclosure.
[0107] Brightness 0% Brightness 10% Brightness 15% Brightness 20% Brightness 50% Brightness 100% Grayscale 0 2D Matrix A 2D matrix D 2D matrix G 2D matrix J 2D matrix M Grayscale 1 2D matrix B 2D matrix E 2D matrix H 2D matrix K 2D matrix N Grayscale N 2D matrix C 2D matrix F 2D Matrix I 2D matrix L 2D matrix O
[0108] Table 1
[0109] In some examples, the display shading compensator 128 may repeat the adjustment matrix of m times n to adjust the entire camera sensor image. For example, the entire camera sensor image may be divided into p times q groups. For example, if the camera sensor is 5000 microns by 5000 microns, and the adjustment matrix is 500 microns by 500 microns, there are 10 times 10 groups in the entire camera sensor image. In some examples, the display shading compensator 128 may apply the same adjustment matrix to 10 times 10 groups. In other examples, the adjustment matrix may be applied only to the area of the display directly above the camera sensor 234. In a further example, the adjustment matrix may be applied to area 232B. In some examples, different adjustment matrices may be applied to different areas. For example, different adjustment matrices may be applied to areas 232A, 232B, and 232C, or to areas of the display above (multiple) image sensors and not above the image sensor. In some examples, as shown in Table 2, matrix 1 may be applied to an area not above the camera sensor, matrix 2 may be applied to an area above the camera sensor 234, and matrix 3 may be applied to an area above the camera sensor 238.
[0110] Matrix 1 Matrix 1 Matrix 1 Matrix 1 Matrix 2 Matrix 1 Matrix 1 Matrix 1 Matrix 3
[0111] Table 2
[0112] In some examples, if the difference between different adjustment matrices is less than a predetermined threshold, all adjustment matrices may not be stored. In some examples, the difference may be the cumulative difference of all elements of the adjustment matrix. In some examples, the difference may be the average difference of all elements of the adjustment matrix. In some examples, the predetermined threshold may be of the order of 0.0001. For example, if the difference between two adjustment matrices is less than a predetermined threshold, one adjustment matrix may be stored and used in two cases, as opposed to storing two adjustment matrices. For example, if the difference between 2D matrix D and 2D matrix A in Table 1 is less than a predetermined threshold, one of the two matrices may be stored and used for a brightness of 10% or 15%. In some examples, if a particular adjustment matrix is greater than three times the threshold of the difference with another adjustment matrix, the adjustment matrix may be directly stored. For example, if the difference between 2D matrix D and 2D matrix A exceeds three times the threshold, 2D matrix D may be stored in preset parameters 144. In some examples, only adjustment matrices with relatively large differences (e.g., 3 times the threshold) with other adjustment matrices may be saved in preset parameters 144.
[0113] return Fig.10, the display shading compensation generator 142 can calculate the regional display average of the sub-pixel values (e.g., RGB) in the area. In some examples, the area is the area of the display directly above the camera sensor 234. In other examples, the area is area 232B. In another example, the area is the entire display. For example, the display shading compensation generator 142 can add the values of the sub-pixels in the area (e.g., each sub-pixel between 0 (black) and 255 (white)) together and divide by the number of pixels multiplied by the number of sub-pixels per pixel. The average can provide an indication of the brightness of the image being displayed. The display shading compensation generator 142 can use the average as an index to select or adjust a given adjustment matrix based on preset parameters. For example, if the average of the sub-pixel values is low, the display shading compensation generator 142 can select one of the 2D matrix A, 2D matrix B, or 2D matrix C. If the average of the sub-pixel values is high, the display shading compensation generator 142 can select one of the 2D matrix M, 2D matrix N, or 2D matrix O. If the average of the sub-pixel values is medium, the display shading compensation generator 142 may select one of the 2D matrix J, 2D matrix K, or 2D matrix L. In some examples, the display shading compensation generator 142 may select one of the matrices indicated by the index (e.g., 2D matrix A, 2D matrix B, or 2D matrix C if the average of the sub-pixel values is 10% brightness) by selecting a gray level that is closest to the average gray or brightness of the sub-pixels (e.g., 10%). In other examples, the display shading compensation generator may use other methods to select an adjustment matrix from the group indicated by the index.
[0114] In the example of Table 1, if the average of the sub-pixel values does not match any of the table entries, the display shading compensation generator 142 can utilize one or more of the table entries and adjust the matrix up or down accordingly. For example, if the average of the sub-pixel values indicates that the brightness is on the order of 40%, the display shading compensation generator 142 can select two adjustment matrices (e.g., 2D matrix K and 2D matrix K) with gray levels of the same index and interpolate between them. In other examples, the display shading compensation generator can select an adjustment matrix and adjust the matrix accordingly, for example, increasing or decreasing the gain.
[0115] For four color channels (e.g., RGBG), an example 3x3 adjustment matrix might be [(1.01358, 1.02586, 1.13256, 1.7465), (1.01358, 1.02486, 1.13256, 1.7265), (1.01358, 1.02586, 1.13256, 1.7463),
[0116] (1.01358,1.02586,1.13256,1.2465),(1.01358,1.02586,1.13256,1.7449),(1.01358,1.02586,1.13256,1.7464),
[0117] (1.01358, 1.02586, 1.13256, 1.2465), (1.01358, 1.02586, 1.13256, 1.2468), (1.01358, 1.02586, 1.13256, 1.7462),]. This example matrix may be applied to image data by the display shading compensator 128 to compensate for display shading.
[0118] In some examples, the display shading compensation generator 142 may select an adjustment matrix (e.g., a two-dimensional gain matrix) from the adjustment matrix that may be stored in the preset parameters 144 from the memory (e.g., the local memory 8 or the external memory 14) to be applied to the output of the FPN compensator 126 based on at least a portion of the content captured in the display capturer 140 (e.g., at least a portion of the displayed content) and the preset parameters 144. In other examples, the display shading compensation generator 142 may generate an adjustment matrix, e.g., a two-dimensional gain matrix, based on at least a portion of the content captured in the display capturer 140 and the preset parameters 144. For example, the display shading compensation generator 142 may determine the adjustment matrix by calculating the average value of R, G, and B (or R, G, B, and W) for the displayed content in the display capturer 140. The display shading compensation generator 142 may then use the average value as an index to determine which adjustment matrix based on the preset parameters to use or which adjustments to make to the adjustment matrix. In some examples, the display shading compensation generator 142 determines the adjustment matrix for each color component (e.g., R, G, and B). The display shading compensation generator 142 may then provide the two-dimensional gain matrix to the display shading compensator (Display Shading Compensation) 128 .
[0119] As mentioned above, sub-pixels in a display age over time. Such sub-pixel aging over time may include aging of the cathode and anode materials of the sub-pixel caused by the drive current of the display content pixel. Sub-pixel aging may reduce the transmittance of the sub-pixel over time. For under-display cameras, display shading may have a strong aging component. In other words, display shading may get worse over time due to aging of the sub-pixels in the display. Display shading may become more prevalent over time and with use as the display becomes less transparent. Such sub-pixel aging may affect the entire ISP pipeline 50, including automatic exposure control, auto focus, and auto white balance.
[0120] In some examples, according to the technology of the present disclosure, the transmittance aging model can be used with the display content and / or predetermined parameters to generate or select an adjustment matrix. For example, the adjustment matrix can be further based on the aging factor 146. For example, the aging factor 146 can be used by the display shading compensation generator 142 to generate the adjustment matrix. In some examples, the display shading compensation generator 142 can multiply the selected or generated adjustment matrix by the aging factor 146. The ISP 50 can determine the aging factor based on statistics related to the display pixel values and time in the area of the display 232 (e.g., the area above the camera sensor 234 or the camera sensor 238, for example, the area 232B or the area 232C, respectively). For example, the ISP 50 can monitor the amount of time that each sub-pixel in the area is turned on and the display pixel value of each sub-pixel in the area during the time when the sub-pixel is turned on, and use the monitored time and display pixel value to determine the aging factor. In some examples, the aging factor can be determined based on measurements of aging experiments in a laboratory environment. Such an experiment may measure display pixel values and time relative to an area (e.g., area 232B or area 232C) of display 232. In some examples, ISP 50 may generate a coefficient curve for the aging factor, or the coefficient curve for the aging factor may be generated by another device (e.g., a computing device in a laboratory). In some examples, the aging factor may be determined by examining statistics in a color space or spectrum.
[0121] In some examples, the ISP pipeline 50 can determine the aging factor as a statistical measure of the transmittance fading comparison of pixels in a region of the display (e.g., region 232B or region 232C), such as an average of the transmittance fading ratios. As used herein, a ratio is a type of comparison. For example, for (each pixel in the region, which is defined by the shape mask)
[0122] {
[0123] Pixel value difference = n(R, G, B)-n-1(R, G, B);
[0124] If (pixel difference > threshold) {
[0125] Start recording new pixel value time;
[0126] The time at which the previous pixel value recording is completed;
[0127] Accumulating previous pixel transmittance decay ratios;
[0128] }
[0129] Regional average transmittance decay ratio = average (m × n) pixel transmittance decay ratio;
[0130] }
[0131] In some examples, rather than modifying the 2D matrix by multiplying the 2D matrix by the aging factor, the aging factor may be used to modify other algorithms in the ISP pipeline 50 or the image signal processor 6. For example, the aging factor may be used to modify the AEC process 20, the AF process 24, and / or the AWB process 22 ( Figure 1 In some examples, an aging factor can be used together with color adaptation.
[0132] Then, the display shading compensator 128 can apply the two-dimensional gain matrix to the output of the FPN compensator 126. In some examples, the display shading compensator 128 can apply the two-dimensional gain matrix to the output of the FPN compensator 126 in a piecewise bilinear manner. The display shading compensator 128 can have multiple programmable parts and can apply a gain in each dimension (e.g., the x-direction and the y-direction). The display shading compensator 128 can perform compensation for the output of the FPN compensator 126 separately for each color channel. For example, the display shading compensator 128 can apply a different gain for red than for blue and for green (e.g., through separate two-dimensional gain matrices).
[0133] The output of the display shading compensator 128 can be provided to a lens shading compensator 130, which can compensate for the shading caused by the camera lens. In some examples, the display shading compensator 128 can be combined with the lens shading compensator (LENS SHADING COMP) 130. The lens shading compensator 130 can provide the output of the lens shading compensation to the automatic white balance unit (WHITE BALANCE) 132, which can perform automatic white balance processing on the signal of the lens shading compensation. The automatic white balance unit 132 can provide an automatic white balance signal to a bad pixel corrector (BAD PIXELCORR) 134. The bad pixel corrector 134 can correct the bad pixels and output an image processed signal. For example, the bad pixel corrector 134 can determine that a pixel in the image is an outlier and attempt to replace the pixel value with a value that is more similar to the values of neighboring pixels. The auto-exposure and auto-white balance statistics unit (AE, AWB STATS) 136 may receive the image signal processed signal and use the information therein to adjust the auto-exposure and / or auto-white balance of an image capture device (e.g., image capture device 230) for subsequently captured images. The image signal processed signal may be provided to a display processor for display, for example, on display 232, and / or stored in a memory (e.g., local memory 8 or external memory 14) as a video file or snapshot.
[0134] In some examples, content may be displayed on display 232 while camera sensor 234 may be capturing image(s). In some examples, content displayed on display 232 may be displayed at a frame rate that is different from a shutter speed of camera sensor 234. As discussed above, display shading compensation generator 142 may select or generate an adjustment matrix (e.g., a two-dimensional gain matrix) to apply to the output of FPN compensator 126. To synchronize the application of the appropriate adjustment matrix with the image(s) captured by camera sensor 234, further techniques of the present disclosure may be applied.
[0135] In some examples, display capturer 140 may include a ring buffer (RB) 148 to provide appropriate temporal display content pixels (which may be at least a portion of one or more frames) to the display shading compensation generator at appropriate times. Ring buffer 148 may be implemented in local memory 8, in external memory 14, in a display processor, or elsewhere in image capture device 2.
[0136] Fig.12 is a conceptual diagram showing a technique for synchronous display shading compensation. Fig.12 In the example of FIG. 1 , each display frame content dumped into the circular buffer can be marked with a timestamp. For example, the display capturer 140 can capture a portion of one or more frames of display content in any number of display frame current write buffers (e.g., display frame current write buffer 0 288A, display frame current write buffer 1 288B, display frame current write buffer 2 288C to display frame current write buffer N (where N can be any positive integer) 288N, which can be implemented as a circular buffer 148 and can be referred to as a display frame current write buffer 288).
[0137] A data processing unit hardware interrupt service request (DPU HW ISR) 284 using a global clock source 282 (e.g., a system on chip (SOC) global clock source) for device 2 can apply a timestamp to each of the portions of one or more frames of display content. As mentioned above, as used herein, a portion of a frame refers to a portion of a frame or an entire frame. The timestamp can indicate the time when the frame is displayed on the display 232. During camera sensor exposure (e.g., when capturing an image), the start time and end time of the exposure can use the global clock source 282. For example, the global clock source 282 can provide the time to the DPU HW ISR 284 and the camera image signal processing hardware interrupt service request (ISP HW ISR) 286. The DPU HW ISR 284 can apply a timestamp to each of the portions of one or more frames of display content stored in the display frame current write buffer 288. The ISP HW ISR 286 can determine the exposure time 296 of the camera sensor (e.g., the start time and the end time of the exposure).
[0138] The display shading compensation generator 142 can retrieve appropriate portions of one or more frames of display content from any of the display frame current write buffers 288 based on the applied timestamp and the exposure time 296 of the camera sensor. For example, the display shading compensation generator 142 can retrieve at least a portion of frame-n, or at least a portion of frame-p to frame-q, as needed, based on its timestamp and camera sensor exposure time 296 from the display frame current write buffer 288. The corresponding display content can be used to select or generate an appropriate adjustment matrix. In the example where the display shading compensation generator 142 generates an adjustment matrix, at least a portion of frame-n can be used to generate matrix-n, and (a portion of frame-p, a portion of frame-q) can be used to generate (matrix-p, matrix-q).
[0139] The display shading compensation generator 142 may generate the adjustment matrices to be applied by the display shading compensator 128 based on these previously mentioned matrices. For example, the display shading compensation generator 142 may generate the adjustment matrices to be applied by the display shading compensator 128 by calculating the temporal sum of (matrix-p, matrix-q). If the camera exposure time falls entirely within one frame period (e.g., frame-n), the display shading compensation generator 142 may generate the adjustment matrices to be applied by the display shading compensator 128 by simply using the matrices themselves (e.g., matrix-n).
[0140] For example, in the case of a 120 Hz display, the display frame time is 8.33 ms. If the exposure time of the camera sensor is 1 / 30 second (33.33 ms), a total of eight (or some other number greater than four to allow for engineering margin and avoid buffer overflow) display frame current write buffers may be used. The display capturer 140 may use four of the display frame current write buffers 288 to capture four frames (frames 0, 1, 2, and 3) or portions thereof. The display shading compensation generator 142 may use these four frames or portions thereof to generate four matrices (matrices 0, 1, 2, and 3). During each of frames 0, 1, 2, and 3, the camera sensor may be exposed for the entire duration of each of the frames. In this example, the display shading compensation generator 142 may calculate the adjustment matrix to be applied by the display shading compensator as follows: adjustment matrix = (8.33 / 33.33 * matrix 0) + (8.33 / 33.33 * matrix 1) + (8.33 / 33.33 * matrix 2) + (8.33 / 33.33 * matrix 3). In some examples, the camera sensor may be exposed for only a portion of one or more frames. Taking the above example as an example, and changing the example so that the camera sensor is only exposed for 7.5 microseconds in frame 0, but is exposed during the entire frame period of frames 1, 2, and 3, the display shading compensation generator 142 may calculate the adjustment matrix to be applied by the display shading compensator as follows: adjustment matrix = (7.5 / 33.33 * matrix 0) + (8.33 / 33.33 * matrix 1) + (8.33 / 33.33 * matrix 2) + (8.33 / 33.33 * matrix 3).
[0141] In another example, in the case of a 60 Hz display, the display frame time is 16.66 ms. If the exposure time of the camera sensor is 1 / 125 second (4 ms), a total of two (or some other number greater than one to allow for engineering margin and avoid buffer overflow) display frame current write buffers 288 may be used. The display capturer 140 may use one of the display frame current write buffers 288 to capture a frame (e.g., frame 1) or a portion of which the camera exposure occurs. In this example, the exposure of the camera sensor occurs entirely in frame 1, so the display shading compensation generator 142 may generate matrix 1 based on frame 1, and then provide matrix 1 to the display shading compensator 128 as an adjustment matrix applied to the output of the FPN compensator 126.
[0142] Fig.13is a flow chart illustrating an example display shading compensation technique of the present disclosure. The ISP pipeline 50 may receive first image data (300) captured by the camera sensor 234. For example, the first image data may represent the face of a user and the background around the face of the user when the user is taking a "selfie". The ISP pipeline 50 may receive at least a portion of the displayed content (302). For example, the ISP pipeline 50 may receive a portion of the displayed content located in the area 232B of the display 232. In other examples, the ISP pipeline 50 may receive a portion of the displayed content directly above the camera sensor 234. In other examples, the ISP pipeline 50 may receive all of the displayed content.
[0143] In some examples, the ISP pipeline 50 (e.g., the FPN compensator 126) can compensate the first image data for FPN (304). Any known technique can be used to compensate for FPN. The ISP pipeline 50 (e.g., the display shading compensation generator 142) can determine an adjustment matrix (306), such as a two-dimensional gain matrix. The adjustment matrix can be based on a portion of the displayed content in the display capturer 140. In some examples, the adjustment matrix is also based on the preset parameters 144. In some examples, the adjustment matrix is also based on an aging factor. In some examples, determining the adjustment matrix may include multiplying a two-dimensional compensation matrix based on at least a portion of the displayed content and the preset parameters by an aging factor. The aging factor can be based on an aging state of pixels above the camera sensor. For example, the aging factor can be based on a statistical measurement of a transmittance fading comparison of pixels in an area (e.g., area 232B) above a camera sensor (e.g., camera sensor 234) under the display (e.g., an average of transmittance fading ratios).
[0144] As discussed above, the preset parameters may be compensation parameters to compensate for the effects of the sub-pixel size, shape, and position (e.g., Figures 6A-6CIn some examples, the ISP pipeline 50 (e.g., the display shading compensation generator 142) may determine an adjustment matrix, such as a two-dimensional gain matrix, by selecting an adjustment matrix from the adjustment matrices stored in the preset parameters 144. In some examples, the preset parameters 144 may be in the local memory 8 or the external memory 14. In other examples, the preset parameters 144 may be in the ISP pipeline 50. In the case where the ISP pipeline 50 selects an adjustment matrix, the ISP pipeline 50 may try to select an adjustment matrix that may best compensate for the display shading. In some examples, the ISP pipeline 50 determines the average sum of the sub-pixels in the area and uses the average as an index to determine which adjustment matrix to select, as discussed above. In some examples, the ISP pipeline 50 may interpolate between two adjustment matrices, or otherwise adjust a given selected adjustment matrix. In other examples, the ISP pipeline 50 may calculate an adjustment matrix, such as a two-dimensional gain matrix, based on at least a portion of the displayed content in the display capturer 140. In some examples, ISP pipeline 50 may calculate an adjustment matrix based on at least a portion of the displayed content in display capturer 140 and preset parameters 144 .
[0145] The ISP pipeline 50 (e.g., the display shading compensator 128) may apply the adjustment matrix to the first image data (whether or not after FPN compensation) to create second image data (308). For example, display shading compensation may be performed on the second image data to reduce or eliminate the effects of shading caused by sub-pixels in the display 232 and light scattering caused by the displayed content.
[0146] In some examples, the ISP pipeline 50 (e.g., the lens shading compensator 130) may compensate the second image data for lens shading (310). In other examples, the ISP pipeline 50 (e.g., the automatic white balance unit 132) may automatically white balance the second image data (310). The ISP pipeline 50 may output the second image data (312). For example, the ISP pipeline 50 may output the second image data to a memory (e.g., an external memory 14) for permanent storage (or storage until deleted by a user). In another example, the ISP pipeline 50 may output the second image data to the display 232, for example to provide a preview image to a user. In yet another example, the ISP pipeline 50 may output the second image data to a zero shutter lag (ZSL) buffer. In another example, the ISP pipeline 50 may output the second image data to an external display. Although Fig.13 The steps are depicted to occur in a particular order, but this order is exemplary only. The depicted order should not be considered limiting.
[0147] Fig.14 is a flow chart illustrating an example display shading compensation technique of the present disclosure. Fig.14 The technology can be used with Fig.13 In some examples, the image capture device may have more than one camera sensor below the display. Fig.14 For example, Fig.13 The camera sensor is the first camera sensor, Fig.13 The at least one portion of the display is at least a first portion of the display, Fig.13 The at least a portion of the displayed content is at least a first portion of the displayed content, and Fig.13 The adjustment matrix is the first adjustment matrix.
[0148] exist Fig.14 In the example of , the image capture device ISP pipeline 50 may receive third image data (314) captured by the second camera sensor 238. For example, the third image data may be different from the first image data because the second camera sensor 238 may be a sensor with a wider or narrower angle than the first camera sensor. The ISP pipeline 50 may receive at least a second portion of the displayed content (315). For example, the ISP pipeline 50 may receive a portion of the displayed content located in the area 232C of the display 232. In other examples, the ISP pipeline 50 may receive a portion of the displayed content directly above the second camera sensor 238. In yet another example, the ISP pipeline 50 may receive all of the displayed content. In some examples, the first area and the second area may be the same. In other examples, the first area and the second area may not be the same. In some examples, the first area and the second area may include at least some of the same pixels.
[0149] The ISP pipeline 50 (e.g., the display shading compensation generator 142) can determine a second adjustment matrix (316), such as a two-dimensional gain matrix. The adjustment matrix can be based on a portion of the displayed content in the display capturer 140. In some examples, the adjustment matrix is also based on the preset parameters 144. In other examples, the adjustment matrix is also based on the aging state of the pixels above the camera sensor. As discussed above, the preset parameters can be compensation parameters to compensate for the aging state of the sub-pixel size, shape, and position (e.g., Figures 6A-6C In Figure 2DIn the example of , camera sensor 234 and second camera sensor 238 are located at different locations below display 232. Therefore, different displayed content may be displayed above camera sensor 234 than above second camera sensor 238. In addition, the sub-pixels (or portions thereof) located above camera sensor 234 may be different from the sub-pixels (or portions thereof) located above camera sensor 238. The aging states of the pixels in region 232B and region 232C may also be different. Therefore, the second adjustment matrix may be different from Fig.13 The adjustment matrix for the example is different.
[0150] In some examples, the ISP pipeline 50 may determine a second adjustment matrix, such as a two-dimensional gain matrix, by selecting an adjustment matrix from the adjustment matrices that may be stored in the preset parameters 144. In some examples, the preset parameters may be in the local memory 8, in the external memory 14, or in the ISP pipeline 50. In such an example, the ISP pipeline 50 may attempt to select an adjustment matrix that may best compensate for display shading. In some examples, the ISP pipeline 50 determines the average sum of sub-pixels in the second area and uses the average as an index to determine which adjustment matrix to select. In some examples, the ISP pipeline 50 may interpolate between two adjustment matrices or otherwise adjust a given selected adjustment matrix. In other examples, the ISP pipeline 50 may calculate a second adjustment matrix, such as a two-dimensional gain matrix, based on at least a second portion of the displayed content in the display capturer 140. In some examples, the ISP pipeline 50 may calculate a second adjustment matrix based on at least a portion of the displayed content in the display capturer 140 and the preset parameters 144.
[0151] The ISP pipeline 50 (e.g., the display shading compensator 128) may apply the second adjustment matrix to the third image data to create fourth image data (317). For example, the fourth image data may be display shading compensated to reduce or eliminate the effects of shading caused by sub-pixels in the display 232, light scattering caused by display content, and aging of pixels above the camera sensor.
[0152] The ISP pipeline 50 may output the fourth image data (318). For example, the ISP pipeline 50 may output the fourth image data to a memory (e.g., external memory 14) for permanent storage (or storage until deleted by a user). In another example, the ISP pipeline 50 may output the fourth image data to the display 232 to, for example, provide a preview image to a user. In yet another example, the ISP pipeline 50 may output the second image data to a zero shutter lag (ZSL) buffer. In another example, the ISP pipeline 50 may output the second image data to an external display. Although Fig.14 The steps are depicted to occur in a particular order, but this order is exemplary only. The depicted order should not be considered limiting.
[0153] Fig.15 is a flow chart illustrating a synchronization technique according to the present disclosure. Fig.15 The technology can be used with Fig.13 and / or Fig.14 For example, the ISP pipeline 50 can be Fig.15 The technology determines the adjustment matrix ( Fig.13 306).
[0154] The ISP pipeline 50 (e.g., the display capturer 140) may store (320) a portion of one or more frames of display content, for example, in the display frame current write buffer 288. The DPU HW ISR 284 may apply a timestamp to each portion of the one or more frames of display content, the timestamp indicating the time at which the frame was displayed (322). For example, the DPU HW ISR 284 may apply a timestamp that indicates the beginning of the frame displayed by the display 232 based on the global clock source 282. In other examples, the DPU HW ISR 284 may apply a timestamp that indicates the end of the frame displayed by the display 232 based on the global clock source 282. In other examples, the DPU HW ISR 284 may apply a timestamp that indicates any other time during the display of the frame by the display 232 based on the global clock source 282.
[0155] The camera ISP HW ISR 286 may determine an exposure time of the camera sensor (324). For example, the camera ISP HW ISR may determine a start time of the camera sensor exposure and an end time of the camera sensor exposure based on the global clock source 282. The ISP pipeline 50 (e.g., the display shading compensation generator 142) may determine which frame(s) are associated with the camera sensor exposure time (326). For example, the display shading compensation generator 142 may determine which frame(s) are associated with the camera sensor exposure time based on the applied timestamp. For example, the display shading compensation generator 142 may compare the camera sensor exposure time with the timestamp. The ISP pipeline 50 (e.g., the display shading compensation generator 142) may determine an adjustment matrix (328) based on a portion of one or more frames that are associated frames. In some examples, the display shading compensation generator 142 may determine the adjustment matrix by selecting one or more best matching adjustment matrices based on a portion of one or more frames that are associated frames. In the event that there is more than one associated frame, display shading compensation generator 142 may select the best matching adjustment matrix for portions of each associated frame, and may perform calculations (e.g., time and) to determine the adjustment matrix to be applied by display shading compensator 128. In some examples, the adjustment matrix may also be based on preset parameters 144.
[0156] In other examples, display shading compensation generator 142 may determine the adjustment matrix by generating one or more matrices based on portions of one or more frames that are associated frames. In the event that there is more than one associated frame, display shading compensation generator 142 may generate matrices for portions of each associated frame and perform calculations (e.g., time and) to determine the adjustment matrix to be applied by display shading compensator 128. In some examples, the adjustment matrix may also be based on preset parameters 144. Although Fig.15 The steps are depicted to occur in a particular order, but this order is exemplary only. The depicted order should not be considered limiting.
[0157] Fig.16 is a flow chart illustrating an example of determining a user interface according to the techniques of this disclosure. Fig.16 The example technique can be used with Figure 13-15The image capture device 102 may determine an ambient light level (330). For example, the one or more processors 110 may query the ambient light sensor 122 to determine the ambient light level, and the ambient light sensor 122 may sense the ambient light level. The image capture device 102 may determine whether the ambient light level is below a predetermined threshold (334). For example, the one or more processors 110 of the image capture device 102 may compare the determined ambient light level to a predetermined threshold (e.g., threshold 107) stored in the memory 114 to determine whether the ambient light level is below the predetermined threshold. Based on the ambient light level being below the predetermined threshold ( Fig.16 ), the image capture device 102 may display a first camera user interface (336). For example, the image capture device 102 may display a first UI 105 on the display 116, the first UI 105 having black pixels in the area 120 above the camera sensor 112.
[0158] In some examples, the ambient light level may not be below a predetermined threshold. In these cases, based on the ambient light level not being below the predetermined threshold ( Fig.16 ), image capture device 102 may display a second camera user interface (338). For example, image capture device 102 may display a second UI 108 on display 116 that has non-black pixels in area 120 over camera sensor 112. For example, image capture device 102 may display content, such as an image, a portion of an image, an icon, a portion of an icon, other content, in area 120 over camera sensor 112.
[0159] In some examples, image capture device 102 may launch an image capture application. In some examples, image capture device 102 determines the ambient light level based on launching the image capture application.
[0160] In some examples, when displaying the first UI 105, the image capture device 102 can fade in the black pixels in the area above the camera under the display. In other words, the image capture device 102 can convert the pixel values of the pixels in the area 120 above the camera sensor 112 from the existing non-zero value to the zero value over a period of time. For example, the image capture device can convert the existing non-zero pixel value to the zero pixel value by reducing the value from the existing non-zero pixel value to the zero pixel value over a period of time. In some examples, for example, when switching from displaying the first camera user interface to displaying something else, the image capture device 102 can fade out the black pixels in the area above the camera sensor under the display. In other words, the image capture device 102 can convert the pixel value from the existing zero pixel value to the non-zero pixel value over a period of time. For example, the image capture device can convert the existing zero pixel value to the non-zero pixel value by increasing the value from the existing zero pixel value to the non-zero pixel value over a period of time. For example, the image capture device 102 can fade out the black pixels based on the image capture application being closed. In some examples, image capture device 102 may fade out black pixels based on the new ambient light level sensed by ambient light sensor 122 not being below a predetermined threshold. Fig.15 The steps are depicted to occur in a particular order, but this order is exemplary only. The depicted order should not be considered limiting.
[0161] Thus, in this manner, by providing a first UI for use in low ambient light situations and by providing a display shading compensator, a camera sensor may be located under the display in order to attempt to maximize the display size on the image capture device without otherwise experiencing image quality issues. By positioning the camera sensor under the screen, the screen size of the image capture device may be larger than an image capture device of the same size using a screen with a notch, and the reliability of the image capture device may be improved over that of an image capture device using a pop-up camera with movable mechanical parts.
[0162] Fig.17 is a flow chart illustrating another example of determining a user interface according to the technology of this disclosure. Fig.17 The example technique can be used with Figure 13-16 The image capture device 102 may receive a signal from a sensor (350). For example, the one or more processors 110 may receive a signal from the ambient light sensor 122. In another example, the one or more processors 110 may receive a signal from the touch sensor 109. In yet another example, the image capture device may receive a signal from the camera sensor 112.
[0163] Image capture device 102 may determine a user interface mode based at least in part on the signal (352). The user interface mode may include a first mode or a second mode. The first mode (e.g., first UI 105) may include a first number of black pixels in an area of the display, and the second mode may include a second number of black pixels in an area of the display. The first number may be greater than the second number. For example, the first mode may include an area (e.g., Fig. 8A For example, the second pattern may include all black pixels in the region (e.g., Figure 8B Zero black pixels in area 514).
[0164] Image capture device 102 may receive image data (354) from camera sensor 112. For example, a user may click an icon on display 116 to cause one or more processors 110 to launch image capture application 104, and camera sensor 112 may capture and send image data to one or more processors 110.
[0165] In some examples, the one or more processors 110 can determine whether the signal is below a threshold, and based on the signal being below the threshold, the one or more processors can control the display 116 to display a first mode (e.g., first UI 105). In some examples, the one or more processors 110 can receive a second signal from the sensor. The one or more processors 110 can determine whether the second signal is below a threshold. Based on the second signal not being below the threshold, the one or more processors 110 can control the display to display a second mode (e.g., second UI 108).
[0166] In some examples, one or more processors 110 may launch image capture application 104. In some examples, one or more processors 110 may determine a user interface mode based on launching the image capture application. In some examples, the UI mode may include a third mode with a third number of black pixels. In some examples, the third number of black pixels is greater than the second number and less than the first number.
[0167] In some examples, one or more processors 110 may control display 116 to fade in black pixels in area 120 over the camera sensor. In some examples, one or more processors 110 may control display 116 to fade out black pixels in area 120 over the camera sensor. In some examples, one or more processors 110 may control display 116 to fade out black pixels in area 120 based on the image capture application being closed. In some examples, one or more processors 110 may control the display to fade out black pixels based on the new ambient light level being not below a threshold.
[0168] The technology of the present disclosure includes the following examples.
[0169] Example 1. An image capture device, comprising: a memory; and one or more processors coupled to a camera sensor and the memory and configured to: receive first image data from the camera sensor, the camera sensor being configured to receive light through at least a portion of a display; receive at least a portion of display content; determine an adjustment matrix based on at least a portion of the display content; apply the adjustment matrix to the first image data to create second image data; and output the second image data.
[0170] Example 2. The apparatus of Example 1, wherein the adjustment matrix comprises a two-dimensional gain matrix.
[0171] Example 3. The apparatus according to Example 1 or Example 2, wherein the adjustment matrix is also based on preset parameters.
[0172] Example 4. The apparatus of Example 3, wherein the preset parameter indicates at least one of the following: a shape, size or position of a sub-pixel in a display, or a position of an active pixel in the display.
[0173] Example 5. The image capture device of any combination of Examples 1-4, wherein the adjustment matrix is further based on an aging factor.
[0174] Example 6. The image capture device of any combination of Examples 1-4, wherein the one or more processors are further configured to multiply the adjustment matrix by an aging factor before applying the adjustment matrix.
[0175] Example 7. The image capture device of Example 5 or Example 6, wherein the aging factor is based on a statistical measurement of a comparison of transmittance decay of pixels in the region above the camera sensor under the display.
[0176] Example 8. The apparatus of any combination of Examples 1-7, wherein at least a portion of the display content is displayed in an area located at least partially over the camera sensor.
[0177] Example 9. The apparatus of any combination of Examples 1-8, wherein the one or more processors are configured to apply the adjustment matrix by applying the adjustment matrix in a piecewise bilinear manner.
[0178] Example 10. The apparatus of any combination of Examples 1-9, wherein at least a portion of the subpixels in the display are disposed over the camera sensor.
[0179] Example 11. The apparatus of any combination of Examples 1-10, wherein subpixels of the display are disposed above and adjacent to the camera sensor without obstructing the camera sensor.
[0180] Example 12. The apparatus of any combination of Examples 1-11, wherein the camera sensor is a first camera sensor, the at least a portion of the display is at least a first portion of the display, the at least a portion of the displayed content is at least a first portion of the displayed content, and the adjustment matrix is a first adjustment matrix, and the one or more processors are further configured to: receive third image data from a second camera sensor, the second camera sensor being disposed below at least a portion of the display and coupled to the one or more processors; receive at least a second portion of the displayed content; determine a second adjustment matrix based on at least the second portion of the displayed content; apply the second adjustment matrix to the third image data to create fourth image data; and output the fourth image data.
[0181] Example 13. The apparatus of any combination of Examples 1-12, wherein the one or more processors are configured to apply the adjustment matrix by applying the adjustment matrix separately for each color channel.
[0182] Example 14. The apparatus of any combination of Examples 1-13, further comprising: the display configured to display content.
[0183] Example 15. The apparatus of any combination of Examples 1-14, wherein the display content is based on an image captured by the camera sensor.
[0184] Example 16. The device of any combination of Examples 1-15, wherein the display comprises an organic light emitting diode (OLED) display.
[0185] Example 17. The device of Example 16, wherein the OLED display includes at least one of a transparent anode and a transparent cathode.
[0186] Example 18. The apparatus of Example 16 or Example 17, wherein the one or more processors are further configured to: display content by at least a subset of pixels forming an area of the OLED display when the first image data is received.
[0187] Example 19. The device of any combination of Examples 1-18, wherein the device comprises a mobile phone.
[0188] Example 20. The apparatus of any combination of Examples 1-19, wherein the one or more processors are configured to determine the adjustment matrix by selecting the adjustment matrix from a plurality of adjustment matrices stored in a memory.
[0189] Example 21. The apparatus of Example 20, wherein the one or more processors are configured to select the adjustment matrix by calculating an average sum of sub-pixel values in a region and using the average sum as an index for selecting the adjustment matrix.
[0190] Example 22. The apparatus of Example 20, wherein the one or more processors are further configured to: adjust the selected adjustment matrix.
[0191] Example 23. The apparatus of any combination of Examples 1-22, wherein the one or more processors are configured to select the adjustment matrix by interpolating between two adjustment matrices.
[0192] Example 24. The apparatus of any combination of Examples 1-19, wherein the one or more processors are configured to determine the adjustment matrix by calculating the adjustment matrix.
[0193] Example 25. The apparatus of any combination of Examples 1-24, wherein the one or more processors are configured to determine the adjustment matrix by: storing a portion of one or more frames of the display content; applying a timestamp to each portion of the one or more frames of the display content; determining a camera sensor exposure time; determining which frames are associated with the camera sensor exposure time based on the applied timestamps; and determining the adjustment matrix based on the portion of the one or more frames being associated frames.
[0194] Example 26. A method of image processing, comprising: receiving, at an image capture device, first image data captured by a camera sensor, wherein the camera sensor is configured to receive light through at least a portion of a display on the image capture device; receiving, at the image capture device, at least a portion of display content; determining, at the image capture device, an adjustment matrix based on at least a portion of the display content; applying the adjustment matrix to the first image data to create second image data; and outputting the second image data.
[0195] Example 27. The method of Example 26, wherein the adjustment matrix comprises a two-dimensional gain matrix.
[0196] Example 28. The method according to Example 26 or Example 27, wherein the adjustment matrix is also based on preset parameters.
[0197] Example 29. The method of Example 28, wherein the preset parameter indicates at least one of the following: a shape, size, or position of a sub-pixel in the display, or a position of an active pixel in the display.
[0198] Example 30. The method of any combination of Examples 26-29, wherein the adjustment matrix is further based on an aging factor.
[0199] Example 31. The method of any combination of Examples 26-29, further comprising: multiplying the adjustment matrix by an aging factor before applying the adjustment matrix.
[0200] Example 32. The method of Example 30 or Example 31, wherein the aging factor is based on a statistical measurement of a comparison of transmittance decay of pixels in a region above the camera sensor.
[0201] Example 33. The method of any combination of Examples 26-32, wherein at least a portion of the display content is displayed in an area located at least partially over the camera sensor.
[0202] Example 34. The method of any combination of Examples 26-33, wherein applying the adjustment matrix comprises applying the adjustment matrix in a piecewise bilinear manner.
[0203] Example 35. The method of any combination of Examples 26-34, wherein at least a portion of the subpixels in the display are disposed over the camera sensor.
[0204] Example 36. The method of any combination of Examples 26-35, wherein the subpixels of the display are disposed above and adjacent to the camera sensor without obstructing the camera sensor.
[0205] Example 37. The method of any combination of Examples 26-36, wherein the camera sensor is a first camera sensor, the at least a portion of the display is at least a first portion of the display, the at least a portion of the displayed content is at least a first portion of the displayed content, and the adjustment matrix is a first adjustment matrix, the method further comprising: receiving, at an image capture device, third image data captured by a second camera sensor, wherein the second camera sensor is disposed below at least a second portion of the display on the image capture device; receiving, at the image capture device, at least a second portion of the displayed content; determining, at the image capture device, a second adjustment matrix based on at least the second portion of the displayed content; applying the second adjustment matrix to the third image data to create fourth image data; and outputting the fourth image data.
[0206] Example 38. The method of any combination of Examples 26-37, wherein applying the adjustment matrix comprises applying an adjustment matrix separately for each color channel.
[0207] Example 39. The method of any combination of Examples 26-38, wherein the displayed content is based on an image captured by the camera sensor.
[0208] Example 40. The method of any combination of Examples 26-39, wherein the display comprises an organic light emitting diode (OLED) display.
[0209] Example 41. The method of Example 40, wherein the OLED display includes at least one of a transparent anode and a transparent cathode.
[0210] Example 42. The method of Example 40 or Example 41 further comprises actively displaying content by at least a subset of pixels forming an area of the OLED display when the first image data is received.
[0211] Example 43. The method of any combination of Examples 26-42, wherein the image capture device comprises a mobile phone.
[0212] Example 44. The method of any combination of Examples 26-43, wherein determining the adjustment matrix comprises selecting the adjustment matrix from a plurality of adjustment matrices stored in a memory.
[0213] Example 45. The method of Example 44, wherein selecting the adjustment matrix comprises calculating an average sum of sub-pixel values in a region, and using the average sum as an index for selecting the adjustment matrix.
[0214] Example 46. The method of Example 44, wherein selecting the adjustment matrix comprises adjusting the adjustment matrix.
[0215] Example 47. The method of any combination of Examples 26-46, wherein determining the adjustment matrix further comprises: interpolating between two adjustment matrices.
[0216] Example 48. The method of any combination of Examples 26-43, wherein determining the adjustment matrix comprises calculating the adjustment matrix.
[0217] Example 49. The method of any combination of Examples 26-48, wherein determining the adjustment matrix comprises: storing a portion of one or more frames of the display content; applying a timestamp to each portion of the one or more frames of the display content; determining a camera sensor exposure time; determining which frames are associated with the camera sensor exposure time based on the applied timestamps; and determining the adjustment matrix based on the portion of the one or more frames being associated frames.
[0218] Example 50. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: receive first image data from a camera sensor configured to receive light through at least a portion of a display; receive at least a portion of display content; determine an adjustment matrix based on at least a portion of the display content; apply the adjustment matrix to the first image data to create second image data; and output the second image data.
[0219] Example 51. An image capture device comprising: a display configured to display a captured image; a camera sensor disposed to receive light through at least a portion of the display; a memory configured to store the captured image; and one or more processors coupled to the camera sensor, the display, and the memory and configured to: receive a signal from the sensor; determine a user interface mode based at least in part on the signal, the user interface mode comprising a first mode or a second mode, wherein the first mode comprises a first number of black pixels in an area of the display and the second mode comprises a second number of black pixels in the area of the display, the first number being greater than the second number; and receive image data from the camera sensor.
[0220] Example 52. The image capture device of Example 51, wherein the sensor is an ambient light sensor.
[0221] Example 53. An image capture device according to Example 51 or Example 52, wherein the one or more processors are further configured to: determine whether the signal is below a threshold; and based on the signal being below the threshold, control the display to display the first mode.
[0222] Example 54. An image capture device according to any combination of Examples 51-53, wherein the signal is a first signal, and the one or more processors are further configured to: receive a second signal from the sensor; determine whether the second signal is below the threshold; and based on the second signal not being below the threshold, control the display to display the second mode.
[0223] Example 55. The image capture device of any combination of Examples 51-54, wherein the one or more processors are further configured to: launch an image capture application, wherein the one or more processors determine the user interface mode based on launching the image capture application.
[0224] Example 56. The image capture device of any combination of Examples 52-55, further comprising: an ambient light sensor configured to sense ambient light levels.
[0225] Example 57. The image capture device of Example 51, wherein the sensor is the camera sensor.
[0226] Example 58. The image capture device of Example 51, wherein the sensor is a touch sensor.
[0227] Example 59. An image capture device according to Example 51, wherein the second number is zero.
[0228] Example 60. The image capture device of any combination of Examples 51-59, wherein the user interface mode further comprises: a third mode including a third number of black pixels.
[0229] Example 61. An image capture device according to Example 60, wherein the third number of black pixels is greater than the second number and less than the first number.
[0230] Example 62. The image capture device of any combination of Examples 51-61, wherein the one or more processors are further configured to: control the display to fade in the black pixels in the area over the camera sensor.
[0231] Example 63. The image capture device of any combination of Examples 51-62, wherein the one or more processors are further configured to: control the display to fade out the black pixels in the area over the camera sensor.
[0232] Example 64. The image capture device of Example 63, wherein the one or more processors are further configured to control the display to fade out the black pixels based on an image capture application being closed.
[0233] Example 65. The image capture device of Example 63, wherein the one or more processors control the display to fade out the black pixels based on the new ambient light level not being below the threshold.
[0234] Example 66. A method comprises: receiving, by an image capture device, a signal from a sensor; determining, by the image capture device and based at least in part on the signal, a user interface mode, the user interface mode comprising a first mode or a second mode, wherein the first mode comprises a first number of black pixels in an area of a display and the second mode comprises a second number of black pixels in the area of the display, the first number being greater than the second number; and receiving, by the image capture device, image data from a camera sensor, wherein the camera sensor is configured to receive light through at least a portion of the display.
[0235] Example 67. The method of Example 66, wherein the sensor is an ambient light sensor.
[0236] Example 68. The method according to Example 66 or Example 67 further includes: determining, by the image capture device, whether the signal is below a threshold; and controlling, by the image capture device, the display to display the first mode based on the signal being below the threshold.
[0237] Example 69. The method of any combination of Examples 66-68, further comprising: receiving a second signal from the sensor; determining whether the second signal is below the threshold; and based on the second signal not being below the threshold, controlling the display to display the second mode.
[0238] Example 70. The method of any combination of Examples 66-69, further comprising launching, by the image capture device, an image capture application, wherein the image capture device determines the user interface mode based on launching the image capture application.
[0239] Example 71. The method of Example 66, wherein the sensor is the camera sensor.
[0240] Example 72. The method of Example 66, wherein the sensor is a touch sensor.
[0241] Example 73. The method of Example 66, wherein the second amount is zero.
[0242] Example 74. The method of any combination of Examples 66-73, wherein the user interface mode further comprises: a third mode including a third number of black pixels.
[0243] Example 75. The method of Example 74, wherein the third number of black pixels is greater than the second number and less than the first number.
[0244] Example 76. The method of any combination of Examples 66-75, further comprising controlling the display to fade in the black pixels in the area over the camera sensor.
[0245] Example 77. The method of any combination of Examples 66-76, further comprising controlling the display to fade out the black pixels in the area over the camera sensor.
[0246] Example 78. The method of any combination of Examples 66-77, further comprising controlling the display to fade out the black pixels based on an image capture application being closed.
[0247] Example 79. The method of any combination of Examples 66-78, further comprising controlling the display to fade out the black pixels based on the new ambient light level not being below the threshold.
[0248] Example 80. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: receive a signal from a sensor; determine a user interface mode based at least in part on the signal, the user interface mode comprising a first mode or a second mode, wherein the first mode comprises a first number of black pixels in an area of a display and the second mode comprises a second number of black pixels in the area of the display, the first number being greater than the second number; and receive image data from a camera sensor, wherein the camera sensor is configured to receive light through at least a portion of the display.
[0249] Example 81. An image capture device comprising: means for receiving a signal from a sensor; means for determining a user interface mode based at least in part on the signal, the user interface mode comprising a first mode or a second mode, wherein the first mode comprises a first number of black pixels in an area of a display and the second mode comprises a second number of black pixels in the area of the display, the first number being greater than the second number; and means for receiving, by the image capture device, image data from a camera sensor, wherein the camera sensor is configured to receive light through at least a portion of the display.
[0250] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or sent through a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media that corresponds to tangible media (e.g., data storage media, or communication media), including any media that facilitates, for example, the transfer of a computer program from one place to another according to a communication protocol. In terms of apparatus, a computer-readable medium may generally correspond to: (1) a tangible computer-readable storage medium that is non-transitory; or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures to implement the techniques described in the present disclosure. A computer program product may include a computer-readable medium.
[0251] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices, or other magnetic storage devices, flash memory, or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection may be appropriately referred to as a computer-readable medium. For example, if the software is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are directed to non-transient, tangible storage media. Disks and optical disks as used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and blue-ray disks, wherein disks typically reproduce data magnetically, and optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.
[0252] Instructions may be executed by one or more processors (e.g., one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits). Thus, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Moreover, these techniques may be fully implemented in one or more circuits or logic elements.
[0253] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in the present disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units. Rather, as described above, the various units may be provided by a collection of interoperable hardware units, including one or more processors as described above in combination with appropriate software and / or firmware.
[0254] Various examples have been described. These and other examples are within the scope of the following claims.
Claims
1. An image capturing device, comprising: a display configured to display the captured image; a camera sensor disposed to receive light through at least a portion of the display; a memory configured to store the captured image; as well as one or more processors coupled to the camera sensor, the display, and the memory and configured to: receiving a signal from a sensor; determining a user interface mode based at least in part on the signal, the user interface mode comprising a first mode or a second mode, wherein the first mode comprises a first number of black pixels in an area of the display and the second mode comprises a second number of black pixels in the area of the display, the first number being greater than the second number; and Image data is received from the camera sensor.
2. The image capture device according to claim 1, wherein: The sensor is an ambient light sensor.
3. The image capture device according to claim 1 or 2, wherein: The one or more processors are further configured to: determining whether the signal is below a threshold; and Based on the signal being below the threshold, the display is controlled to display the first pattern.
4. The image capture device according to claim 1, wherein: The signal is a first signal, and the one or more processors are further configured to: receiving a second signal from the sensor; determining whether the second signal is below a threshold; and Based on the second signal not being lower than the threshold, the display is controlled to display the second mode.
5. The image capture device according to claim 1, wherein: The one or more processors are further configured to: Launch the image capture application, Wherein the one or more processors determine the user interface mode based on launching the image capture application.
6. The image capture device of claim 2, further comprising: An ambient light sensor is configured to sense ambient light levels.
7. The image capture device according to claim 1, wherein: The sensor is the camera sensor.
8. The image capture device according to claim 1, wherein: The sensor is a touch sensor.
9. The image capture device according to claim 1, wherein: The second amount is zero.
10. The image capture device according to claim 1, wherein: The user interface mode also includes: a third mode including a third number of black pixels.
11. The image capture device of claim 10, wherein: The third number of black pixels is greater than the second number and less than the first number.
12. The image capture device of claim 1, wherein: The one or more processors are further configured to control the display to fade in the black pixels in the area over the camera sensor.
13. The image capture device of claim 1, wherein: The one or more processors are further configured to control the display to fade out the black pixels in the area over the camera sensor.
14. The image capture device of claim 13, wherein: The one or more processors are further configured to control the display to fade out the black pixels based on the image capture application being closed.
15. The image capture device of claim 13, wherein: The one or more processors control the display to fade out the black pixels based on the new ambient light level not being below a threshold.
16. A method for image processing, comprising: receiving, by an image capture device, a signal from a sensor; determining, by the image capture device and based at least in part on the signal, a user interface mode, the user interface mode comprising a first mode or a second mode, wherein the first mode comprises a first number of black pixels in an area of the display and the second mode comprises a second number of black pixels in the area of the display, the first number being greater than the second number; and Image data is received by the image capture device from a camera sensor, wherein the camera sensor is configured to receive light through at least a portion of the display.
17. The method according to claim 16, wherein: The sensor is an ambient light sensor.
18. The method according to claim 16 or 17, further comprising: determining, by the image capture device, whether the signal is below a threshold; as well as Based on the signal being below the threshold, the display is controlled by the image capture device to display the first pattern.
19. The method according to claim 16, further comprising: receiving a second signal from the sensor; determining whether the second signal is below a threshold; as well as Based on the second signal not being lower than the threshold, the display is controlled to display the second mode.
20. The method of claim 16, further comprising: launching an image capture application by the image capture device, Wherein, the image capture device determines the user interface mode based on launching the image capture application.
21. The method according to claim 16, wherein: The sensor is the camera sensor.
22. The method according to claim 16, wherein: The sensor is a touch sensor.
23. The method according to claim 16, wherein: The second amount is zero.
24. The method according to claim 16, wherein: The user interface mode also includes: a third mode including a third number of black pixels.
25. The method according to claim 24, wherein: The third number of black pixels is greater than the second number and less than the first number.
26. The method of claim 16, further comprising: The display is controlled to fade in the black pixels in the area over the camera sensor.
27. The method of claim 16, further comprising: The display is controlled to fade out the black pixels in the area over the camera sensor.
28. The method of claim 16, further comprising: Based on the image capture application being closed, the display is controlled to fade out the black pixels.
29. The method of claim 16, further comprising: Based on the new ambient light level not being below a threshold, the display is controlled to fade out the black pixel.
30. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: receiving a signal from a sensor; determining a user interface mode based at least in part on the signal, the user interface mode comprising a first mode or a second mode, wherein the first pattern comprising a first number of black pixels in an area of the display, and the second pattern comprising a second number of black pixels in the area of the display, the first number being greater than the second number; as well as Image data is received from a camera sensor, wherein the camera sensor is configured to receive light through at least a portion of the display.
31. An image capture device comprising: a unit for receiving a signal from a sensor; means for determining a user interface mode based at least in part on the signal, the user interface mode comprising a first mode or a second mode, wherein the first mode comprises a first number of black pixels in an area of a display and the second mode comprises a second number of black pixels in the area of the display, the first number being greater than the second number; and Means for receiving, by the image capture device, image data from a camera sensor, wherein the camera sensor is configured to receive light through at least a portion of the display.
32. A method of image processing, comprising: receiving first image data from a camera sensor, the camera sensor being configured to receive light associated with the first image data through at least a portion of the display while the display is displaying display content; applying a timestamp to each portion of one or more frames of the display; determining, based on the applied timestamp, which frames of the one or more frames of the display content are associated with a camera sensor exposure time; determining an adjustment matrix based on the portion of the one or more frames being the associated frame; as well as The adjustment matrix is applied to the first image data to create second image data.
33. A method of image processing, comprising: receiving first image data from a camera sensor, the camera sensor being configured to receive light associated with the first image data through at least a portion of the display while the display is displaying display content; determining an adjustment matrix based on an aging factor, wherein the aging factor is based on a statistical measurement of a comparison of transmittance decay of pixels in an area above the camera sensor; as well as The adjustment matrix is applied to the first image data to create second image data.
