Under-display camera synchronization with display pixel operation

Abstract

The image capture device includes a memory configured to store display content and image information received from a camera sensor, the camera sensor configured to receive light through at least a portion of a display. The image capture device includes one or more processors coupled to the memory. The one or more processors are configured to determine a camera sensor blanking period (310). The one or more processors are configured to control the display to display content via one or more of a plurality of pixels in at least a portion of the display during the camera sensor blanking period (310). The one or more processors are configured to control the display to not display content via one or more of the plurality of pixels outside the camera sensor blanking period (310).

Description

[Technical Field]

[0001]

[0001] This application claims priority to U.S. Application No. 17 / 069,683, filed October 13, 2020, the entire contents of which are incorporated herein by reference.

[0002] FIELD OF THE DISCLOSURE

[0002] The present disclosure relates generally to image capture devices. [Background technology]

[0003] Image capture devices (e.g., digital cameras) are typically incorporated into a wide variety of devices. In this disclosure, an image capture device refers to any device capable of capturing one or more digital images, including devices capable of capturing still images and devices capable of capturing a sequence of images to record video. By way of example, an image capture device may comprise a standalone digital camera or digital video camcorder, a wireless communication device with a camera such as a mobile phone (including a cellular or satellite radiotelephone), a camera-equipped tablet or personal digital assistant (PDA), a computing device including a camera such as a so-called “webcam,” a virtual reality and / or augmented reality head-mounted device, or any device with digital imaging or video capabilities. Summary of the Invention

[0004]

[0004] Generally, this disclosure describes techniques for displaying content while the camera is on in an image capture device having a camera sensor disposed under a display. As used herein, "content" may include image(s) captured by the image capture device, visual representations of files stored in memory locations, software applications, user interfaces including GUIs, network-accessible content objects, images received but not captured by the image capture device (e.g., downloaded images, images sent by text message, images sent by email, images sent through an application, etc.), background images, and other information. Also, as used herein, a camera sensor disposed under the display means that the camera sensor is configured to receive light through at least a portion of the display. Such a camera sensor is sometimes referred to herein as an under-display camera.

[0005]

[0005] One way to maximize the display size on an image capture device is to place one or more camera sensors below the display so that the one or more camera sensors receive light that passes through at least a portion of the display. In some examples, the display may be a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, an active matrix organic light-emitting diode (AMOLED), or other display. When a camera sensor is positioned to receive light that passes through at least a portion of the display and the display is displaying content, the light that may be received by the camera sensor may be scattered. For example, when one or more pixels in an area on the camera sensor are active or addressed, less light may reach the camera sensor through the display than may reach the camera sensor when no pixels in the area on the camera sensor are active or addressed. Thus, the scattered light may attenuate the light that reaches the camera sensor. The attenuated light may reduce the image quality of the captured image.

[0006]

[0006] This disclosure describes techniques for avoiding or mitigating attenuation of received light caused by displayed content. This disclosure also describes techniques for avoiding or mitigating attenuation of received light caused by displayed content while avoiding degrading the quality of the displayed content.

[0007]

[0007] In one example, the present disclosure describes an image capture device including a memory configured to store display content and image information received from a camera sensor, and one or more processors coupled to the memory, the camera sensor configured to receive light through at least a portion of a display, wherein the one or more processors are configured to: determine a camera sensor blanking period; control the display to display content via one or more of a plurality of pixels in at least a portion of the display during the camera sensor blanking period; and control the display to not display content via one or more of the plurality of pixels outside the camera sensor blanking period.

[0008]

[0008] In another example of the present disclosure, a method includes determining a camera sensor blanking period; displaying content via one or more of a plurality of pixels in at least a portion of a display during the camera sensor blanking period, the camera sensor blanking period being associated with a camera sensor configured to receive light through at least a portion of the display; and refraining from displaying content via one or more of the plurality of pixels outside the camera sensor blanking period.

[0009]

[0009] In another example, the present disclosure describes a non-transitory computer-readable storage medium that stores instructions that, when executed, cause one or more processors to determine a camera sensor blanking period; control a display to display content via one or more of a plurality of pixels in at least a portion of the display during the camera sensor blanking period, the camera sensor blanking period being associated with a camera sensor configured to receive light through at least a portion of the display; and control the display to not display content via one or more of the plurality of pixels outside the camera sensor blanking period.

[0010]

[0010] In another example, an image capture device includes means for determining a camera sensor blanking period, means for controlling the display to display content through one or more of a plurality of pixels in at least a portion of the display during the camera sensor blanking period, the camera sensor blanking period being associated with a camera sensor configured to receive light through at least a portion of the display, and means for controlling the display not to display content through one or more of the plurality of pixels outside the camera sensor blanking period.

[0011] The details of one or more aspects of this disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described in this disclosure will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0012] [Figure 1]

[0012] A block diagram of an exemplary image capture device configured to implement the techniques of this disclosure. [Figure 2A]

[0013] FIG. 1 is a block diagram illustrating an example of an image capture device having a front-facing camera sensor and a display. [Figure 2B] FIG. 1 is a block diagram illustrating an example of an image capture device having a front-facing camera sensor and a display. [Figure 2C] FIG. 1 is a block diagram illustrating an example of an image capture device having a front-facing camera sensor and a display. [Figure 2D] FIG. 1 is a block diagram illustrating an example of an image capture device having a front-facing camera sensor and a display. [Figure 3]

[0014] FIG. 1 is a block diagram illustrating an exploded cross-sectional view of an exemplary image capture device configured to implement the techniques of this disclosure. [Figure 4A]

[0015] FIG. 1 is a block diagram illustrating characteristics of an exemplary organic light-emitting diode (OLED) display in accordance with techniques of this disclosure. [Figure 4B] FIG. 1 is a block diagram illustrating characteristics of an exemplary organic light-emitting diode (OLED) display in accordance with techniques of this disclosure. [Figure 5]

[0016] 1 is a conceptual diagram illustrating an example technique for displaying content on an image capture device having an under-display camera sensor while the camera is on. [Figure 6]

[0017] 1 is a timing diagram illustrating an example display frame and camera sensor blanking period in accordance with the techniques of this disclosure. [Figure 7]

[0018] FIG. 10 is a timing diagram illustrating another example display frame and camera sensor blanking period in accordance with the techniques of this disclosure. [Figure 8]

[0019] 1 is a timing diagram illustrating an example display frame, buffer delta(s), and camera sensor blanking period in accordance with techniques of this disclosure. [Figure 9A]

[0020] 1 is a conceptual diagram illustrating an exemplary display technique according to the present disclosure. [Figure 9B] 1 is a conceptual diagram illustrating an exemplary display technique according to the present disclosure. [Figure 10]

[0021] FIG. 1 is a block diagram of an example image capture device that may implement the techniques of this disclosure. [Figure 11]

[0022] 1 is a flowchart illustrating an example technique for displaying content on an image capture device having an under-display camera sensor, according to this disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013]

[0023] This disclosure describes techniques for displaying images on a display while the camera is on in an image capture device having an under-display camera sensor. The display may use a transparent material with a pixel structure designed to allow light to transmit through the display to the camera sensor. Camera sensors used in such a manner may be larger and have wider aperture lenses than other front-facing “selfie” cameras. Furthermore, multiple camera sensors may be disposed below the display. For example, the number and / or size of camera sensors need not be limited or constrained by the bezel or border space around the display. By positioning the camera sensors so that they receive light through at least a portion of the display on the device (e.g., below the display), the size of the usable display space may be expanded compared to a similarly sized device with space on the front of the device dedicated to the camera sensor(s). Alternatively, a smaller form factor may be used to provide the same usable display size. For example, when the camera sensor(s) are positioned so that they receive light through at least a portion of the display (e.g., below the display), a larger bezel may not be required because the camera sensor(s) are positioned below the display rather than on the bezel. This allows for the use of a smaller form factor to achieve the same usable display size. Furthermore, by positioning the camera sensor so that it receives light at least partially through the display (e.g., below the display), the camera sensor can be placed anywhere below the display. For example, the camera sensor can be placed where a user's eyes may be directed when taking a "selfie."In this way, the line of sight of the eyes in an image captured by the camera sensor may appear to be looking at the camera, rather than below the camera, as may occur if the camera sensor is located above a display or near the top of the image capture device. Although the techniques of this disclosure are generally described herein with respect to "selfie" cameras, the techniques of this disclosure may also be applied to rear-facing cameras located below a display on a device.

[0014]

[0024] For many image capture devices, it may be desirable to maximize the size of the display on the image capture device. This is especially true for smaller image capture devices such as mobile phones and other mobile devices. Many image capture devices (e.g., mobile devices) include a front-facing camera (a "selfie" camera) that faces toward the user of the mobile device. Maximizing the display size on image capture devices with front-facing camera(s) is not without limitations. The front-facing camera is located on the front of the image capture device between the edge of the device and the edge of the display. To maximize the display size on image capture devices with a front-facing camera, some manufacturers have enlarged the display and introduced a notch in the display to avoid covering the camera sensor with the display. Others have enlarged the display to substantially cover the front of the image capture device and added a pop-up camera rather than placing the camera sensor on the body of the image capture device.

[0015]

[0025] One way to maximize display size is to position the camera sensor below the display (e.g., in a manner such that the camera sensor receives light through at least a portion of the display). However, by positioning the camera sensor below the display, the display may cause attenuation, spatial stride and shadow, light scattering or diffusion, and / or other undesirable effects in the image signal captured by the camera sensor, especially when the display is displaying content. For example, some or all of the sub-pixels immediately above the camera sensor may be actively displaying content, and the camera sensor may receive less light than it would if the sub-pixels immediately above the camera sensor were not actively displaying content, which may result in an image of poor quality being captured by the camera sensor.

[0016]

[0026] Some image capture devices may employ techniques to attempt to reduce the impact of light attenuation through the display when the display is actively displaying content on captured image quality. For example, some image capture devices may address or turn on only a subset of pixels above the camera sensor while displaying an image. While this may reduce the amount of light attenuation received by the camera sensor due to the displayed content, it may also adversely affect the quality of the displayed content in the area on the camera sensor because fewer pixels are used to display the content in that area than the rest of the display. The use of fewer pixels to display content may result in a blockiness effect or lack of detail in that area of ​​the display compared to other areas of the display. This use of fewer pixels to display content on the camera sensor may be aesthetically pleasing and / or distracting to the user.

[0017]

[0027] The techniques of this disclosure may be used separately or in any combination. According to the techniques of this disclosure, in one example, an image capture device may include a memory configured to store display content and image information received from a camera sensor, the camera sensor being configured to receive light through at least a portion of a display. The image capture device may include one or more processors coupled to the memory. The one or more processors may be configured to determine a camera sensor blanking period, control the display to display content via one or more of a plurality of pixels in at least a portion of the display during the camera sensor blanking period, and control the display to not display content via one or more of the plurality of pixels outside the camera sensor blanking period.

[0018]

[0028] In accordance with the techniques of this disclosure, in another example, an image capture device may include a memory configured to store display content and image information received from a camera sensor, the camera sensor configured to receive light through at least a portion of a display. The image capture device may include one or more processors coupled to the memory. The one or more processors may be configured to determine a first level of brightness for one or more of the plurality of pixels. The one or more processors may be configured to determine a second level of brightness for one or more of the plurality of pixels. The one or more processors may be configured to control the display to display content via one or more of the plurality of pixels at the second level of brightness during a camera sensor blanking period. The second level of brightness may be greater than the first level of brightness and may be based at least in part on the first level of brightness.

[0019]

[0029] 1 is a block diagram illustrating device 2 that may be configured to implement the techniques of this disclosure. Device 2 may form all or part of an image capture device or digital video device capable of coding and transmitting and / or capturing still images and / or video sequences. By way of example, device 2 may form part of a wireless mobile communications device such as a cellular or satellite radiotelephone, a smartphone, a standalone digital camera or video camcorder, a personal digital assistant (PDA), a tablet computer, a laptop computer, or any device with imaging or video capabilities in which image processing is desirable.

[0020]

[0030] As shown in FIG. 1, device 2 includes an image processor 4 for storing raw image data and performing various processing techniques on such data. Image processor 4 may comprise one or more integrated circuits, including a digital signal processor (DSP), on-chip memory, and possibly hardware logic or circuitry. More generally, image processor 4 may comprise any combination of processor, hardware, software, or firmware, and various components of image processor 4 may be implemented as such. Image processor 4 may also comprise a single integrated chip or encoder / decoder (codec), if desired.

[0021]

[0031] 1, image processing device 4 includes local memory 8, memory controller 10, and image signal processor 6. Image signal processor 6 may be a general-purpose processing unit or may be a processor specifically designed for imaging applications, for example, for a handheld electronic device. As shown, image signal processor 6 may be coupled to local memory 8 and external memory 14 via memory controller 10. In some examples, local memory 8 may be incorporated into image signal processor 6, for example, as a cache memory.

[0022]

[0032] 1 , image signal processor 6 may be configured to perform an automatic exposure control (AEC) process 20, an auto white balance (AWB) process 22, an auto focus (AF) process 24, a lens shade compensation (LSC) process 28, and / or a fixed pattern noise compensation (FPNC) process 30. In some examples, image signal processor 6 may include hardware specific circuitry (e.g., an application specific integrated circuit (ASIC)) configured to implement AEC process 20, AWB process 22, AF process 24, LSC process 28, and / or FPNC process 30. In other examples, image signal processor 6 may be configured to execute software and / or firmware to implement AEC process 20, AWB process 22, AF process 24, LSC process 28, and / or FPNC process 30. When configured in software, code for AEC process 20, AWB process 22, AF process 24, LSC process 28, and / or FPNC process 30 may be stored in local memory 8 and / or external memory 14. In other examples, image signal processor 6 may use a combination of hardware, firmware, and / or software to implement AEC process 20, AWB process 22, AF process 24, LSC process 28, and / or FPNC process 30. When configured as software, AEC process 20, AWB process 22, AF process 24, LSC process 28, and / or FPNC process 30 may include instructions that configure image signal processor 6 to perform various image processing and device management tasks.

[0023]

[0033] AEC process 20 may include instructions for configuring, calculating, storing, and / or applying exposure settings for camera module 12. The exposure settings may include a shutter speed (e.g., how long camera module 12 may capture an image) and an aperture setting to be used to capture an image. In accordance with the techniques of this disclosure, image signal processor 6 may use depth information received by a depth sensor (not shown) of camera module 12 to better identify subjects in the image and set exposure settings based on the identified subjects. AF process 24 may include instructions for configuring, calculating, storing, and / or applying autofocus settings for camera module 12.

[0024]

[0034] The AWB process 22 may include instructions for configuring, calculating, storing, and / or applying AWB settings (e.g., AWB gain) that may be applied to one or more images captured by the camera module 12. In some examples, the AWB gain determined by the AWB process 22 may be applied to the image from which the AWB gain was determined. In other examples, the AWB gain determined by the AWB process 22 may be applied to one or more images captured after the image from which the AWB gain was determined. Thus, the AWB gain may be applied to a second image captured following a first image from which the AWB gain is determined. In one example, the second image may be an image captured immediately after the first image from which the AWB gain was determined. That is, if the first image is frame N, the second image to which the AWB gain is applied is frame N+1. In another example, the second image may be an image captured two images after the first image from which the AWB gain was determined. That is, if the first image is frame N, the second image to which the AWB gain is applied is frame N+2. In other examples, the AWB gain may be applied to an image captured further in time from the first image (e.g., frame N+3, frame N+4, etc.) In other examples, the AWB gain may be applied to the first image from which the AWB gain is determined.

[0025]

[0035] LSC process 28 may include instructions for configuring, calculating, storing, and / or applying lens shade compensation gains. For example, LSC process 28 may compensate for light falloff toward the edges of an image due to a camera lens.

[0026]

[0036] FPNC process 30 may include instructions for configuring, calculating, storing, and / or applying an FPN compensation process. For example, FPNC process 30 may subtract a master dark frame from a captured image to compensate for FPN.

[0027]

[0037] Local memory 8 may store raw image data and may also store processed image data following any processing performed by image signal processor 6. Local memory 8 may be formed by any of a variety of non-transitory memory devices, such as synchronous dynamic random access memory (DRAM), including DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Memory controller 10 may control memory organization within local memory 8. Memory controller 10 may also control memory loading from local memory 8 to image signal processor 6 and write-back from image signal processor 6 to local memory 8. Images to be processed by image signal processor 6 may be loaded directly from camera module 12 to image signal processor 6 following image capture or may be stored in local memory 8 during image processing.

[0028]

[0038] As mentioned, device 2 may include camera module 12 for capturing images to be processed, although the present disclosure is not necessarily limited in this respect. Camera module 12 may comprise an array of solid-state sensor elements, such as complementary metal-oxide semiconductor (CMOS) sensor elements, charge-coupled device (CCD) sensor elements, etc. Alternatively or additionally, camera module 12 may comprise a set of image sensors including a color filter array (CFA) arranged on the surface of each sensor. Camera module 12 may be directly coupled to image signal processor 6 to avoid latency in image processing. Camera module 12 may be configured to capture still images or full-motion video sequences, in which case image processing may be performed on one or more image frames of the video sequence.

[0029]

[0039] Camera module 12 may send pixel values ​​(e.g., in Bayer or red-green-blue (RGB) format) and / or raw statistics messages describing the captured image to image signal processor 6. In some examples, the pixel values ​​may include information related to the brightness level of each of the pixels. Generally, image signal processor 6 may be configured to analyze the raw statistics and depth information to calculate and / or determine imaging parameters, such as sensor gain, R / G / B gain, AWB gain, shutter speed, aperture size, etc. The calculated and / or determined imaging parameters may be applied to the captured image, applied to one or more subsequently captured images, and / or sent to camera module 12 to adjust exposure and / or focus settings.

[0030]

[0040] In some examples, the first level of brightness may be a level of brightness that device 2 may display for a given pixel when the camera of device 2 is off. For example, the first level of brightness of a pixel may be a level of brightness captured by camera module 12, a level of brightness captured by camera module 12 and image processed by image signal processor 6 (e.g., by AWB process 22), or a level of brightness stored in memory. In some examples, rather than obtaining the first level of brightness for a pixel from pixel values ​​in camera module 12 or from image signal processor 6, device 2 may determine the first level of brightness based on levels of brightness from one or more pixels outside the area above camera module 12. For example, device 2 may determine the first level of brightness for a pixel by determining levels of brightness for pixels outside the area immediately above camera module 12. In another example, one or more processors 110 may determine the first level of brightness for a pixel by determining an average level of brightness for multiple pixels outside the area above camera module 12.

[0031]

[0041] Device 2 may include a display 16 configured to display content. In some examples, the display content may be an image captured by camera module 12. After image processing of the captured image by image signal processor 6, the image may be written to local memory 8 or external memory 14. The processed image may then be sent to display 16 for presentation to a user. Display 16 may additionally or alternatively display other information, including visual representations of files stored in memory locations (e.g., external memory 14), software applications installed in image signal processor 6, user interfaces including GUIs, network-accessible content objects, images received but not captured by an image capture device (e.g., downloaded images, images sent by text message, images sent by email, images sent through an application, etc.), background images, and other information. Display 16 may display content at a frame rate. For example, display 16 may display 30 frames per second (fps). Each displayed frame may have a display frame length, i.e., the length of time the frame is displayed, equal to the inverse of the frame rate (e.g., 1 divided by the frame rate). For example, if display 16 has a frame rate of 30 fps, then each display frame is approximately 0.0333 seconds long.

[0032]

[0042] In some examples, device 2 may include multiple memories. For example, device 2 may include external memory 14, which generally comprises a relatively large memory space. External memory 14 may comprise, for example, DRAM or flash memory. In other examples, external memory 14 may comprise non-volatile memory or any other type of data storage unit. In contrast to external memory 14, local memory 8 may comprise a smaller and faster memory space, although the present disclosure is not necessarily limited in this respect. By way of example, local memory 8 may comprise SDRAM. In any case, external memory 14 and local memory 8 are merely examples and may be combined into the same memory unit or implemented in any number of other configurations.

[0033]

[0043] Device 2 may also include a transmitter (not shown) for transmitting the processed image or a coded sequence of images to another device. Indeed, the techniques of this disclosure may be used in handheld wireless communication devices (such as smartphones) that include digital camera or digital video capabilities. In that case, the device would also include a modulator-demodulator (modem) to enable wireless modulation of a baseband signal onto a carrier waveform to enable wireless communication of the modulated information.

[0034]

[0044] Local memory 8, display 16, and external memory 14 (and other components, if desired) may be coupled via a communications bus 15. Several other elements may also be included in device 2 but are not specifically shown in Figure 1 for simplicity and ease of explanation. The architecture shown in Figure 1 is exemplary only, as the techniques described herein may be implemented with a variety of other architectures.

[0035]

[0045] Device 2 represents an exemplary image capture device including a camera sensor configured to receive light through at least a portion of a display, a display including a plurality of pixels and configured to display content via the plurality of pixels, a memory configured to store the displayed content, and one or more processors coupled to the camera sensor, the display, and the memory, wherein the one or more processors are configured to determine a camera sensor blanking period and control the display to display content via one or more of the plurality of pixels in at least a portion of the display during the camera sensor blanking period.

[0036]

[0046] 2A-2D are block diagrams illustrating example image capture devices (such as smartphones). Each image capture device in FIGS. 2A-2D may be an example of device 2. Each image capture device is illustrated with respect to a display and a front-facing camera sensor. In this context, a front-facing camera sensor is a camera sensor that faces a user of the image capture device in typical operation. For example, a front-facing camera sensor is generally on the same side of the device as the main display. Each front-facing camera sensor may be part of a camera module, such as camera module 12. For example, in FIG. 2A , image capture device 200 includes display 202, camera sensor 204, and button 206. Button 206 may serve multiple purposes, such as activating image capture device 200 or changing what is displayed on display 202. As can be seen, button 206 and camera sensor 204 occupy space on the front of image capture device 200. By locating camera sensor 204 and button 206 on the front of image capture device 200, less area is available for display 202.

[0037]

[0047] In the example of FIG. 2B , image capture device 210, on the other hand, does not have a button on the front. In this case, the button may be on the side, or the button functionality may be included in display 212 (e.g., through a touch display interface). Image capture device 210 is shown with respect to camera sensor 214 and notch 216. Notch 216 may be an area removed from the display prior to assembly of image capture device 210. In this example, the area covered by notch 216 is therefore not part of display 212 and does not display content. Notch 216 may be employed to increase the proportion of the front surface of image capture device 210 occupied by display 212, as compared to image capture device 200 of FIG. 2A .

[0038]

[0048] In the example of Figure 2C, image capture device 220 has a display 222 and a pop-up camera 226. A camera sensor 224 may be included in pop-up camera 226. In the example of image capture device 220, the entire display 222 may display content, and there is no notch as in image capture device 210 of Figure 2B.

[0039]

[0049] In the example of FIG. 2D , image capture device 230 includes display 232 and camera sensor 234. In some examples, image capture device 230 may include two or more camera sensors. For example, image capture device 230 may include camera sensor 234 and camera sensor 238. Image capture device 230 may comprise or be an example of device 2, and display 232 may be an example of display 16. In the example of image capture device 230 of FIG. 2D , unlike the examples of FIGS. 2A-2C , camera sensor 234 and camera sensor 238 are disposed below display 232. In some examples, only a portion, but not all, of camera sensor 234 or camera sensor 238 may be disposed below display 232. Display 232 may include a transparent layer. Region 232A of display 232, region 232B of display 232, and region 232C of display 232 are further described with respect to Figures 5, 7, 8A, 8B, and 9. Although the techniques of this disclosure are generally described with reference to image capture devices having a camera sensor disposed below a display, such as image capture device 230, the techniques of this disclosure may be used with other image capture devices, such as image capture devices 200, 210, and 220, or image capture devices having a camera sensor disposed partially below a display.

[0040]

[0050] 2A-2D , image capture device 200 has a larger form factor than image capture devices 210, 220, and 230, but has a display 202 that is the same size as display 222 of image capture device 220 and display 232 of image capture device 230, and has a display 202 that is slightly larger than display 212 of image capture device 210 due to notch 216. Image capture device 210 has the same size form factor as image capture devices 220 and 230, but it has less usable display space on display 212 due to notch 216. Furthermore, notch 216 may be distracting to some users. Image capture device 220 has the same form factor and usable display size as image capture device 230, but image capture device 220 has a moving part in pop-up camera 226. These moving parts can break or become jammed with repeated use or if a user drops image capture device 220 on a hard surface. Therefore, it may be desirable to place the camera sensor below the display, as this may maximize display space while avoiding notching and moving mechanical parts.

[0041]

[0051] Further, in image capture device 230 of FIG. 2D , camera sensor 234 may be located anywhere below display 232. In this example, the camera sensor is shown positioned in the center of display 232. Such a location may be more desirable than the location of the forward-facing camera sensor in image capture device 200, image capture device 210, and image capture device 220. For example, a user attempting to take a “selfie” may view a live image of themselves on the display of the image capture device. The farther the camera sensor is located from where the user's eyes are pointed, the more likely the captured image will show the eyes staring away from the camera sensor. This gaze phenomenon may result in aesthetically unappealing images, with the user's gaze appearing below (or above) where one might expect the user to be looking (e.g., downward (or upward) from the image viewer's eyes, such as below or above the image viewer's head, rather than toward the image viewer's eyes).

[0042]

[0052] FIG. 3 is a block diagram of an exploded side view of an example of the image capture device 230 shown in FIG. 2D . For simplicity, the camera sensor 238 is not shown in FIG. 3 or the remaining figures and may function similarly to the camera sensor 234 described herein. In the example of FIG. 3 , the image capture device 230 includes a display 232, a camera sensor (CS) 234, and a housing 236. The housing 236 may include an electronic circuit board, a processor, memory, a battery, a radio frequency circuit, an antenna, and other components. As shown, the display 232 is disposed above the camera sensor 234, and the camera sensor 234 is disposed below or beneath the display 232. In this example, as in FIG. 2D , the camera sensor 234 is a forward-facing camera. The camera sensor 234 is configured to capture images by receiving ambient light that passes through at least a portion of the display 232. That is, the camera sensor 234 may receive ambient light that passes through at least a portion of the display 232 before entering the camera sensor 234. As used herein, a camera sensor being under, below, or beneath a display, or a display being on the camera sensor, describes a camera sensor configured or arranged to capture an image by receiving ambient light that passes through at least a portion of a display, such as the display 232. The display 232, when displaying content, may emit light toward the user and generally away from the camera sensor 234. The camera sensor 234 may be actively capturing an image(s) while the display 232 is or is actively displaying content. In other words, the camera sensor 234 may receive ambient light that passes through at least a portion of the display 232 while the display 232 may emit light toward the user.

[0043]

[0053] 4A and 4B are simplified diagrams of an exemplary OLED display that may be used in accordance with the techniques of the present disclosure. While the display in FIGS. 4A and 4B is shown as an OLED display, the techniques of the present disclosure may be used with any display configured to allow light to pass through the display to a camera sensor located below the display, such as an LCD, LED, AMOLED (which is a particular example of an OLED display), or other display. While the exemplary OLED display is shown in FIGS. 4A and 4B with three layers, the OLED display may consist of more layers.

[0044]

[0054] In the example of FIG. 4A , OLED display 250 includes a cathode layer 252, an organic luminescent material layer 254, and an anode layer 256. For example, OLED display 250 can emit light when a current is passed between cathode layer 252 and anode layer 256 through organic luminescent material layer 254, causing an image or images to appear on OLED display 250. In this manner, OLED organic luminescent material layer 254 can emit light through cathode layer 252 toward a user. In some examples, camera sensor 234 can receive ambient light at the same time that organic luminescent material layer 254 can emit light. In the example of FIG. 4A , ambient light can strike a surface of cathode layer 252. A portion of this ambient light can pass through cathode layer 252, organic luminescent material layer 254, and anode layer 256. In this example, the cathode layer 252 and the anode layer 256 may not be transparent. Additionally, the organic luminescent material layer 254 may have RGB, RGBW (where W is white), WRGB (where W is white), RGBG (red-green-blue-green), or other subpixels that may block ambient light from passing through the organic luminescent material layer 254 or attenuate or distort the ambient light. Thus, the amount of ambient light that passes through the OLED display 250 may be relatively small (shown as transmitted light). Thus, the camera sensor 234 may not receive much of the transmitted light, as represented by the thin arrow. This may result in poor image quality of the image captured by the camera sensor.

[0045]

[0055] In the example of Figure 4B, OLED display 260 includes a transparent cathode layer 262, an organic luminescent material layer 264, and a transparent anode layer 266. As in the example of Figure 4A, OLED display 260 can emit light when a current is passed between transparent cathode layer 262 and transparent anode layer 266 through organic luminescent material layer 264, causing image(s) to appear on OLED display 260. In this manner, OLED organic luminescent material layer 264 can emit light through transparent cathode layer 262 toward the user. In the example of Figure 4B, because both transparent cathode layer 262 and transparent anode layer 266 are transparent, more of the ambient light can be transmitted through OLED display 260 to camera sensor 234. The camera sensor 234 may receive ambient light that passes through at least a portion of the OLED display 260 before entering the camera sensor 234 (shown as transmitted light, represented by a thick arrow). In some examples, the camera sensor 234 may receive ambient light at the same time that the organic luminescent material layer 264 may emit light. In the example of FIG. 4B , as in the example of FIG. 4A , the organic luminescent material layer 264 may include RGB, RGBW, or WRGB subpixels that may block, attenuate, or distort the ambient light passing through the organic luminescent material layer 264. Overall, the attenuation or distortion in the example of FIG. 4B may be less than that of FIG. 4A , for example, due to the transparency of the transparent cathode layer 262 and the transparent anode layer 266. However, image quality may still be undesirable due to the subpixels blocking, attenuating, or distorting the ambient light.

[0046]

[0056] 5 is a conceptual diagram illustrating an example technique for displaying content on an image capture device having an under-display camera sensor while the camera is on. When the camera is on, such as during image capture or while an image capture application (e.g., a camera application or a video application) is running, the image capture device may be turned off or not address some display pixels above the camera sensor to improve image capture quality and reduce interference that may come from displayed content above the camera sensor, for example. When the camera is off, those pixels may be used as display pixels. In some examples, when the image capture application is running on device 2, the camera is on and the image capture application is on. In some examples, when the image capture application is not running on device 2, the camera is off and the image capture application is off.

[0047]

[0057] In the example of Figure 5, one region, area, or zone of the display may actively use all of its pixels to display content, while another region, area, or zone of the display may actively use only some of its pixels to display content. Figure 5 shows two different views of an image that may be displayed, with each box in each view representing a pixel in the display. View 270 shows an image that may be displayed in a region of a display, such as display 232, that may not be located over a camera sensor, such as camera sensor 234. In Figure 2D, this region of the display is shown as region 232A. All of the pixels in region 232A may be utilized to display content, such as that shown in view 270.

[0048]

[0058] View 272 shows an image that may be displayed in a region of the display above a camera sensor, such as region 232B or region 232C in FIG. 2D . In view 272, a subset of pixels are actively displaying content. In some examples, region 232B may be the same size as camera sensor 234. In other examples, region 232B may be a different size than camera sensor 234. For example, region 232B may be larger than camera sensor 234, or region 232B may be smaller than camera sensor 234. In some examples, region 232C may be the same size as camera sensor 238. In other examples, region 232C may be a different size than camera sensor 238. For example, region 232C may be larger than camera sensor 238, or region 232C may be smaller than camera sensor 238. In the example view 272, the active pixel location that display 232 utilizes is the top left pixel (e.g., pixel 273) of every group of four pixels (e.g., group 274 separated from other groups of four pixels by a dashed white line) to display content, with the other pixels in each group of four pixels not utilized. The use of the top left pixel of every group of four pixels is provided as an example. Other configurations may be used.

[0049]

[0059] In some examples, view 272 may be displayed in region 232B above camera sensor 234, and view 270 may be displayed in region 232A, only when the camera is on (e.g., device 2 is running an image capture application, such as a camera application or a video application). In some examples, view 272 may be displayed in region 232B above camera sensor 234, and view 270 (e.g., using all pixels) may be displayed in region 232A, only when camera sensor 234 is actively being used to capture image(s). In some examples, view 272 may be displayed in region 232C above camera sensor 234, and view 270 may be displayed in region 232A, only when the camera is on (e.g., device 2 is running an image capture application, such as a camera application or a video application). In some examples, view 272 may be displayed in region 232C above camera sensor 238 only when camera sensor 238 is actively being used to capture image(s), and view 270 (e.g., using all pixels) may be displayed in region 232A. In some examples, view 270 (e.g., using all pixels) may be used in regions 232A, 232B, and 232C when the camera is off. In some examples, view 270 (e.g., using all pixels) may be used in regions 232A, 232B, and 232C when camera sensor 234 is not actively being used to capture image(s). For example, view 272 may be displayed in region 232B when an image capture application (e.g., a camera application or a video application) is being executed by device 2.

[0050]

[0060] It may be desirable for view 272 to be displayed in region 232B when camera sensor 234 is actively being used to capture image(s) to reduce light scattering and the resulting attenuation of ambient light caused by displayed content and that may be received by camera sensor 234. A subset of pixels in region 232B may thereby be active or addressed when the camera is on or when camera sensor 234 is actively capturing image(s). Similarly, a subset of pixels in region 232C may be active when the camera is on or when camera sensor 238 is actively capturing image(s). By reducing the number of pixels above camera sensor 234 (or camera sensor 238) that display content during image capture, the attenuation of light received by camera sensor 234 due to light scattering may be reduced. However, by reducing the number of pixels above the camera sensor 234, the resolution (e.g., number of active pixels per unit of area) of region 232B during image capture may be reduced compared to region 232A, so the image displayed in region 232B may appear blocky or lacking in detail, which may appear distracting. Because some of the display pixels above the camera sensor may be off or not addressed when the camera is on, the user experience related to displayed content in the area above the camera sensor may be relatively poor compared to other areas of the display where all pixels may be used to display content.

[0051]

[0061] According to the techniques of this disclosure, in some examples, device 2 may control one or more pixels in an area (e.g., region 232B) on a camera sensor (e.g., camera sensor 234) to actively display content during a camera sensor blanking period. In some examples, device 2 may control all of the pixels in an area (e.g., region 232B) on a camera sensor (e.g., camera sensor 234) to display content during a camera sensor blanking period, thereby facilitating better image quality in the displayed image.

[0052]

[0062] FIG. 6 is a timing diagram illustrating an example display frame and camera sensor blanking period in accordance with the techniques of this disclosure. FIG. 6 shows a display frame 300, an image capture event 302, an image readout event 304, and a camera sensor blanking period 310. In some examples, the content being displayed is an image captured by the camera sensor 234, such as when a user is taking a "selfie." In the example of FIG. 6, the length of the camera sensor blanking period 310 is less than the length of the display frame 300. The display frame 300 has a length that extends from a display frame start 312 to a display frame end 314. Thus, the display frame 300 represents the length of time that a frame may be displayed. For example, in a 30 frames per second (fps) display, the length of the display frame 300 may be equal to 1 / 30 of a second, or approximately 0.033 seconds. For example, in a 60 fps display, the length of a display frame 300 may be equal to 1 / 60th of a second or approximately 0.0167 seconds.

[0053]

[0063] In many instances, an image capture event may be shorter than the length of a display frame. During an image capture event 302, the camera sensor 234 may be capturing an image. In examples where the camera sensor 234 is a rolling shutter camera sensor, the camera sensor 234 may capture the image row by row over the time period indicated by the image capture event 302. In examples where the camera sensor 234 is a global shutter camera sensor, the camera sensor 234 may capture all rows of the image at once, in which case the length of time of the image capture event 302 may be significantly shorter than that shown in FIG. 6.

[0054]

[0064] During an image readout event 304, device 2 may be reading data from camera sensor 234 that is indicative of the image captured during image capture event 302. In some examples, the readout may be performed row-by-row, starting with the first row 306 and ending with the last row 308. In some examples, image readout event 304 may overlap in time with image capture event 302, as shown. In other examples, image readout event 304 may not overlap in time with image capture event 302. One such example may be when camera sensor 234 is a global shutter camera sensor.

[0055]

[0065] The image capture event 302 and the image readout event 304 may be completed before the end of the display frame 314. For example, the last row 308 may be read out at time 316, before the end of the display frame 314. This time between the end of the image readout event 304 at time 316 and the end of the display frame 314 may be referred to as the camera sensor blanking period 310. The camera sensor blanking period 310 may be a time during the display frame 300 when no images are being captured or read out.

[0056]

[0066] In some examples, unlike the example of FIG. 6 , the length of the camera sensor blanking period may not be less than the length of a display frame. For example, in extremely bright light conditions, for a camera at 30 frames per second (33.33 ms frame time) and a 10 ms exposure time with a display rate of 60 frames per second (16.67 ms frame time), the camera sensor blanking period may be 23 ms long. In this example, the camera integration duty cycle may be 10 ms (exposure time) / 33.33 ms (frame time), which is 30%. Two display frames may fit within the camera frame time. Of the two display frames, the camera sensor 234 may be on for 70% of the time, or 1.4 display frames. In some such examples, the pixel value may be enhanced by a factor of 2 / 1.4 or 1.42 (e.g., the second level of brightness may be 1.42 times the first level of brightness). In this example, the display pixel value that may be enhanced may be the last display pixel value for that pixel or an average value over two display frames. In some examples, the camera sensor blanking period may not vary with the auto-exposure parameters, as some auto-exposure parameters may vary without varying the exposure time.

[0057]

[0067] In another example, in low light conditions, the camera frame rate may be 15 frames per second, and the camera frame time may be 66.67 ms. In this example, the camera sensor blanking period may be small, such as 2 ms, and the exposure time may be 64.67 ms. The camera integration duty cycle may be 64.67 ms (exposure time) / 66.67 ms (frame time), or 97%. In this example, there may be four display frames that fit within a given camera frame time. Of the four display frames, a display pixel may be on or addressed for only 3% of the time, or 0.12 display frames. In this example, the pixel value may be enhanced by 4 / 0.12, or 33.33 times (e.g., a second level of brightness may be 33.33 times the first level of brightness). In this example, the display pixel value that may be enhanced may be the last display pixel value for that pixel or an average value over four display frames.

[0058]

[0068] 7 is a timing diagram illustrating an example display frame and camera sensor blanking period in accordance with the techniques of this disclosure. FIG. 7 shows a display frame 300, an image capture event 302, an image readout event 304, and a camera sensor blanking period 280. In the example of FIG. 7, the camera sensor blanking period 280 overlaps with the image readout event 304. The example of FIG. 7 is otherwise similar to the example of FIG. 6. During the camera sensor blanking period 280, pixels in region 232B may display pixel values ​​of a second level of brightness in accordance with the techniques of this disclosure.

[0059]

[0069] 8 is a timing diagram illustrating an example display frame, buffer delta(s), and camera sensor blanking period in accordance with techniques of this disclosure. In some examples, a buffer delta, such as buffer delta 317A or buffer delta 317B, may be used in conjunction with a camera sensor blanking period to provide separation between the camera sensor blanking period, such as camera sensor blanking period 319, and the image readout event 304 and / or the next image capture event 318. Thus, the camera sensor blanking period 319 in FIG. 8 may be shorter than the camera sensor blanking period 310 in FIG. 6 or the camera sensor blanking period 280 in FIG. 7.

[0060]

[0070] For example, if the current exposure time is 10 ms and the frame time is 33 ms, the blanking time may be equal to the frame time - the exposure time - the buffer delta. If a 3 ms buffer delta is used, the camera sensor blanking period may be equal to 33 ms - 10 ms - 3 ms = 20 ms. In some examples, the buffer delta, e.g., buffer delta 317A, may be immediately after the last row 308 is read (time 316). In one example, the buffer delta, e.g., buffer delta 317B, may be immediately before the start of the next image capture event 318 (e.g., at the time of the end of the display frame 314). In some examples, the buffer delta may be split between just after the last row 308 is read (time 316) and just before the start of the next image capture event 318 (e.g., at the time of the end of the display frame 314), or two buffer deltas may be used, one just after the last row 308 is read (time 316), e.g., buffer delta 317A, and one just before the start of the next image capture event 318 (e.g., at the time of the end of the display frame 314), e.g., buffer delta 317B. Buffer deltas such as buffer delta 317A or buffer delta 317B may provide some protection against latency between image capture and readout events of the camera sensor 234 and the display of content on pixels in region 232B of the display 232 on the camera sensor 234. This latency may, in some cases, cause the display of content at pixels in region 232B while the image capture or readout event is occurring.

[0061]

[0071] 9A-9B are conceptual diagrams illustrating an example display technique according to the present disclosure. In FIG. 9A, the horizontal axis represents time, and the vertical axis represents brightness levels. Display pixels in an area, such as region 232B, on a camera sensor, such as camera sensor 234, are on during time 320, which may be camera sensor blanking period 340 or a portion of a camera sensor blanking period. During integration time 322, the pixels are off or not addressed (e.g., not displaying content). For example, device 2 may not display a portion of content at the pixels outside of the camera sensor blanking period (e.g., outside camera sensor blanking period 340 and camera sensor blanking period 342). In some examples, integration time 322 includes time during which camera sensor 234 is actively capturing an image. In some examples, integration time 322 includes time during which camera sensor 234 is actively capturing an image and time during which device 2 is reading data from camera sensor 234. In some examples, integration time 322 includes the time during which camera sensor 234 is actively capturing an image, the time during which device 2 is reading data from camera sensor 234, and one or more buffer deltas (e.g., buffer delta 317A or buffer delta 317B). Because the pixels are off or not addressed while camera sensor 234 is actively capturing an image, device 2 can avoid or mitigate optical scattering and resulting attenuation of received light caused by displayed content at the pixels. After integration time 322, the pixels may be turned on or addressed again during time 324, which may be camera sensor blanking period 342 or a portion of a camera sensor blanking period.

[0062]

[0072] In some examples, each of the pixels in an area on the camera sensor, such as region 232B, may be on or addressed during times 320 and 324, and off or unaddressed during integration time 322. In this manner, the pixels in region 232B on the camera sensor 234 may be off or unaddressed while the camera sensor 234 is actively capturing images, thereby reducing or avoiding optical scattering and resulting attenuation of received light caused by displayed content.

[0063]

[0073] 9A, the level of brightness of the pixel when it is on or addressed is equal to the first level of brightness for the pixel 326. For example, if the first level of brightness is a gray value of 30, then when the pixel is on or addressed during time 320, the pixel displays a level of brightness of the gray value of 30.

[0064]

[0074] The human eye may act like a low-pass filter. For example, fluorescent lights flicker during normal use, but humans may not perceive the flickering. Instead, humans may perceive a constant level of brightness that may be equal to the average level of brightness between the fluorescent light being on and the fluorescent light being off. Thus, a user of device 2 in the example of FIG. 9A may perceive the brightness of a pixel to be perceived brightness 328. That is, the user may not perceive first level of brightness 326, but may instead perceive an average level of brightness. This average level of brightness, perceived brightness 328, may be equal to or approximately equal to the average of the pixel's brightness when the pixel is on or addressed during a given display frame, and may be equal to or approximately equal to 0 when the pixel is off or not addressed. Thus, in the example of FIG. 8A, region 232B may not appear brighter than region 232A, which may be distracting or aesthetically pleasing to a user of device 2.

[0065]

[0075] According to the techniques of this disclosure, device 2 may increase pixel brightness during camera sensor blanking periods to maintain an average brightness level equal to or approximately equal to a first level of brightness that the pixel would have if the camera were off. Thus, the average brightness level may be the same as or close to the brightness levels of the surrounding pixels.

[0066]

[0076] 9B , the horizontal axis represents time and the vertical axis represents brightness levels. Display pixels in an area, such as region 232B, on a camera sensor, such as camera sensor 234, are turned on or addressed during time 330, which may be camera sensor blanking period 350 or a portion of a camera sensor blanking period. During integration time 332, the pixels are off or not addressed (e.g., not displaying content). For example, device 2 may not display a portion of an image at the pixels outside of a camera sensor blanking period (e.g., camera sensor blanking period 350 and camera sensor blanking period 352). In some examples, integration time 332 includes time during which camera sensor 234 is actively capturing an image. In some examples, integration time 332 includes time during which camera sensor 234 is actively capturing an image and time during which device 2 is reading data from camera sensor 234. In some examples, integration time 332 includes the time during which camera sensor 234 is actively capturing images, the time during which device 2 is reading data from camera sensor 234, and one or more buffer deltas (e.g., buffer delta 317A or buffer delta 317B).

[0067]

[0077] In some examples, each of the pixels in an area on the camera sensor, such as region 232B, may be on or addressed during times 330 and 334, and off or unaddressed during integration time 332. In this manner, the pixels in region 232B on the camera sensor 234 may be off or unaddressed while the camera sensor 234 is actively capturing images, thereby reducing or avoiding optical scattering and resulting attenuation of received light caused by displayed content.

[0068]

[0078] In the example of FIG. 9B , the luminance level of the pixel above the camera sensor is at a second level of luminance 337. The second level of luminance 337 is higher than the first level of luminance 326 in the example of FIG. 9A . In some examples, the second level of luminance 337 for the pixel above the camera sensor may be determined such that the first level of luminance 336 (e.g., the gray value assigned for the pixel) may be equal to or approximately equal to the perceived luminance 338. For example, device 2 may determine the second level of luminance for the pixel.

[0069]

[0079] Information used to implement the techniques of this disclosure may be readily available to device 2. For example, the start of a frame, the end of a frame, exposure time, etc. may be readily available. Information related to camera sensor 234, such as the start of an image capture event, the end of an image capture event, the start of an image readout event, the end of an image readout event, and the next frame exposure time, may also be readily available to device 2. In some examples, camera sensor 234 may output an on / off signal on a pin, which a display controller (which may be an example of one or more processors 110) may treat as an event to control. For example, an on signal may indicate a camera sensor blanking period, and an off signal may indicate an integration time.

[0070]

[0080] For example, device 2 may have information related to the exposure time of camera sensor 234 (e.g., from AEC process 20 of FIG. 1). Device 2 may also have information related to the length of display frame 300 (FIGS. 6-8). From this information, device 2 may determine the length of a camera sensor blanking period (e.g., camera sensor blanking period 310 of FIG. 6, camera sensor blanking period 280 of FIG. 7, or camera sensor blanking period 319 of FIG. 8). One or more processors of device 2 may use the length of the camera sensor blanking period length and the length of the display frame to determine what percentage of time during the display frame the pixel is on or addressed (e.g., to determine the duty cycle of the pixel during the display frame). For example, the pixel may be on or addressed 50% (0.5) of the time. Device 2 may also have information regarding what the first level of luminance is for the pixel, e.g., a gray value of 30. The one or more processors of device 2 may determine a second level of brightness to use for a pixel by dividing the first level of brightness by the percentage of time the pixel is on or addressed during a display frame. For example, the one or more processors of device 2 may determine the second level of brightness to be a gray value of 60 by dividing the first level of brightness for the pixel (a gray value of 30) by the percentage of time the pixel is on or addressed during a display frame (0.5).

[0071]

[0081] In another example, a pixel is on or addressed 25% (0.25) of the time during a display frame, and the first level of brightness is a gray value of 30. In this example, one or more processors of device 2 may determine a second level of brightness for the pixel to be a gray value of 120 by dividing the gray value of 30 by the percentage of time that the pixel is on or addressed during a display frame, 0.25. In this way, perceived brightness 338 perceived by the user may be equal to or approximately equal to first level of brightness 336 of the portion of the image to be displayed by the pixel. In some examples, one or more processors of device 2 may determine a second level of brightness for each of the pixels above the camera sensor (e.g., in region 232B or region 232C of FIG. 2D ). Thus, the techniques of this disclosure may not only avoid or mitigate attenuation of light received by the camera sensor 234 due to light scattering caused by the displayed content, but may also enable the display of content in a display frame when the camera is on, which may appear the same or nearly the same as it would if the same content were to be displayed in the display frame when the camera is off.

[0072]

[0082] In some examples, one or more processors of device 2 may determine the second level of brightness using the following equation:

[0073]

number

[0074] where SB is the second level of brightness, DF is the length of the display frame, BP is the length of the camera sensor blanking period, and FB is the first level of brightness. In some examples, one or more processors of device 2 may determine the second level of brightness using the following equation:

[0075]

number

[0076] where SB is the second level of brightness, DF is the length of the display frame, BP is the length of the camera sensor blanking period, FB is the first level of brightness, and W is a weighting coefficient. In some examples, one or more processors of device 2 may use another calculation to determine the second level of brightness. For example, one or more processors of device 2 may use a Gaussian function to determine the second level of brightness.

[0077]

[0083] 10 is a block diagram of an exemplary image capture device that may implement the techniques of this disclosure. Image capture device 102 may be an example of, and may substantially conform to, device 2 of FIG. 1. By way of example, image capture device 102 may comprise a wireless mobile communications device, such as a cellular or satellite radiotelephone, a smartphone, a standalone digital camera or video camcorder, a personal digital assistant (PDA), a tablet computer, a laptop computer, or any device with imaging or video capabilities in which image processing is desirable.

[0078]

[0084] The image capture device 102 may include one or more processors 110, a camera sensor 112, an image signal processor 106, a memory 114, a display 116, a communication circuit 118, an ambient light sensor 122, and a proximity sensor 124. The display 116 may include an area 120 (which may be an example of area 232B or area 232C in FIG. 2D ) disposed over the camera sensor 112. In some examples, one or both of the ambient light sensor 122 or the proximity sensor 124 may be disposed below the display 116. In some examples, one or both of the ambient light sensor 122 or the proximity sensor 124 may be disposed below area 120. In some examples, one or both of the ambient light sensor 122 and the proximity sensor may be utilized by the image signal processor 106 or one or more processors 110 to determine an exposure time for the camera sensor 112 or to determine autofocus for the camera sensor 234. In some examples, techniques of this disclosure described with respect to a camera sensor, such as camera sensor 112, may be used for ambient light sensor 122 or proximity sensor 124. For example, ambient light sensor 122 may sense the amount of ambient light while display 116 is not displaying content. Similarly, proximity sensor 124 may emit infrared light while display 116 is not displaying content. In some examples, image signal processor 106 may be one of one or more processors 110. In some examples, image signal processor 106 may be two or more of one or more processors. In some examples, image signal processor 106 may be separate from one or more processors 110, as shown.

[0079]

[0085] The memory 114 may include an image capture application (image capture app) 104 for execution by the one or more processors 110. The image capture application 104 may be utilized by a user to turn on the camera function of the image capture device 102. The memory 114 may also be configured to store pixel values ​​related to images captured by the camera sensor 112. The memory 114 may also store instructions for causing the one or more processors 110 to perform the techniques of this disclosure.

[0080]

[0086] Camera sensor 112 may capture an image during an image capture event, such as image capture event 302 of FIGS. 6-7. For example, camera sensor 112 may capture pixel values ​​when capturing an image. In some examples, the captured values ​​may include a level of luminance, or the captured values ​​may be convertible to include a level of luminance. Image signal processor 106 may process the captured or converted values ​​as described above with respect to image signal processor 6 of FIG. 1. The level of luminance for a given pixel, either captured by camera sensor 112, processed by image signal processor 106, or possibly received by image capture device 102, may be a first level of luminance.

[0081]

[0087] The one or more processors 110 may retrieve pixel values ​​from the image signal processor 106 and provide the pixel values ​​to the memory 114 for storage, to the communication circuitry 118 for transmission to another device, and / or to the display 116 for display. When the camera is off, the one or more processors 110 may, for example, retrieve pixel values ​​from the memory 114 and provide the pixel values ​​to the display 116 for display.

[0082]

[0088] When the camera is on, the one or more processors 110 may enhance a first level of brightness for pixels in region 120. For example, the one or more processors 110 may determine a first level of brightness for pixels located in a region of the display above the camera sensor. The one or more processors 110 may determine a length of a camera sensor blanking period. The one or more processors 110 may determine a length of a display frame. The one or more processors 110 may determine a second level of brightness based on the first level of brightness, the length of the camera sensor blanking period, and the length of the display frame. The one or more processors 110 may display a portion of the image at the pixels at the second level of brightness during the camera sensor blanking period.

[0083]

[0089] The one or more processors 110 may control the display 116 so that when the camera is on, the pixels in the region 120 are only used to display content during camera sensor blanking periods. When the camera is off, the one or more processors 110 may control the display 116 to display content in pixels that include the pixels of the region 120 during an entire frame.

[0084]

[0090] 11 is a flowchart illustrating an example technique for displaying content in an image capture device having an under-display camera sensor according to this disclosure. One or more processors 110 may determine a camera sensor blanking period (e.g., camera sensor blanking period 310), where the camera sensor blanking period is associated with a camera sensor configured to receive light through at least a portion (e.g., region 120) of display 116 (150). For example, one or more processors 110 may determine a camera sensor blanking period to be a time when a camera or image capture application is on (e.g., image capture application 104 is being executed by one or more processors 110) and camera sensor 112 is not capturing an image. In another example, one or more processors 110 may determine a camera sensor blanking period to be a time when a camera or image capture application is on (e.g., image capture application 104 is being executed by one or more processors 110), camera sensor 112 is not capturing an image, and one or more processors 110 are not reading data from camera sensor 112. In some examples, the one or more processors 110 may determine the camera sensor blanking period such that the camera sensor blanking period does not include time in a buffer delta (e.g., buffer delta 317A or buffer delta 317B).

[0085]

[0091] The one or more processors 110 may control the display 116 to display content via one or more of a plurality of pixels in at least a portion (e.g., region 120) of the display 116 during a camera sensor blanking period (e.g., camera sensor blanking period 310) (152). For example, the region 120 of the display 116 may display content during the camera sensor blanking period, such as an image captured by the image capture device 102, a visual representation of a file stored in a memory location, a software application, a user interface including a GUI, a network-accessible content object, an image received but not captured by the image capture device (e.g., a downloaded image, an image sent by text message, an image sent by email, an image sent through an application, etc.), a background image, and other information.

[0086]

[0092] The one or more processors 110 may control the display 116 to refrain from displaying content via one or more of the pixels in the region 120 outside of a camera sensor blanking period (154). For example, one or more of the pixels in the region 120 may not actively display content outside of a camera sensor blanking period. In some examples, the one or more processors 110 may control the display 116 to cease displaying content via one or more of the pixels in the region 120 at the end of a camera sensor blanking period. In some examples, the display may not display content via one or more of the pixels in the region 120 until the start of another camera sensor blanking period. In some examples, the camera sensor blanking period is a first camera sensor blanking period, and the one or more processors 110 may determine a second camera sensor blanking period and control the display 116 to display content via one or more of the plurality of pixels in the region 120 during the second camera sensor blanking period.

[0087]

[0093] In some examples, the one or more processors 110 may determine a first level of brightness for one or more of the plurality of pixels in region 120. For example, the one or more processors 110 may determine the first level of brightness for one or more of the plurality of pixels by reading a level of brightness value for each of one or more of the plurality of pixels from camera sensor 112. In another example, the one or more processors 110 may determine the first level of brightness for one or more of the plurality of pixels by reading a level of brightness value for each of one or more of the plurality of pixels from image signal processor 106. In another example, the one or more processors 110 may determine the first level of brightness for one or more of the plurality of pixels by reading a level of brightness for each of one or more of the plurality of pixels from memory 114. In another example, the one or more processors 110 may determine the first level of brightness for one or more of the plurality of pixels by determining a level of brightness for pixels outside region 120. In yet another example, the one or more processors 110 may determine the first level of brightness for one or more of the plurality of pixels by determining an average level of brightness for the plurality of pixels outside the region 120. In some examples, the first level of brightness is a level of brightness displayed by the image capture device when the camera or image capture application of the image capture device is off.

[0088]

[0094] In some examples, the one or more processors 110 may determine a second level of luminance of one or more of the plurality of pixels in the region 120. In some examples, the second level of luminance is greater than the first level of luminance and is based at least in part on the first level of luminance. For example, the one or more processors 110 may determine a length of a camera sensor blanking period and a length of a display frame of content. For example, the one or more processors 110 may have information related to an exposure time of the camera sensor 112 and information related to the length of a display frame and may use this information to determine the length of the camera sensor blanking period. In some examples, the second level of luminance is further based on the length of the camera sensor blanking period and the length of the display frame. In some examples, the second level of luminance is based on the display content and the first level of luminance. In some examples, the length of the camera sensor blanking period is less than the length of the display frame.

[0089]

[0095] In some examples, as part of determining the second level of brightness, the one or more processors 110 apply a formula to the first level of brightness. In some examples, the formula comprises a Gaussian function. In some examples, the formula is

[0090]

number

[0091] where SB is the second level of brightness, DF is the length of the display frame, BP is the length of the camera sensor blanking period, and FB is the first level of brightness. In some examples, the formula may be:

[0092]

number

[0093] where SB is the second level of brightness, DF is the length of the display frame, BP is the length of the camera sensor blanking period, FB is the first level of brightness, and W is a weighting factor.

[0094]

[0096] In some examples, the one or more processors 110 may control the display 116 to display content via one or more of the plurality of pixels at a second level of brightness during a camera sensor blanking period. For example, one or more of the plurality of pixels in the region 120 may display content at the second level of brightness. In some examples, the second level of brightness is greater than the first level of brightness and is based at least in part on the first level of brightness.

[0095]

[0097] In some examples, the content is at least a portion of an image captured by a camera sensor 112. In some examples, the image capture device 102 is a wireless mobile communication device, such as a smartphone or cellular phone. In some examples, the image capture device 102 includes a camera sensor 112 and a display 116.

[0096]

[0098] Thus, by determining a second level of brightness and applying the second level of brightness to pixels during a camera sensor blanking period, a camera sensor may be placed below the display to attempt to maximize the display size on the image capture device without particularly introducing image quality issues. By placing the camera sensor below the screen, the screen size of the image capture device may be larger than an image capture device of the same size that uses a notched screen, and the reliability of the image capture device may be improved over the reliability of an image capture device that uses a pop-up camera with moving mechanical parts.

[0097]

[0099] In accordance with the above examples, it should be recognized that some acts or events of any of the techniques described herein may be performed in a different sequence, added, merged, or entirely excluded (e.g., not all described acts or events may be required to practice the techniques). Moreover, in some examples, acts or events may be performed simultaneously rather than sequentially, for example, through multithreaded processing, interrupt processing, or multiple processors.

[0098]

[0100] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media such as data storage media, or communication media, including any medium that enables transfer of a computer program from one place to another, for example, according to a communications protocol. In this manner, computer-readable media may generally correspond to (1) tangible computer-readable storage media that is non-transitory, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0099]

[0101] By way of example, and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but instead cover non-transitory, tangible storage media. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy discs, and Blu-ray discs, where disks typically reproduce data magnetically and discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0100]

[0102] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the terms "processor" and "processing circuitry" as used herein may refer to any of the above structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Also, the techniques may be fully implemented in one or more circuits or logic elements.

[0101]

[0103] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Although various components, modules, or units have been described in this disclosure to highlight functional aspects of devices configured to implement the disclosed techniques, those components, modules, or units do not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit or provided by a collection of interoperable hardware units, including one or more processors described above, along with suitable software and / or firmware.

[0102]

[0104] Various examples have been described. These and other examples are within the scope of the following claims. The inventions described in the claims of the present application as originally filed are set forth below. [C1] 1. An image capture device, comprising: a memory configured to store display content and image information received from a camera sensor, the camera sensor configured to receive light through at least a portion of the display; one or more processors coupled to the memory; wherein the one or more processors: determining a camera sensor blanking period; controlling the display to display content via one or more of a plurality of pixels in the at least a portion of the display during the camera sensor blanking period; and controlling the display to not display content via the one or more of the plurality of pixels outside of the camera sensor blanking period. [C2] the one or more processors: controlling the display to cease displaying content via the one or more of the plurality of pixels at an end of the camera sensor blanking period. 3. The image capture device of claim 1, further configured to: [C3] the camera sensor blanking period is a first camera sensor blanking period, and one or more processors: determining a second camera sensor blanking period; controlling the display to display content via the one or more of the plurality of pixels during the second camera sensor blanking period; 3. The image capture device of claim 2, further configured to: [C4] the one or more processors: determining a first level of luminance of the one or more of the plurality of pixels; determining a second level of luminance of the one or more of the plurality of pixels; controlling the display to display content via the one or more of the plurality of pixels at the second level of brightness during the camera sensor blanking period; further configured to: wherein the second level of luminance is greater than the first level of luminance and is based at least in part on the first level of luminance. 10. An image capture device as described in C1. [C5] The image capture device of C4, wherein the first level of brightness is a level of brightness that the image capture device displays when an image capture application of the image capture device is off. [C6] As part of determining the second level of luminance, the one or more processors: determining a length of the camera sensor blanking period; determining a display frame length of the content; configured to: wherein the second level of brightness is further based on the length of the camera sensor blanking period and the length of the display frame. 3. An image capture device as described in C5. [C7] The image capture device of C6, wherein the length of the camera sensor blanking period is less than the length of the display frame. [C8] The image capture device of C6, wherein the second level of brightness is based on the displayed content and the first level of brightness. [C9] The image capture device of C6, wherein as part of determining the second level of luminance, the one or more processors apply a Gaussian function to the first level of luminance. [C10] the one or more processors:

number

number

number

number

Claims

1. 1. An image capture device, comprising: The camera sensor and The display and a memory configured to store display content and image information received from the camera sensor, the camera sensor configured to receive light through at least a portion of the display; one or more processors coupled to the memory; wherein the one or more processors: determining a camera sensor blanking period; determining a first level of luminance for one or more of a first plurality of pixels outside the portion of the display where the camera sensor receives light, wherein the first level of luminance is a level of luminance displayed by the image capture device when an image capture application of the image capture device is off; determining a second level of luminance for one or more of a second plurality of pixels in the portion of the display from which the camera sensor receives light, wherein as part of determining the second level of luminance, the one or more processors: determining a length of the camera sensor blanking period; determining a display frame length of the content; configured to: wherein the second level of luminance is greater than the first level of luminance and is based on at least the first level of luminance, the length of the camera sensor blanking period, and the length of the display frame. controlling the display to display content via the one or more of the second plurality of pixels at the second level of luminance during the camera sensor blanking period; and controlling the display to not display content via the one or more of the second plurality of pixels outside of the camera sensor blanking period.

2. the one or more processors: controlling the display to cease displaying content via the one or more of the plurality of pixels at the end of the camera sensor blanking period. The image capture device of claim 1 , further configured to:

3. the camera sensor blanking period is a first camera sensor blanking period, and one or more processors: determining a second camera sensor blanking period; controlling the display to display content via the one or more of the plurality of pixels during the second camera sensor blanking period; The image capture device of claim 2 , further configured to:

4. The image capture device of claim 1 , wherein the length of the camera sensor blanking period is less than the length of the display frame.

5. The image capture device of claim 1 , wherein the second level of brightness is based on the displayed content and the first level of brightness.

6. The image capture device of claim 1 , wherein as part of determining the second level of luminance, the one or more processors apply a Gaussian function to the first level of luminance.

7. the one or more processors: [Equation 1] 2. The image capture device of claim 1, wherein SB is the second level of luminance, DF is a length of a display frame, BP is a length of the camera sensor blanking period, and FB is the first level of luminance.

8. the one or more processors: [Equation 2] 2. The image capture device of claim 1, wherein SB is the second level of luminance, DF is a length of a display frame, BP is a length of the camera sensor blanking period, FB is the first level of luminance, and W is a weighting factor.

9. The image capture device of claim 1 , wherein the camera sensor blanking period comprises a time when an image capture application is on and the camera sensor is not capturing images.

10. The image capture device of claim 9 , wherein the camera sensor blanking period further comprises a time during which the one or more processors are not reading data from the camera sensor.

11. The image capture device of claim 1 , wherein the content is at least a portion of an image captured by the camera sensor.

12. 1. A method for displaying content on an image capture device, the method comprising: determining a camera sensor blanking period, the camera sensor blanking period associated with a camera sensor configured to receive light through at least a portion of a display; determining a first level of luminance for one or more of a first plurality of pixels outside the portion of the display where the camera sensor receives light, wherein the first level of luminance is a level of luminance displayed by the image capture device when an image capture application of the image capture device is off; determining a second level of luminance for the one or more of a second plurality of pixels in the portion of the display from which the camera sensor receives light, wherein determining the second level of luminance includes: determining a length of the camera sensor blanking period; determining a display frame length of the content; Equipped with wherein the second level of luminance is greater than the first level of luminance and is based on at least the first level of luminance, the length of the camera sensor blanking period, and the length of the display frame. displaying content via one or more of the second plurality of pixels at the second level of luminance during the camera sensor blanking period; refraining from displaying content via the one or more of the second plurality of pixels outside of the camera sensor blanking period; and A method comprising:

13. ceasing to display content via the one or more of the plurality of pixels at the end of the camera sensor blanking period. The method of claim 12 further comprising:

14. A non-transitory computer-readable storage medium storing a program, the program, when executed, causing one or more processors to: determining a camera sensor blanking period, the camera sensor blanking period associated with a camera sensor configured to receive light through at least a portion of a display; determining a first level of luminance for one or more of a first plurality of pixels outside the portion of the display where the camera sensor receives light, wherein the first level of luminance is a level of luminance displayed by the image capture device when an image capture application of the image capture device is off; determining a second level of luminance for one or more of a second plurality of pixels in the portion of the display from which the camera sensor receives light, wherein as part of determining the second level of luminance, the one or more processors: determining a length of the camera sensor blanking period; determining a display frame length for the content; configured to: wherein the second level of luminance is greater than the first level of luminance and is based on at least the first level of luminance, the length of the camera sensor blanking period, and the length of the display frame. controlling the display to display content via the one or more of the second plurality of pixels at the second level of luminance during the camera sensor blanking period; controlling the display not to display content via the one or more of the second plurality of pixels outside of the camera sensor blanking period; A non-transitory computer-readable storage medium storing a program that causes the computer to perform the above.

Images