Under-display camera and sensor control

Abstract

An under-display camera is positioned below the display of the mobile device. The under-display camera captures an image using light passing through a portion of the display. The mobile device displays a display image on the display, where the display image is based on the image. The mobile device displays an indicator overlaid on the display image on an indicator area of ​​the display overlapping the portion of the display. The indicator may identify the location of the camera. If a more recently captured image includes an occlusion, the mobile device can compensate for the camera occlusion by continuing to display the previous display image. The mobile device can provide a user with alternative ways to select an area of ​​the display image, for example, using hardware buttons and / or touchscreen interface elements, to avoid camera occlusion.

Description

Technical Field

[0001]

[0001] This application relates to image capture and image processing. More particularly, this application relates to techniques and methods for controlling an under-display camera and / or an under-display sensor and receiving user input for compensating for occlusion of the under-display camera and / or the under-display sensor.

Background Art

[0002]

[0002] Some mobile devices include a front-facing camera that faces the user while the user holds the mobile device. The front-facing camera can be adjacent to the display screen of the mobile device. This enables the mobile device to use the front-facing camera and the adjacent display screen, for example, for video conferencing. Since mobile devices are generally small for portability, the front-facing camera is often located within the bezels around the display screen. Some mobile devices include a display screen with a notch or cutout, within which the display cannot display anything and the front-facing camera is placed. When the front-facing camera is placed in a bezel, notch, or cutout, the user can generally see the location where the front-facing camera is placed. However, bezels, notches, and cutouts occupy valuable area on the front of the mobile device, preventing the mobile device from having a display screen that occupies all of the front of the mobile device.

Summary of the Invention

[0003]

[0003] This specification describes systems and techniques for using an under-display camera and / or under-display sensor positioned beneath the display of a device (for example, a mobile device or other device). The device may display an indicator on the display showing the location of the under-display camera and / or under-display sensor. The device may compensate for occlusion of the under-display camera and / or under-display sensor. For example, such occlusion may be caused by a user or input device touching the display in a way that obstructs the under-display camera and / or under-display sensor, a user or input device hovering over the display in a way that obstructs the under-display camera and / or under-display sensor, or a user or input device being otherwise close enough to the display to obstruct the under-display camera and / or under-display sensor. The device may control the under-display camera and / or under-display sensor based on input. The device may include user interface (UI) elements that allow selection of an area of ​​an image without, for example, touching, hovering over, or otherwise covering the area of ​​the image, where such input in the area of ​​the image would obstruct an under-display camera and / or under-display sensor.

[0004]

[0004] In one example, an apparatus for image processing is provided. The apparatus includes memory. The apparatus includes one or more processors coupled to the memory and configured to carry out a method. The method includes receiving an image captured by a camera based on incident light received by the camera. The camera is positioned relative to the display to receive incident light that passes through a portion of the display before reaching the camera. The method includes displaying a display image based on the image on the display. The method includes displaying an indicator overlaid on the display image in an indicator area of ​​the display while the display is displaying the display image. The indicator area includes at least a subset of the portion of the display.

[0005]

[0005] In another example, an apparatus for image processing is provided. The apparatus includes one or more memory units for storing instructions. The apparatus includes one or more processors for executing instructions, wherein the execution of instructions by one or more processors causes one or more processors to perform a method. The method includes receiving an image captured by a camera based on incident light received by the camera. The camera is positioned relative to the display to receive incident light that passes through a portion of the display before reaching the camera. The method includes displaying a display image based on the image on the display. The method includes displaying indicators overlaid on the display image in an indicator area of ​​the display while the display is displaying the display image. The indicator area includes at least a subset of the portion of the display.

[0006]

[0006] In another example, a method for image processing is provided. The method includes receiving image data captured by an image sensor. The method includes receiving an image captured by a camera based on incident light received by a camera. The camera is positioned relative to the display to receive incident light that passes through a portion of the display before reaching the camera. The method includes displaying a display image based on the image on the display. The method includes displaying an indicator overlaid on the display image in an indicator area of ​​the display while the display is displaying the display image. The indicator area includes at least a subset of the portion of the display.

[0007]

[0007] In another example, a non-temporary computer-readable storage medium comprising a program is provided. The program is executable by a processor to perform a method of image processing. The method includes receiving an image captured by a camera based on incident light received by the camera. The camera is positioned relative to the display to receive incident light that passes through a portion of the display before reaching the camera. The method includes displaying a display image based on the image on the display. The method includes displaying indicators overlaid on the display image in an indicator area of ​​the display while the display is displaying the display image. The indicator area includes at least a subset of the portion of the display.

[0008]

[0008] In another example, an apparatus for image processing is provided. The apparatus includes memory. The apparatus includes means for receiving an image captured by a camera based on incident light received by the camera. The camera is positioned relative to the display to receive incident light that passes through a portion of the display before reaching the camera. The apparatus includes means for displaying a display image based on the image on the display. The apparatus includes means for displaying an indicator overlaid on the display image in an indicator area of ​​the display while the display is displaying the display image. The indicator area includes at least a subset of the portion of the display.

[0009]

[0009] In another example, a method for image processing is provided. The method includes receiving an image captured by a camera. The camera is positioned below an area of ​​the display and captures the image using light passing through the display. The method includes displaying the image on the display. The method includes setting image processing properties to specific settings determined based on an area of ​​the display. The image processing properties are one of the following: white balance, black level compensation, gamma correction, gain, color correction, color saturation, noise filter, contrast control, brightness control, tone mapping, sharpness, blur, or red-eye correction.

[0010]

[0010] In another example, a method for image processing is provided. The method includes receiving an image captured by a camera. The camera captures an image based on incident light received by the camera. The camera is positioned relative to the touchscreen display to receive incident light that passes through a portion of the touchscreen display before reaching the camera. The method includes displaying a display image based on the image on the touchscreen. The method includes displaying an area selection user interface (UI) element on the touchscreen. The method includes receiving touch input to the touchscreen display in an area selection input area that overlaps with the area selection UI element. The method includes selecting a selection area of ​​the display image in response to receiving the touch input. The method includes setting the camera properties of the camera to a setting determined based on the selection area of ​​the display image in response to the selection of the selection area, wherein the camera properties of the camera include at least one of image capture properties or image processing properties.

[0011]

[0011] In some embodiments, the indicator area of ​​the display is at least as large as the camera lens. In some embodiments, the indicator is at least as large as the camera lens.

[0012]

[0012] In some embodiments, the methods, apparatus, and computer-readable media described above further comprises receiving an input, wherein an indicator is displayed overlaid on an indicator area in response to the reception of an input. In some embodiments, the input includes at least one of the following: a touch on the touch-sensitive touchscreen surface of the display, a predefined gesture across the touch-sensitive touchscreen surface of the display, a hover on the display, a predefined gesture on the display, or voice input of speaking a predefined phrase recorded by a microphone.

[0013]

[0013] In some embodiments, the methods, apparatus, and computer-readable media described above further comprise generating a display image by modifying the size of the image using at least one of scaling and cropping. In some embodiments, modifying the size of the image includes modifying the size of the image so that the display image is displayed on a display other than the indicator area.

[0014]

[0014] In some embodiments, the methods, apparatus, and computer-readable media described above further comprise generating a display image by processing an image, wherein processing an image includes modifying at least one of luminance, contrast, color saturation, tone, white balance, black balance, levels, gamma correction, gain, color correction, noise filter, sharpness, blur, and red-eye correction.

[0015]

[0015] In some embodiments, the camera is positioned adjacent to the display along an axis perpendicular to the flat surface of the display. In some embodiments, the indicator includes a shape around at least a portion of the display. In some embodiments, displaying the indicator includes displaying a color in multiple pixels in the indicator area. In some embodiments, the color is black. In some embodiments, the indicator is animated.

[0016]

[0016] In some embodiments, displaying indicators overlaid on the display image in an indicator area while the display is displaying the display image includes modifying the display image to merge the indicators with the display image. In some embodiments, the display is a touchscreen.

[0017]

[0017] In some embodiments, the methods, apparatus, and computer-readable media described above further include displaying an image capture UI element on an image capture user interface (UI) area on a display; receiving input in the image capture UI area, wherein the input is one of touch input, hover input, and gesture input, receiving a second image captured by a camera in response to receiving input in the image capture UI area, and storing the second image in a non-temporary computer-readable storage medium.

[0018]

[0018] In some embodiments, the methods, apparatus, and computer-readable media described above further include receiving input in an area selection user interface (UI) area, wherein the input is one of touch input, hover input, and gesture input, and selecting a selection area of ​​a display image in response to receiving input in the area selection UI area, wherein the area selection UI area is separate from the selection area of ​​the display image. In some embodiments, the selection area of ​​the display image includes at least a portion of an indicator area. In some embodiments, the methods, apparatus, and computer-readable media described above further include displaying an area selection UI element in the area selection UI area. In some embodiments, the methods, apparatus, and computer-readable media described above further include generating an area selection UI element so as to draw a copy of at least a portion of the selection area.

[0019]

[0019] In some embodiments, the methods, apparatus, and computer-readable media described above further comprise setting image capture parameters to specific settings determined based on the selected area of ​​the displayed image in response to the selection of a selection area, wherein the image capture parameters include at least one of focus and exposure parameters. In some embodiments, the methods, apparatus, and computer-readable media described above further comprise setting image processing properties to specific settings determined based on the selected area of ​​the displayed image in response to the selection of a selection area, wherein the image processing properties are one of white balance, black level compensation, gamma correction, gain, color correction, color saturation, noise filter, contrast control, brightness control, tone mapping, sharpness, blur, and red-eye correction.

[0020]

[0020] In some embodiments, the methods, apparatus, and computer-readable media described above further include receiving a second image captured by a camera after the capture of an image, determining that the second image is obscured by occlusion, and displaying the display image on a display in response to the determination that the second image is obscured by occlusion. In some embodiments, the methods, apparatus, and computer-readable media described above further include receiving an input, wherein occlusion is associated with the reception of the input, wherein the input is one of touch input, hover input, and gesture input, determining a setting based on the selected area of ​​the display image, applying the setting, and receiving a third image captured by a camera while the setting is applied and after the capture of the second image.

[0021]

[0021] In some embodiments, the methods, apparatus, and computer-readable media described above further comprises receiving a second image captured by a second camera based on secondary incident light received by the second camera, wherein the second camera is positioned relative to the display to receive secondary incident light passing through a second portion of the display before reaching the second camera. In some embodiments, the methods, apparatus, and computer-readable media described above further comprises determining that the second image is obscured by occlusion, wherein the display image is displayed in response to the determination that the second image is obscured by occlusion. In some embodiments, the methods, apparatus, and computer-readable media described above further comprises determining depth information corresponding to one or more objects depicted in the image and the second image by processing the image and the second image.

[0022]

[0022] In some embodiments, the apparatus comprises a camera, a mobile device (e.g., a mobile phone or a so-called "smartphone" or other mobile device), a wireless communication device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other devices. In some embodiments, one or more processors include an image signal processor (ISP). In some embodiments, the apparatus includes a camera or a plurality of cameras for capturing one or more images. In some embodiments, the apparatus includes an image sensor for capturing image data. In some embodiments, the apparatus further includes a display for displaying images, one or more notifications relating to the processing of images, and / or other displayable data.

[0023]

[0023] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to the appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

[0024]

[0024] Together with other features and embodiments, the above will become more apparent upon reference to the following specification, the claims, and the accompanying drawings.

[0025]

[0025] Exemplary embodiments of the present application will be described in detail below with reference to the following figures.

Brief Description of the Drawings

[0026] [Figure 1]

[0026] A block diagram showing an example of the architecture of an image capture and processing device. [Figure 2A]

[0027] An exploded front view of a mobile device having a display, an under-display camera, and an under-display speaker. [Figure 2B]

[0028] An assembled front view of the mobile device of FIG. 2A. [Figure 2C]

[0029] A broken side view of the mobile device of FIG. 2A. [Figure 2D]

[0030] A front view of the mobile device of FIG. 2A having a first under-display camera, a second under-display camera, and an under-display sensor. [Figure 3A]

[0031] A front view of a mobile device having an under-display camera on which the display shows a black bezel-style indicator over the camera. [Figure 3B]

[0032] Front view of a mobile device with an under-display camera where the display shows a black notch-shaped indicator above the camera. [Figure 3C]

[0033] Front view of a mobile device with an under-display camera where the display shows a black circular indicator above the camera. [Figure 4]

[0034] Front view of a mobile device with an under-display camera where the display shows a triangular indicator around the area where the camera is positioned. [Figure 5]

[0035] Front view of a mobile device with an under-display camera, where the display shows a circular indicator on the camera and area selection user interface (UI) elements for selecting a selectable area on the camera. [Figure 6]

[0036] Front view of a mobile device compensating for under-display camera occlusion by the user's hand during touch, hover, or gesture input. [Figure 7]

[0037] Front view of a mobile device with multiple under-display cameras and under-display sensors, where the display shows an interface area for controlling multiple under-display cameras and under-display sensors. [Figure 8]

[0038] A flowchart illustrating examples of image processing techniques. [Figure 9]

[0039] A flowchart illustrating another example of an image processing technique. [Figure 10]

[0040] A diagram illustrating an example of a system for implementing several aspects of this technology. [Modes for carrying out the invention]

[0027]

[0041] Several aspects and embodiments of this disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some may be applied in combination. Specific details are provided below for illustrative purposes to provide a complete understanding of the embodiments of this application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0028]

[0042] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the Disclosure. Rather, the following description of exemplary embodiments provides a usable description for implementing the exemplary embodiments for those skilled in the art. It should be understood that various modifications may be made to the function and configuration of the elements without departing from the spirit and scope of the application as set forth in the appended claims.

[0029]

[0043] An image capture device (e.g., a camera) is a device that uses an image sensor to receive light and capture an image frame, such as a still image or a video frame. The terms “image,” “image frame,” and “frame” are used interchangeably herein. An image capture device typically includes at least one lens that receives light from a scene and directs the light toward the image sensor of the image capture device. The light received by the lens can pass through an aperture controlled by one or more control mechanisms. After passing through the lens, the light received by the lens is received by the image sensor. One or more control mechanisms can control exposure, focus, and / or zoom based on information from the image sensor and / or from an image processor (e.g., a host or application process and / or an image signal processor). In some examples, one or more control mechanisms include a motor or other control mechanism that moves the lens of the image capture device to a target lens position.

[0030]

[0044] As will be described in more detail below, this specification describes systems and techniques for using one or more under-display cameras and / or one or more under-display sensors positioned beneath the display of a device (e.g., a mobile device or other device). Some image capture devices include one or more forward-facing cameras on the front of the image capture device. In some examples, one or more forward-facing cameras face the same direction as the display screen of the image capture device. In an exemplary video conference, one or more forward-facing cameras capture the user's image (e.g., video frames), while the display screen faces the user and displays the images (e.g., video frames) of one or more other users connected to the video conference, and / or the user themselves. The user can use one or more forward-facing cameras facing the user to capture an image of themselves, referred to as a “selfie.” The user can view the displayed image frame shown on the display screen facing the user, while the user can align the image capture to the frame and align the user’s drawing in the image by moving the image capture device to a predetermined position for capturing the image. The displayed image frame is obtained based on image frames captured by one or more forward-facing cameras and / or temporarily stored in an image buffer, and can therefore function as a preview of the view of one or more forward-facing cameras.

[0031]

[0045] In some cases, one or more front-facing cameras may be positioned adjacent to the display screen in a direction parallel to the flat surface of the display screen. For example, one or more front-facing cameras may be positioned in the bezel around the display screen. In some cases, the display screen may include a notch or cutout, within which the display screen does not display anything. A cutout may sometimes be referred to as a “hole” or “hole punch” in the display screen. Each of the one or more front-facing cameras may be positioned within such a bezel, notch, or cutout. One benefit of positioning a front-facing camera in a bezel, notch, or cutout is that the user understands where the front-facing camera is positioned on the front of the image capture device. In some cases, a bezel, notch, or cutout may be one of the few locations on the front of the device that is not covered by the display screen. The presence of a bezel, notch, or cutout on the front of the device may be seen as an indicator that a front-facing camera is positioned within a bezel, notch, or cutout. However, the presence of bezels, notches, and / or cutouts on a device prevents the display screen from occupying the entire surface of the image capture device. Such side effects of bezels, notches, and / or cutouts can cause problems, especially in devices with small fronts, such as cellular phones. The use of smaller display screens can lead to accessibility issues. For example, users with poor eyesight or visual impairments may have difficulty seeing the visual elements displayed on the display screen. If the display screen is a touchscreen, users with poor eyesight may have difficulty seeing and successfully interacting with the touch-based user interface elements displayed on the display screen. The placement of the camera in bezels, notches, and / or cutouts can also cause gaze mismatch issues.Gaze mismatch refers to a user looking at the display instead of the camera while their image is being captured by the camera. This results in the user appearing to have their eyes looking to the side in the captured image. This often becomes a problem in video conferencing, where a user will generally look at the display to see the second user they are speaking to, but in doing so, they are not looking at the camera. Consequently, it appears that no user is making eye contact with other users during the video conference.

[0032]

[0046] A device may include a camera positioned beneath the display screen. Such a camera may be called an under-display camera. An under-display camera can capture an image using the light passing through the display screen beneath it. In some examples, an under-display camera can capture an image using the light passing through the display screen while the display screen is in use and displaying visual content. In some examples, an under-display camera can capture an image using the light passing through the display screen while the display screen is off and not displaying any visual content. In some examples, an under-display camera can capture an image using the light passing through the display screen during a short period when the display screen is flickering off while the display screen is otherwise on and displaying visual content.

[0033]

[0047] Devices with one or more under-display cameras may include a display screen that occupies the entire front of the device, facing the same direction as the under-display cameras and without any bezels, notches, or cutouts. However, devices with under-display cameras can have problems. For example, if the camera is below the display screen, the user may not know where the camera is located. For instance, a user who wants to look at the front-facing camera while an image is being captured, as is common with selfie or group photos, may not know where to look on the device. A user interacting with the touchscreen while the under-display camera is active may also cover the camera with their finger or stylus by touching the portion of the touchscreen that is positioned above the under-display camera. For example, a device may include a touchscreen user interface that allows the user to touch that area of ​​an image to select that area of ​​the image. In some cases, the user may inadvertently cover the camera while touching the display to try to select an area of ​​the image.

[0034]

[0048] Issues related to under-display cameras are not limited to front-facing displays and / or front-facing cameras. Some devices may have multiple displays and / or multiple under-display cameras on different surfaces. For example, some devices may have a front and a back, and displays with under-display cameras on both the front and back. Some devices may have a foldable display, or a fold or hinge between two separate displays. Those displays, and the under-display cameras beneath each of them, may face different directions at different times. Some under-display cameras may be obscured and disabled by the device when the foldable device is folded and closed.

[0035]

[0049] Figure 1 is a block diagram showing the architecture of the image capture and processing system 100. The image capture and processing system 100 includes various components used to capture and process images of a scene (for example, images of scene 110). The image capture and processing system 100 can capture individual images (or photographs) and / or capture videos containing multiple images (or video frames) in a specific sequence. The lens 115 of the system 100 faces the scene 110 and receives light from the scene 110. The lens 115 directs the light toward the image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130.

[0036]

[0050] One or more control mechanisms 120 may control exposure, focus, white balance, black balance, and / or zoom based on information from the image sensor 130 and / or from the image processor 150. One or more control mechanisms 120 may include multiple mechanisms and components, for example, control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. One or more control mechanisms 120 may also include additional control mechanisms other than those shown, such as control mechanisms for analog gain, flash, HDR, depth of field, and / or other image capture properties.

[0037]

[0051] The focus control mechanism 125B of the control mechanism 120 can acquire the focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B may adjust the focus by moving the lens 115 closer to or further away from the image sensor 130 by acting a motor or servo (or other lens mechanism). In some cases, additional lenses may be included in the system 100, such as one or more microlenses on each photodiode of the image sensor 130. Each microlens can direct the light received from the lens 115 towards the corresponding photodiode before the light reaches the photodiode. The focus setting may be determined via contrast-detection autofocus (CDAF), phase-detection autofocus (PDAF), hybrid autofocus (HAF), or any combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may also be referred to as the image capture setting and / or image processing setting.

[0038]

[0052] The exposure control mechanism 125A of the control mechanism 120 can acquire the exposure setting. In some cases, the exposure control mechanism 125A stores the exposure setting in a memory register. Based on this exposure setting, the exposure control mechanism 125A can control the aperture size (e.g., aperture size or f / aperture), the duration the aperture is open (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be referred to as the image capture setting and / or image processing setting.

[0039]

[0053] The zoom control mechanism 125C of the control mechanism 120 can acquire the zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C can control the focal length of an assembly of lens elements (lens assembly) including lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by acting one or more motors or servos (or other lens mechanisms) to move one or more of the lenses relative to each other. The zoom setting is sometimes called the image capture setting and / or image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focus zoom lens. In some examples, the lens assembly may include a focusing lens (which may be lens 115 in some cases) that first receives light from the scene 110, and then the light passes through an infinite focus zoom system between the focusing lens (e.g., lens 115) and the image sensor 130 before the light reaches the image sensor 130. An infinity zoom system may, in some cases, include two positive (e.g., converging, convex) lenses of equal or similar focal lengths (e.g., within a threshold difference between them) with a negative (e.g., diverging, concave) lens in between. In some cases, the zoom control mechanism 125C moves one or more of the lenses in the infinity zoom system, such as one or both of the negative and positive lenses.

[0040]

[0054] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image produced by the image sensor 130. In some cases, different photodiodes may be covered by different color filters and thus measure light that matches the color of the filter covering the photodiode. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, and each pixel in the image is produced based on red light data from at least one photodiode covered by the red filter, blue light data from at least one photodiode covered by the blue filter, and green light data from at least one photodiode covered by the green filter. Other types of color filters may use yellow, magenta, and / or cyan (also called "emerald") color filters instead of, or in addition to, red, blue, and / or green color filters. Some image sensors (e.g., image sensor 130) may have no color filters at all and instead use different photodiodes across the entire pixel array (in some cases stacked vertically). Different photodiodes across the entire pixel array have different spectral sensitivity curves and can therefore respond to light of different wavelengths. Monochrome image sensors also lack color filters and therefore may lack color depth.

[0041]

[0055] In some cases, the image sensor 130 may include, as an alternative or addition, an opaque mask and / or reflective mask that blocks light from reaching a photodiode or part of a photodiode at a certain time and / or angle, which may be used for phase-detection autofocus (PDAF). The image sensor 130 may also include an analog gain amplifier for amplifying the analog signal output by the photodiode, and / or an analog-to-digital converter (ADC) for converting the analog signal output of the photodiode (and / or amplified by the analog gain amplifier) ​​into a digital signal. In some cases, some components or functions discussed with respect to one or more of the control mechanisms 120 may be included in the image sensor 130 instead or in addition thereto. The image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplier CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide-semiconductor (CMOS) sensor, an N-type metal-oxide-semiconductor (NMOS) sensor, a hybrid CCD / CMOS sensor (e.g., sCMOS), or any other combination thereof.

[0042]

[0056] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other types of processors 1010 discussed in relation to the computing system 1000. The host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, the image processor 150 is a single integrated circuit or chip (for example, called a system-on-a-chip or SoC) including the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (for example, input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (for example, 3G, 4G, or LTE®, 5G, etc.), memory, connectivity components (for example, Bluetooth®, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. The I / O port 156 may include any suitable input / output port or interface by one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a Serial General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 Physical (PHY) Layer port or interface), an Advanced High Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one exemplary example, the host processor 152 may communicate with the image sensor 130 using the I2C port, and the ISP 154 may communicate with the image sensor 130 using the MIPI port.

[0043]

[0057] The image processor 150 can perform several tasks, including demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging of image frames to form an HDR image, image recognition, object recognition, feature recognition, input reception, output management, memory management, or any combination thereof. The image processor 150 can store image frames and / or processed images in random access memory (RAM) 140 / 1020, read-only memory (ROM) 145 / 1025, cache, memory unit, other storage device, or any combination thereof.

[0044]

[0058] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, keyboard, keypad, touchscreen, trackpad, touch-sensitive surface, printer, any other output device 1035, any other input device 1045, or any combination thereof. In some cases, captions may be input to the image processing device 105B through the physical keyboard or keypad of the I / O device 160, or through the virtual keyboard or keypad of the touchscreen of the I / O device 160. The I / O 160 may include one or more ports, jacks, or other connectors that enable wired connections between the system 100 and one or more peripheral devices, through which the system 100 may receive data from and / or send data to one or more peripheral devices. The I / O 160 may include one or more wireless transceivers that enable wireless connectivity between the system 100 and one or more peripheral devices, through which the system 100 may receive data from and / or transmit data to one or more peripheral devices. Peripheral devices may include any of the previously discussed types of I / O devices 160, and when coupled to ports, jacks, wireless transceivers, or other wired and / or wireless connectors, they themselves may be considered I / O devices 160.

[0045]

[0059] In some cases, the image capture and processing system 100 may be a single device. In some cases, the image capture and processing system 100 may be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B may be coupled to each other, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B may be separated from each other.

[0046]

[0060] As shown in Figure 1, the vertical dashed line divides the image capture and processing system 100 in Figure 1 into two parts, representing the image capture device 105A and the image processing device 105B, respectively. The image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. The image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), RAM 140, ROM 145, and I / O 160. In some cases, some components shown in the image capture device 105A, such as the ISP 154 and / or the host processor 152, may be included in the image capture device 105A.

[0047]

[0061] The image capture and processing system 100 may include electronic devices such as mobile or landline telephone handsets (e.g., smartphones, cellular phones, etc.), desktop computers, laptop or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi® communication, wireless local area network (WLAN) communication, or any combination thereof. In some implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device such as a mobile handset, a desktop computer, or other computing device.

[0048]

[0062] Although the image capture and processing system 100 is shown to include several components, those skilled in the art will understand that the image capture and processing system 100 may include more components than those shown in Figure 1. The components of the image capture and processing system 100 may include one or more combinations of software, hardware, or software and hardware. For example, in some implementations, the components of the image capture and processing system 100 may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits) to perform the various operations described herein, and may also include and / or be implemented using computer software, firmware, or any combination thereof, electronic circuits, or other electronic hardware. The software and / or firmware may include one or more instructions that are stored on a computer-readable storage medium and are executable by one or more processors of the electronic devices implementing the image capture and processing system 100.

[0049]

[0063] I / O156 and / or I / O160 may include connections to a display. The display connections included in I / O156 and / or I / O160 may include output connections through which the image processor 150 transfers images captured by the image sensor 130 and processed using the image processor 150 to a display and / or controller display. The display controller receives the images and displays them on the display.

[0050]

[0064] In some implementations, the display may be a touchscreen. Connections to the display included in I / O 156 and / or I / O 160 may include input connections through which the image processor 150 can receive input from and / or through the touchscreen. For example, the touchscreen may receive touch input, and the touchscreen controller may send information about the touch input to the image processor 150. Touch input may be an instance in which a portion of the touchscreen detects a touch by a user's finger, stylus, pointer, pen, pencil, or another input device. Information about the touch input may include, for example, the coordinates of the touch input on the touchscreen. Touch input may be received on the touchscreen while the touchscreen is displaying an image. Information about the touch input may identify the portion of the image displayed on the touchscreen in the portion of the touchscreen that received the touch input. The portion of the image may include, for example, an area spanning one or more pixels of an image. Touch input may be received on the touchscreen while the touchscreen is displaying a user interface (UI). The UI may include one or more UI elements, such as virtual buttons or sliders. Information regarding touch input can identify the portion of the UI (e.g., UI elements) displayed on the touchscreen in the portion of the touchscreen that received the touch input. The touchscreen may be a resistive touchscreen, a capacitive touchscreen, a surface acoustic wave (SAW) touchscreen, an infrared (IR) touchscreen, an optical imaging touchscreen, a dispersed signal touchscreen, a piezoelectric touchscreen, an acoustic pulse recognition touchscreen, another type of touchscreen discussed herein, or any combination thereof.

[0051]

[0065] In some implementations, the display may include one or more other input components. For example, the display may include one or more active depth sensors next to or below the display. The active depth sensors may face the same direction as the display. The active depth sensors may include a stereo camera. The active depth sensor can be any type of sensor discussed herein with respect to sensor 260. The connections to the display included in I / O 156 and / or I / O 160 may include input connections through which the image processor 150 can receive input from the active depth sensor. The input from the active depth sensor may be processed by the image processor 150 or another processor to determine the distance from the sensor to an object, which may be called depth. If the active depth sensor includes a camera, this depth determination may be performed using stereoscopic image processing, which will be discussed further herein. A device with an active depth sensor may thus determine the three-dimensional coordinates of objects in the environment around the device. The image processor 150 (or another processor) may process sensor measurements from the active depth sensor to determine whether the user's hand, finger, or pointing device (e.g., a stylus) is hovering over, pointing at, gestured towards, or gestured around a user interface (UI) element displayed on the display. If the image processor 150 (or another processor) detects that the user (or pointing device) is hovering over, pointing at, gestured towards, or gestured around a particular UI element, the image capture and processing system 100 may trigger a function corresponding to that UI element. Hover-based, pointing-based, or gesture-based interactions with the display may be more hygienic than touchscreen-based interactions with the display, especially in the image capture and processing system 100, where many different people are expected to interact with it.For example, these non-touch inputs may be more hygienic in image capture and processing systems 100 included in point-of-sale (POS) terminals, automated teller machines (ATMs), restaurant ordering kiosks, information kiosks, or other kiosks.

[0052]

[0066] At least a subset of the image capture and processing system 100 may be located beneath the display. In some examples, at least the image sensor 130 and lens 115 are located beneath the display. In one example, the image capture device 105A is located beneath the display. In some examples, the image processing device 105B is located beneath the display. The term “under-display camera” may refer to at least a subset of the image capture and processing system 100, at least a subset of the image capture device 105A, at least a subset of the image processing device 105B, or any combination thereof, that is located at least partially beneath the display. Components or devices located beneath the display may be referred to as under-display components or devices.

[0053]

[0067] Figure 2A is an exploded front view of a mobile device 205 having a display 210, an under-display camera 220, and an under-display speaker 225. Legend 290 is included in Figure 2A showing three axes representing different spatial dimensions in Figure 2A. These axes are identified as the X-axis, Y-axis, and Z-axis. The X-axis and Y-axis are parallel to the flat plane of the display 210. The Z-axis is perpendicular to the flat plane of the display 210. The mobile device 205 is shown in Figure 2A with a body 215 including a white front 255. The body 215 also includes a housing 250 which is shaded using diagonal stripes in Figure 2A. As indicated by dashed lines and dashed arrows, the display 210 is in front of the front 255 of the body 215 along the Z-axis and will be positioned on the front 255 of the body 215. The front 255 of the body 215 includes the camera 220 and the speaker 225. Placing the display 210 on the front of the main unit 215 includes positioning the display 210 above and / or on the camera 220 and speaker 225 along the Z-axis. By positioning the display 210 on the front of the main unit 215, the camera 220 and speaker 225 are covered by the surface of the display 210 along the Z-axis.

[0054]

[0068] The main unit 215 also includes a processor 270, a memory 275, and a battery 280. The camera 220 and speaker 225 are connected to the processor 270 and the memory 275 via one or more data connection lines. In some cases, the processor 270 may include an image processor 150, an ISP 154, a host processor 152, or any combination thereof. The processor 270 can process image data captured by the image sensor of the camera 220. The memory 275 can store images captured by the camera 220 and / or processed by the processor 270. The memory 275 can also store audio to be output by the speaker 225. The processor 270 can process audio to be output by the speaker 225. The camera 220, speaker 225, processor 270, and / or memory 275 receive power from the battery 280 via one or more data lines.

[0055]

[0069] Camera 220 may be any type of camera discussed herein. For example, camera 220 may include at least a subset of the image capture device 105A, at least a subset of the image processing device 105B, at least a subset of the image capture and processing system 100, at least a subset of any other camera device or system discussed herein, or a combination thereof. Camera 220 may use any type of lens. For example, the camera may use a conventional lens, a wide-angle lens, a fisheye lens, a long-focus lens, a telephoto lens, a zoom lens, or a combination thereof. Camera 220 may detect light within a particular spectrum of light. The spectrum of light may be a subset of the electromagnetic spectrum. For example, the spectrum of light may be the visible light (VL) spectrum, the infrared (IR) light spectrum, the near-infrared (NIR) light spectrum, the ultraviolet (UV) light spectrum, another subset of the electromagnetic spectrum, or a combination thereof, or may include them. The NIR light spectrum may refer to a subset of the IR light spectrum adjacent to the VL spectrum. The image sensor of camera 220 may be a complementary metal-oxide-semiconductor (CMOS) image sensor, a charge-coupled device (CCD) image sensor, another type of image sensor, or a combination thereof. The image sensor of camera 220 may be an image sensor of any size.

[0056]

[0070] Figure 2B is an undisassembled front view of the mobile device 205 of Figure 2A. The camera 220 is located below the display 210 in Figure 2B and is covered by the display 210. Legend 290 in Figure 2A is also included in Figure 2B. The camera 220 is sometimes referred to as an under-display camera. The speaker 225 is located below the display 210 along the Z-axis in Figure 2B and is covered by the display 210. The speaker 225 is sometimes referred to as an under-display speaker. The speaker 225 is located near the top of the mobile device 205 along the Y-axis and can output audio toward the display 210 so that sound waves travel through the display 210. A user holding the mobile device 205 adjacent to their ear can hear the audio output by the speaker 225 through the display 210. The speaker 225 and camera 220 may be adjacent to the display 210 along the Z-axis. Since camera 220 is completely covered by display 210, any incident light reaching camera 220 passes through the light-transmitting area 285 of display 210 before reaching camera 220. The light-transmitting area 285 is illustrated using a dashed line and is at least as large as the lens of camera 220. In some devices, the camera may be only partially covered by display 210, in which case some incident light reaching such camera may pass through area 285 of display 210, while other incident light reaching such camera may pass through another display, a non-display surface, or not pass through any surface at all (e.g., air or another fluid). In some cases, the light-transmitting area 285 of display 210 may simply be referred to as a portion of display 210 or an area of ​​display 210.

[0057]

[0071] For brevity, the processor 270, memory 275, and battery 280 are not shown in Figure 2B. Similarly, the data and power lines between the camera 220, speaker 225, processor 270, memory 275, and battery 280 are not shown in Figure 2B. It should be understood that these components can also be located below the display 210 along the Z-axis and exist in the mobile device 205 of Figure 2B in a configuration similar to that shown in Figure 2A.

[0058]

[0072] The display 210 is shown as being slightly smaller than the front 255 of the body 215 of the mobile device 205, so that the display 210 is clearly visible in Figures 2A and 2B as a distinct component of the mobile device 205. The mobile device 205 therefore includes a small bezel around the display 210. However, this bezel does not necessarily have to be present on the mobile device 205. In some cases, the display 210 can cover the entire vertical and / or horizontal length of the front 255 of the body 215 of the mobile device 205, edge to edge, if there is no bezel around the display 210. In some cases, the display 210 may include one or more curved portions around the edges of the front 255 of the body 215 of the mobile device 205, so that the display 210 can even wrap around to at least a portion of the side of the mobile device 210 connecting the front 255 of the body 215 of the mobile device 205 to the back 240 of the body 215.

[0059]

[0073] Figure 2C is a broken side view of the mobile device 205 of Figure 2A. Due to the change in viewpoint from Figures 2A and 2B to Figure 2C, Figure 2C includes a legend 295 showing only the Y and Z axes of legend 290. In the side view of the mobile device 205 of Figure 2C, the front 252 of the display 210 appears on the left side of the mobile device 205, while the back 240 of the body 215 appears on the right side of the mobile device 205. The body 215 includes a housing 250, which is shown as a shaded area with horizontal stripes. The body 215 also includes a camera 220 and a speaker 225. Parts of the body 215 may be coupled to the camera 220 and / or speaker 225. The front portion of the camera 220 and speaker 225 forms part of the front 255 of the body 215. The camera 220 and speaker 225 are shown as being adjacent to and / or coupled to the display 210 along the Z axis. In some cases, there may be some space along the Z-axis between the camera 220 and the display 210, and / or between the speaker 225 and the display 210.

[0060]

[0074] Some light rays 245 are shown as arrows that pass through the display 210 and reach the camera 220. The image sensor 130 of the camera 220 receives the light 245 that passes through at least a portion of the display 210. In some examples, the display 210 may include microlenses in the gap regions between the pixels of the display. The display 210 may be a touchscreen. In some embodiments, the gap regions and therefore the microlenses are also located between traces on the touch-sensitive touchscreen surface of the display 210. These traces may be capacitive, resistive, or other traces, depending on the type of touchscreen. In some examples, the light 245 passes through transparent and / or translucent layers of liquid crystal, glass, and / or plastic of the display 210 before reaching the under-display camera. In some cases, the camera 220 and / or associated processor 270 may perform image processing, such as color filters, to compensate for colors that bias the light 245 from colors associated with the materials of the display 210 that it passes through before reaching the camera 220. The light 245 received by the image sensor 130 of the camera 220 may be incident light that passes through at least a portion of the display 210 before reaching the camera 220 (for example, before hitting the image sensor 130 of the camera 220).

[0061]

[0075] Some sound waves 260 are indicated as arrows passing through the display 210. The sound waves 260 travel past the front 252 of the display 210 and propagate through the air. The sound waves 260 can be heard by a user when they are adjacent to or within range of the mobile device 205.

[0062]

[0076] For brevity, the processor 270, memory 275, and battery 280 are not shown in Figure 2C. Similarly, the data and power lines between the camera 220, speaker 225, processor 270, memory 275, and battery 280 are not shown in Figure 2C. It should be understood that these components can also be present in the mobile device 205 of Figure 2C in a configuration similar to that shown in Figure 2A.

[0063]

[0077] Figure 2D is a front view of the mobile device 205 of Figure 2A, which has a first under-display camera 220A, a second under-display camera 220B, and an under-display sensor 260. The first under-display camera 220A and the second under-display camera 220B are sometimes referred to as the first camera 220A and the second camera 220B, respectively. The under-display sensor 260 is sometimes referred to as sensor 260. Legend 290 in Figure 2A is also included in Figure 2D. The mobile device 205 is an example of a device with multiple under-display cameras. As with the cameras 220 in Figures 2A to 2C, the first camera 220A and the second camera 220B are positioned below the display 210 along the Z-axis. In the context of Figure 2C, the first camera 220A and the second camera 220B may be positioned between the display 210 of the mobile device 205 and the housing 250 of the main body 215, similar to the camera 220 and speaker 225 in Figure 2C. The first camera 220A can receive incident light that passes through at least a first portion of the display 210 before reaching the first camera 220A. The second camera 220B can receive incident light that passes through at least a second portion of the display 210 before reaching the second camera 220B. The incident light reaching the cameras may include incident light that strikes the camera's image sensor. The first camera 220A and the second camera 220B can capture images based on the reception of this incident light at their respective image sensors. The first camera 220A is closer to the first side of the mobile device 205, while the second camera 220B is closer to the second side of the mobile device 205. In the exemplary mobile device 205 shown in Figure 2D, the first side is the top of the mobile device 205, while the second side is the bottom of the mobile device. In other examples, the first side and / or the second side may be the left or right side.

[0064]

[0078] Since the first camera 220A and the second camera 220B are positioned at a distance from each other, the first camera 220A and the second camera 220B can be used as stereoscopic cameras. Within the same time window, the first camera 220A can capture the first image and the second camera 220B can capture the second image. The time window can be a predetermined duration, such as one or more picoseconds, one or more nanoseconds, one or more milliseconds, one or more seconds, or a combination thereof. The mobile device 205, or another device receiving the first and second images from the mobile device 205, can process the first and second images using stereoscopic image processing to determine depth information corresponding to the first and second images. The depth information can refer to the depth of different objects depicted in the first and second images. Depth can refer to the distance between the captured object and the mobile device 205. The greater the distance between the captured object and the mobile device 205, the greater the depth of the object. The smaller the distance between the captured object and the mobile device 205, the smaller the object's depth. Stereoscopic image processing may also be called stereo image processing, multi-image processing, binocular image processing, stereoscopic camera processing, stereo camera processing, multi-camera processing, binocular camera processing, or a combination thereof.

[0065]

[0079] Stereoscopic image processing can use stereo parallax, binocular parallax, stereoscopic, stereo triangulation, triangulation, stereophotogrammetry, photogrammetry, three-dimensional (3D) mapping, three-dimensional modeling, or a combination thereof, to determine depth information corresponding to at least a subset of pixels in a first image and / or at least a subset of pixels in a second image. In some examples, stereoscopic image processing may include detecting a first drawing of a feature (e.g., a corner or edge) in the first image and detecting a second drawing of the same feature in the second image. Stereoscopic image processing can determine the depth of a feature by determining the distance between the location of the first drawing of the feature in the first image and the location of the second drawing of the feature in the second image. This distance may be called stereo parallax, stereoscopic parallax, or binocular parallax. The greater the stereo parallax, the smaller the depth of the feature, and the closer the feature is to the mobile device 205. The smaller the stereo parallax, the greater the feature depth, and the further away the feature is from the mobile device 205.

[0066]

[0080] For example, both the first and second images can depict a person in front of the background, similar to the display image 350 shown on the display 210 in Figure 3A. Because the person is close to the mobile device 205, the mobile device 205 can use stereoscopic image processing to process the first and second images and identify that the pixels depicting the person correspond to small depth values. Because the background is far from the mobile device 205, the mobile device 205 can use stereoscopic image processing to process the first and second images and identify that the pixels depicting the background correspond to large depth values.

[0067]

[0081] The mobile device 205 may include three or more under-display cameras beneath the display 210. For example, the mobile device 205 may include three, four, five, six, seven, eight, nine, ten, or eleven or more under-display cameras. A mobile device 205 with N under-display cameras beneath the display 210 can capture at least N images of a scene within the same time window. Each of the N images of a scene may be captured using one of the N under-display cameras beneath the display 210. All N images may be processed using stereoscopic image processing or multi-image processing to determine depth information of pixels and / or features depicted in at least a subset of the N images. When N is 3 or greater, the depth information determined may be more accurate and detailed than when N is 2.

[0068]

[0082] In some examples, the first camera 220A and the second camera 220B share at least one component, type of component, characteristic, function, or combination thereof. For example, the first camera 220A may use the same type of lens as the second camera 220B, the first camera 220A may be able to detect light in the same light spectrum as the second camera 220B, the first camera 220A may use the same type of image sensor as the second camera 220B, the first camera 220A may use an image sensor of the same dimensions as the second camera 220B, or a combination thereof.

[0069]

[0083] In some examples, the first camera 220A differs from the second camera 220B by the use of at least one different component, type of component, characteristic, function, or combination thereof. For example, the first camera 220A may use a first lens, while the second camera 220B may use a second lens. For example, if the first lens is a wide-angle lens or a fisheye lens, the first lens may have a wider angle than the second lens. The wider angle of the first lens may give the first camera 220A a wider field of view than the second camera 220B. The first lens may be a telephoto lens, while the second lens is not, thereby providing the first camera 220A with a longer focus than the second camera 220B. The first lens may be a zoom lens, while the second lens is not, thereby providing the first camera 220A with a greater zoom than the second camera 220B. In some examples, the image sensor of the first camera 220A detects light in a first spectrum of light, while the image sensor of the second camera 220B detects light in a second spectrum of light. Each spectrum of light can refer to a different subset of the entire electromagnetic spectrum. For example, the first spectrum of light may be the visible light (VL) spectrum, while the second spectrum of light may be the infrared (IR) spectrum. The second spectrum of light may be the near-infrared (NIR) spectrum, which is a subset of the IR spectrum adjacent to the VL spectrum. The second spectrum of light may also be the ultraviolet (UV) spectrum. In some examples, the first image sensor of the first camera 220A may be a different type of image sensor than the second image sensor of the second camera 220B. For example, the first image sensor may be a complementary metal-oxide-semiconductor (CMOS) image sensor, while the second image sensor is a charge-coupled device (CCD) image sensor. The dimensions of the first image sensor may differ from those of the second image sensor, such that one is larger than the other. The first camera 220A and the second camera 220B can differ in any combination as discussed herein.

[0070]

[0084] Sensor 260 can be any type of sensor. For example, sensor 260 may include at least one of the following: radio detection and ranging (RADAR) sensors, light detection and ranging (LIDAR) sensors, electromagnetic detection and ranging (EMDAR) sensors, multispectral imaging detection and active reflectance (MIDAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, laser rangefinders, time-of-flight (TOF) emitters / sensors, structured light emitters / sensors, cameras with coded apertures, active depth sensors, distance sensors, microphones, accelerometers, gyroscopes, global navigation satellite system (GNSS) receivers, global positioning system (GPS) receivers, other sensors discussed herein, or combinations thereof. As with camera 220 in Figures 2A-2C, sensor 260 is located below the display 210. In the context of Figure 2C, the sensor 260 may be positioned between the display 210 of the mobile device 205 and the housing 250 of the main body 215, similar to the camera 220 and speaker 225 in Figure 2C. The sensor 260 can receive incident light, sound, radio waves, microwaves, IR waves (e.g., NIR waves), UV waves, electromagnetic (EM) radiation, and / or other signals that pass through at least a portion of the display 210 before hitting the sensor 260. In some cases, the sensor 260 may emit light, sound, electromagnetic (EM) radiation, and / or other signals that pass through at least a portion of the display 210 after being emitted by the sensor 260 and travel into the environment. In some cases, the signal emitted by the sensor 260 may be reflected from objects in the environment, bounced back to the sensor 260, and received by the sensor 260.

[0071]

[0085] Figure 3A is a front view of a mobile device 205 having an under-display camera 220 in which the display 210 displays a black bezel-style indicator 310A in an indicator area. The term “indicator area” may refer to the area in which an indicator (such as the black bezel-style indicator 310A or another indicator) is displayed. The indicator area may include at least a subset of the light-passing area 290 on the camera 220. The legend 290 in Figure 2A is also included in Figure 3A. The camera 220 of the mobile device 205 captures one or more images. In some examples, the camera 220 may continuously capture images and temporarily store those images in an image buffer. The mobile device 205 is shown by displaying a display image 350 on the display 210. In some examples, the display image 350 is one of the images captured by the camera 220 and / or stored in the image buffer on the display 210. In some examples, the display image 350 is based on one of the images captured by the camera 220 and / or stored in the image buffer on the display 210. In some examples, the displayed image 350 is a modified variant of one of the images captured by the camera 220 and / or stored in an image buffer on the display 210. For example, the mobile device 205 may generate the displayed image 350 by modifying the image captured by the camera 220 using cropping, upscaling, downscaling, stretching, skewing, distortion, warping, brightness adjustment, contrast adjustment, color saturation adjustment, tone mapping adjustment, white balance adjustment, black balance adjustment, histogram adjustment, level adjustment, red-eye correction, text overlay, user interface element overlay, graphics overlay, another image overlay, another image element overlay, merging an image with another image (for example, to produce an HDR image), any other image processing image modification discussed herein, or a combination thereof.In one example, the displayed image 350 may be a version of an image captured by the camera 220, scaled so that most of the image is visible on the display 210 based on the resolution of the display 210, cropped to fit the dimensions of the display 210, and its brightness, contrast, and saturation adjusted so that it looks good on the display 210. The displayed image 350 may be used as a preview image to preview the field of view of the camera 220 by previewing the displayed image 350 based on the most recently captured image by the camera 220 (for example, the most recently captured image by the camera 220). In Figure 3A, the displayed image 350 displayed on the display 210 is shown as a user selfie image in front of a background, which includes buildings, trees, grass, and other foliage.

[0072]

[0086] Because the camera 220 is located below the display 210, users of the mobile device 205 may not know where the camera 220 is located relative to the front 255 of the display 210. For example, when capturing a selfie image displayed on the display 210, as in the display image 350 of Figure 3A, a user who wants to look at the camera while the image is being captured may not know where to look. To solve this problem, the mobile device 205 in Figure 3A displays a bezel-shaped indicator 310A overlaid on the display image 350 within the indicator area of ​​the display 210. The bezel-shaped indicator 310 has a rounded rectangular shape that roughly indicates the area of ​​the image where the camera 220 is located below the display 210 along the z-axis while the display image 350 is displayed on the display 210. The indicator 310 is shown as a black-shaded area. The mobile device 205 displays the indicator 310A by displaying black on multiple pixels that are grouped together within the indicator area. Depending on the type of display 210, the display 210 can instruct its pixels to display black by turning off power to those pixels. A user of the mobile device 205 who wishes to view the camera 220 will find that they should look within the indicator area in which the bezel-style indicator 310A is displayed. The rounded rectangular shape of the indicator 310A in Figure 3A mimics a bezel. Since the pixels of the display 210 display black within the indicator 310A, any interference to the light 245 that reaches the camera 220 after passing through the display 210 is minimized.

[0073]

[0087] Indicator 310A is large enough that the user would not understand which part of indicator 310A is above camera 220 along the Z-axis. To help the user identify the location of camera 220, the indicator area may include additional indicators that indicate the location of camera 220 below display 210 may be overlaid on indicator 310A on display 210. For example, two indicator arrows 340 pointing towards camera 220 are shown in Figure 3A as being displayed overlaid on bezel-style indicator 310A in the indicator area on display 210. Indicator arrows may be displayed in a different color from the rest of indicator area 310. Other types of additional indicators, such as indicator shapes, may be overlaid on indicator 310A or otherwise overlaid within the indicator area on display 210. Indicator shapes may abbreviate areas smaller than indicator 310A within the indicator area. For example, indicator shapes may abbreviate the location of camera 220 located below this smaller area along the Z-axis. Examples of indicator shapes include indicator 410 in Figure 4 and indicator 510 in Figure 5.

[0074]

[0088] The mobile device 205 includes an image capture hardware button 330. The mobile device 205 may receive an image capture input when the image capture hardware button 330 is pressed. In some cases, the image capture hardware button 330 may be replaced and / or supplemented with another hardware user interface that receives an image capture input, such as a switch, a touch-sensitive surface separate from the display 210, a fingerprint sensor, another type of bioscanner, a microphone, a wheel, a knob, a slider, an accelerometer, a gyroscope, or any combination thereof. The mobile device 205 also includes an image capture user interface (UI) element 320 displayed on the UI area of ​​the display 210, which is overlaid on the displayed image 350. The mobile device 205 may receive an image capture input when the touch-sensitive touchscreen surface of the display 210 receives a touch in the UI area of ​​the display 210.

[0075]

[0089] In some examples, the reception of an image capture input triggers the camera 220 of the mobile device 205 to capture a second image, and the mobile device 205 receives and stores the second image from the camera 220. In some examples, the reception of an image capture input triggers the transfer of an image already captured by the camera 220 of the mobile device 205 from temporary storage in an image buffer to long-term non-temporary storage in memory 175.

[0076]

[0090] Figure 3B is a front view of a mobile device 205 having an under-display camera 220, where the display 210 displays a black notch-shaped indicator 310B on its camera 220. The mobile device 205 displays the black notch-shaped indicator 310B on the camera 220 in the indicator area along the Z axis. The legend 290 in Figure 2A is also included in Figure 3B. The mobile device 205 in Figure 3B is similar to the mobile device 205 in Figure 3A. The bezel-shaped indicator 310A in Figure 3A is replaced by the notch-shaped indicator 310B in Figure 3B. The notch-shaped indicator 310B occupies less of the display 210 than the bezel-shaped indicator 310A, thereby allowing the displayed image 350 to fill more of the display 210.

[0077]

[0091] Figure 3C is a front view of a mobile device 205 having an under-display camera 220, where the display 210 displays a black circular indicator 310C on its camera 220. The mobile device 205 displays the black circular indicator 310C on the camera 220 in the indicator area along the Z axis. The legend 290 in Figure 2A is also included in Figure 3C. The mobile device 205 in Figure 3C is similar to the mobile device 205 in Figures 3A and 3B. The bezel-shaped indicator 310A in Figure 3A is replaced by the circular indicator 310C in Figure 3C. The circular indicator 310C occupies less of the display 210 than the bezel-shaped indicator 310A or the notch-shaped indicator 310B, thereby allowing the displayed image 350 to fill more of the display 210.

[0078]

[0092] Indicators 310A, 310B, and 310C are shown in Figures 3A, 3B, and 3C as being filled with black pixels or turned-off pixels. In some examples, indicators 310A, 310B, and / or 310C may be filled with pixels of different colors, such as white, gray, red, green, blue, yellow, or any mixture of these colors. The color that fills indicators 310A, 310B, and / or 310C may have any hue, any luminosity, and any opacity. In some examples, indicators 310A, 310B, and 310C may be filled with a pattern, texture, gradation, or even a separate image distinct from the display image 350 on which indicators 310A, 310B, and 310C are overlaid. In some examples, indicators 310A, 310B, and 310C may be at least partially translucent or transparent.

[0079]

[0093] In some examples, indicators 310A, 310B, and 310C may be consistently overlaid on the display image 350 over a period of time, such as the period during which the photo capture software application is running on the mobile device 205 and / or the period during which the under-display camera 220 is activated. In some examples, indicators 310A, 310B, and 310C may blink on and off periodically during the period of time. In some examples, indicators 310A, 310B, and 310C may fade in and out periodically during the period of time. In some examples, the indicators may be shown, called, highlighted, hidden, or any combination thereof in response to user input. User input includes at least one of the following: a tap on the display, a hover on the display, a pre-configured gesture across the display, a user preference setting, or voice input from the user. In some examples, at least a subset of indicators 310A, 310B, or 310C are animated to include borders that, for example, periodically expand, contract, pulsate, or perform some combination thereof.

[0080]

[0094] In some examples, the mobile device 205 displays indicators 310A, 310B, and / or 310C overlaid on the display image 350 by generating a display image 350 that includes indicators 310A, 310B, and / or 310C. The camera 220 may capture an image, and the mobile device 205 may modify this image as previously discussed to generate the display image 350. The mobile device 205 may further modify the display image 350 to merge it with indicators 310A, 310B, and / or 310C. In some examples, indicators 310A, 310B, and / or 310C are overlaid on the display image 350, but the display image 350 remains separate from indicators 310A, 310B, and / or 310C. In some examples, the mobile device 205 uses a blending effect to overlay indicators 310A, 310B, and / or 310C onto the displayed image 350. Blending effects may include, for example, dissolve, multiply, darken, color burn, linear burn, lighten, screen, color dodge, linear dodge, overlay, soft light, hard light, vivid light, pin light, hard mix, difference, exclusion, subtraction, division, threshold, hue blend, saturation blend, color blend, luminosity blend, lighter color comparison, darker color comparison, or any combination thereof.

[0081]

[0095] Figure 4 is a front view of a mobile device 205 having an under-display camera 220, from which a display 210 displays a triangular indicator 410 around an area positioned above its camera 220. The legend 290 in Figure 2A is also included in Figure 4. As with the mobile device 205 in Figures 3A, 3B, and 3C, the mobile device 205 in Figure 4 is shown using its display 210 to display a display image 350. The display image 350 may be based on an image captured by the camera 220. The display image 350 may be temporarily displayed on the display 210 as a preview image to show the user of the mobile device 205 how the scene from which the camera 220 receives light is framed within the camera 220's field of view. The mobile device 205 displays an indicator 410 overlaid on the display image 350. The indicator 410 includes an indicator shape that abbreviates the area positioned above the camera 220. The exemplary indicator 410 shown in Figure 4 is a triangle. The indicator 410 can be any shape. For example, indicator 410 may be a circle, semicircle, square, triangle, rectangle, oval, trapezoid, parallelogram, rhombus, kite, quadrilateral, pentagon, hexagon, heptagon, octagon, nonagon, decagon, polygon with 11 or more sides, or any other shape.

[0082]

[0096] In addition to including a triangular area abbreviation, indicator 410 is shown in Figure 4 filled with a shaded pattern. This shaded pattern, like the black shading of indicators 310A, 310B, and 310C, may represent a shaded area or otherwise filled area within the indicator shape of indicator 410. The shaded area within the abbreviated indicator shape of indicator 410 may have any combination of the properties and features discussed with respect to indicators 310A, 310B, and 310C. For example, the shaded area within the abbreviated indicator shape of indicator 410 may be patterned and semi-transparent. In some cases, the abbreviated indicator shape of indicator 410 acts as a border or edge of the shaded area. In some cases, the shaded area of ​​indicator 410 and the abbreviated indicator shape of indicator 410 cover areas that overlap but are distinct. In some examples, the shaded area may be removed from indicator 410 or may blink on and off periodically. Indicator 410 may be an exclusive indicator of the location of camera 220 below display 210, or it may be used in addition to other indicators such as one of indicators 310A-C and / or indicator arrow 340.

[0083]

[0097] The lines constituting the abbreviated indicator shape of indicator 410 may be of any color, hue, luminosity, transparency, and / or thickness. The lines constituting the abbreviated indicator shape of indicator 410 may be solid color, patterned, include gradation, and / or constitute an image separate from the display image 350 on which the abbreviated indicator shape of indicator 410 is overlaid. The lines constituting the abbreviated indicator shape of indicator 410 may include solid lines, dashed lines, dotted lines, dashed lines, any other type of line, or combinations thereof. The lines constituting the abbreviated indicator shape of indicator 410 may include single lines, double lines, triple lines, quadruple lines, five or more lines, or combinations thereof. In some examples, the abbreviated indicator shape of indicator 410 is consistently overlaid on the display image 350 over a period of time, such as the period during which the photo capture software application is running on the mobile device 205 and / or the period during which the under-display camera 220 is activated. In some examples, the abbreviated indicator shape of indicator 410 blinks on and off periodically during a time period. In some examples, the abbreviated indicator shape of indicator 410 fades in and out periodically during a time period. In some examples, at least a subset of the abbreviated indicator shapes of indicator 410 are animated, for example, by periodically expanding, contracting, pulsating, or any combination thereof.

[0084]

[0098] In some examples, the mobile device 205 displays an indicator 410 overlaid on the display image 350 by modifying the display image 350 to include the indicator 410. The modified display image may be a copy of the merged image with the indicator 410. In some examples, the mobile device 205 blends the display image 350 and the indicator 410 it overlays using a blending effect, such as one of the blending effects discussed with respect to indicator areas 310A, 310B, and 310C.

[0085]

[0099] The mobile device 205 in Figure 4 also includes an image capture UI element 320 and an image capture hardware button 330. These elements may function similarly to the image capture UI element 320 and image capture hardware button 330 in Figures 3A, 3B, and 3C.

[0086]

[0100] In some cases, the indicator 410, or any such indicator, may be permanently overlaid on the display image 350 on the display in the area of ​​the display through which the camera 220 receives incident light. In some cases, the indicator 410, or any such indicator, may be hidden by default and may be overlaid on the display image 350 on the display in response to the reception of input. The input may be a touch on the touch-sensitive touchscreen surface of the display, a predefined gesture across the touch-sensitive touchscreen surface of the display, a hover over the display, a predefined gesture on the display, or voice input by speaking a predefined phrase recorded by the microphone. For example, a user of the mobile device 205 may trigger the mobile device 205 to display the indicator 410 by touching the touch-sensitive touchscreen surface of the display 210. The user may speak a phrase that can act as voice input, such as "Where is my camera?" or a similar phrase. The microphone of the mobile device 205 (or a microphone coupled to the mobile device 205) may record a user saying this phrase as voice input, and the mobile device 205 (or a remote voice-to-text interpretation server) may recognize that the voice input is saying a predefined phrase that triggers the display of indicator 410, and may trigger the mobile device 205 to display indicator 410.

[0087]

[0101] Figure 5 is a front view of a mobile device 205 having an under-display camera 220, the display 210 of which displays a circular indicator 510 on its camera 220 and an area selection user interface (UI) element 520 for selecting a selectable area 540 on its camera 220. The legend 290 in Figure 2A is also included in Figure 5. As with the mobile device 205 in Figures 3A, 3B, 3C, and 4, the mobile device 205 in Figure 5 is shown using its display 210 to display a display image 350. The display image 350 may be based on an image captured by the camera 220. The display image 350 may be temporarily displayed on the display 210 as a preview image to show the user of the mobile device 205 how the scene from which the camera 220 receives light is framed within the camera 220's field of view. The mobile device 205 displays a circular indicator 510 overlaid on the display image 350. The circular indicator 510 indicates an area located on the camera 220, which is not filled inside.

[0088]

[0102] In some software applications and associated user interfaces running on the mobile device 205, it may be useful to select certain parts of an image. An image capture software application and its associated user interface may be configured to allow the selection of areas of a displayed image 350. For example, an image capture software application may be configured to select a selectable area 540 of the displayed image 350 in response to a touch input or touch gesture input to a touchscreen within or around the selectable area 540 of the displayed image 350. In some cases, an image capture software application may be configured to select a selectable area 540 of the displayed image 350 in response to a hover input, pointing input, or hover gesture input in which the user hovers (along the Z-axis) and / or makes a gesture (along the Z-axis) over the selectable area 540. In some cases, an image capture software application may be configured to select a selectable area 540 of the displayed image 350 in response to a voice input in which a predefined phrase recorded by a microphone is spoken. For example, a user of the mobile device 205 may speak a phrase such as "select an area on the camera" or a similar phrase, which can act as voice input. The microphone of the mobile device 205 (or a microphone coupled to the mobile device 205) may record the user saying this phrase as voice input, and the mobile device 205 (or the remote voice-to-text interpretation server) may recognize that the voice input is saying a predefined phrase that triggers the selection of the selectable area 540 and trigger the mobile device 205 to select the selectable area 540.

[0089]

[0103] In some examples, selecting an area of ​​the preview image causes the mobile device 205 to instruct the camera 220 to perform an autofocus operation to adjust the camera's focus to improve focus on the portion of the shooting scene depicted in the selected selectable area 540 of the displayed image 350. This provides an intuitive interface that allows the user of the mobile device to correct the camera's focus, for example, from focusing on an object in the background of the scene to focusing on an object in the foreground of the scene, or vice versa. For example, in the displayed image 350 shown on the display 210 in Figure 4, the selectable area 540 depicts a portion of a person's face. The user may provide input in the selectable area 540 of the display 210 by touching the touch-sensitive touchscreen surface of the display 210 in the selectable area 540 of the display 210, by hovering over the surface of the display 210 in the selectable area 540 of the display 210, by pointing to the selectable area 540 of the display 210, by making gestures toward and / or around the selectable area 540 of the display 210, or by other means. This input may be called touch input, hover input, pointing input, gesture input, or another type of input, depending on the type of interaction with the display 210. When the mobile device 205 detects this input, it identifies that the coordinates of the input match the coordinates of the display 210, and then displays an area of ​​a preview image that depicts a person's face. The mobile device 205 instructs the camera 220 to adjust its focus to improve the focus on the person's face, for example, by shifting from a first focus setting that optimizes the focus on another object in the scene to a second focus setting that optimizes the focus on the person's face. The camera 220 can then capture a second image while the camera 220's focus is set to the second focus setting. Due to the change in focus, the second image may be more in focus on the person's face than the displayed image 350.A person's face may appear clearer and sharper in the second image than in the first image (displayed as image 350).

[0090]

[0104] In some examples, by selecting an area or region of the display image 350, such as a selectable area 540, the mobile device 205 can instruct the camera 220 to set one or more settings associated with one or more properties of the camera 220 to improve the application of one or more properties based on the portion of the shooting scene depicted in the selected area or region of the display image 350. For example, one or more properties may include white balance, black level compensation, gamma correction, gain, color correction, color saturation, noise filter, contrast control, brightness control, tone mapping, sharpness, blur, red-eye correction, or any combination thereof. The camera 220 can then capture a second image while one or more settings associated with one or more properties of the camera 220 are set. Assuming the selected region or area is the selectable area 540, one or more settings are set based on the selection of the selectable area 540, so the second image may use settings that improve the rendering of the human face compared to the rendering of the human face in the display image 350. The human face may appear clearer and sharper in the second image than in the display image 350.

[0091]

[0105] Since the camera 220 is located below the display 210 along the Z-axis, the reception of some inputs, such as input for selecting certain areas of the displayed image 350, may obstruct the view of the camera 220. Such inputs may include touch input, touch gesture input, hover input, pointing input, hover gesture input, or any other type of input discussed herein. For example, while the under-display camera 220 is active, a user providing touch input or touch gesture input by interacting with the touch-sensitive touchscreen surface of the display 210 may at least partially obstruct the view of the camera 220 by touching a portion of the display 210 positioned above the under-display camera 220 with the user's finger, stylus, hand, arm, or pointer. Even if the touch input is received in a particular area of ​​the display 210 that is not directly above the camera 220, a portion of the user's arm or hand may reach above the camera 220 and thus at least partially obstruct the view of the camera 220 in order to reach a particular area of ​​the display 210. Similarly, the user may provide hover input, pointing input, or hover gesture input by hovering over the display 210, pointing at the display 210, and / or making gestures on the display 210 with the user's hand and / or pointing device. Such hover input, pointing input, and hover gesture input may similarly obstruct the camera 220 by blocking light from reaching areas of the display that light would otherwise pass through before reaching the camera 220.

[0092]

[0106] In some cases, occlusion of the camera 220's view may refer to the camera 220 being steered away from the target field of view (FOV). The mobile device 205 may determine the target FOV based on the software application being run by the mobile device 205. For example, if the mobile device 205 is running a video conferencing software application, the mobile device 205 may determine that the target FOV is some FOV of the camera 220 that includes a rendering of the user's face. If the mobile device 205 is moved (e.g., accidentally dropped), the camera 220's FOV no longer conforms to the predetermined criteria for the target FOV in that the camera 220's FOV no longer includes a rendering of the user's face. In some cases, the mobile device 205 may treat the camera 220's FOV as not matching the target FOV during any given time, such as the time the camera 220 was occluded.

[0093]

[0107] The mobile device 205 in Figure 5 includes an area selection hardware button 530. The mobile device 205 may receive an area selection input when the area selection hardware button 530 is pressed. In some cases, the area selection hardware button 530 may be replaced and / or supplemented by another hardware user interface that receives an image capture input. In response to receiving an area selection input, the mobile device 205 may select a selectable area 540 of the display image 350 displayed on the display 210.

[0094]

[0108] The mobile device 205 also includes an area selection UI element 520 displayed on the user interface (UI) area of ​​the display 210, which is overlaid on the display image 350. The mobile device 205 may receive area selection input at the area selection user UI element 520 of the display 210. Area selection input may include touch input, touch gesture input, hover input, hover pointing input, hover gesture input, or any other type of input discussed herein. For example, touch input or touch gesture input may include a touch on the touch-sensitive touchscreen surface of the display 210 and receive a touch area at the area selection user UI element 520 of the display 210. Hover input may include a user hover (on the Z-axis) over the area selection user UI element 520 of the display 210. In some cases, voice input may be used. In some cases, an image capture software application may be configured to select a selectable area 540 of the display image 350 in response to voice input in which a predefined phrase is spoken, recorded by a microphone as discussed herein.

[0095]

[0109] As discussed above, by selecting a selectable area 540 of the display image 350 using area selection hardware button 530 and / or area selection UI element 520, the mobile device 205 can instruct camera 220 to set one or more settings associated with one or more properties of camera 220 to improve the application of one or more properties based on the portion of the shooting scene depicted in the selectable area 540 of the display image 350. For example, by selecting a selectable area 540 of the display image 350 using area selection input, the mobile device 205 can instruct camera 220 to adjust its focus to improve the focus on the portion of the shooting scene depicted in the selectable area 540 of the display image 350. Since the user can select a selectable area 540 of the display image 350 by pressing the area selection hardware button 530 and / or by providing input in area selection UI element 520, the user no longer needs to obstruct camera 220 to select a selectable area 540. The selectable area 540 shown in Figure 5 includes a portion of the display 210 directly positioned on the camera 220, but the selectable area does not have to include a portion of the display 210 directly positioned on the camera 220. In some cases, the selectable area 540 of the display image 350 may include a portion of the display image 350 that the user may need to reach on the camera 220 to touch it, hover over it, point to it, make a gesture toward it, make a gesture around it, or otherwise provide input to it.

[0096]

[0110] To clarify that the selection area 540 of the displayed image 350 is selected upon receiving input in the area selection UI element 520, the exemplary area selection UI element 520 in Figure 5 may include the word “select” on a visual copy of the selectable area 540 of the image. Alternatively, an input indicator (e.g., an arrow indicator, a hand shape indicator, a palm shape indicator, etc.) may be provided to instruct the user to touch the selectable area. The visual copy of the selectable area 540 of the displayed image 350 may be modified to fit the size of the area selection UI element 520 (e.g., downscale, upscale, scaling, crop, stretch, squish, skew, distort, or a combination thereof). In some examples, the area selection UI element 520 may function as a virtual button, and when any part of the area selection UI element 520 receives input such as touch, hover, point, gesture, any input type discussed herein, or a combination thereof, the entire selectable area 540 may be selected. In some examples, a visual copy of the selectable area 540 of the displayed image 350 within the area selection UI element 520 may function as an offset version of the selectable area 540. A user providing input to a specific portion of the visual copy of the selectable area 540 of the displayed image 350 within the area selection UI element 520 may select that specific portion of the selectable area 540 of the displayed image 350. In some examples, the area selection UI element 520 may control the cursor. A user may move touch input on the area selection UI element 520 by swiping, sliding, or otherwise moving the cursor. A user may also use swipe or slide gestures while hovering over the area selection UI element 520 to move the cursor. Selection of the selectable area 540 may be performed by moving the cursor to the selectable area 540 of the displayed image 350 and tapping the area selection UI element 520, pressing a button such as the area selection hardware button 530, using a pointing movement gesture, or triggering another hardware interface element.In some examples, the mobile device 205 may also allow the user to select a selectable area 540 of the displayed image 350 by drawing a shape around the selectable area 540 using a cursor. In some examples, the mobile device 205 may also allow the user to select a selectable area 540 of the displayed image 350 by drawing a shape around the selectable area 540 using touch input, hover input, gesture input, or another form of input around the selectable area 540.

[0097]

[0111] The mobile device 205 in Figure 5 also includes an image capture UI element 320 and an image capture hardware button 330. These elements may function similarly to the image capture UI element 320 and image capture hardware button 330 in Figures 3A, 3B, 3C, and 4. Although the image capture hardware button 330 and area selection hardware button 530 are shown along a single side of the mobile device 205, one or both may be located along different parts of the mobile device 205. For example, one or both may be located along the back, top, or bottom of the mobile device 205. Any other hardware interface elements used to position the image capture hardware button 330 and / or area selection hardware button 530 may similarly be located along the back, top, or bottom of the mobile device 205.

[0098]

[0112] Figure 6 is a front view of a mobile device 205 that compensates for the occlusion of the under-display camera 220 by the user's hand 605 during touch, hover, or gesture input. The legend 290 in Figure 2A is also included in Figure 6. As with the mobile device 205 in Figures 3A, 3B, 3C, 4, and 5, the mobile device 205 in Figure 6 is shown using its display 210 to display a display image 350. The display image 350 may be based on an image captured by the camera 220. The display image 350 may be temporarily displayed on the display 210 as a preview image to show the user of the mobile device 205 how the scene from which the camera 220 receives light is framed during the view of the camera 220.

[0099]

[0113] Hand 605 is shown in Figure 6, obstructing the under-display camera 220 of the mobile device 205. Hand 605 has its fingers fully extended, with the fingertips positioned over the camera 220, blocking light from the scene from reaching the camera 220. The fully extended fingers of hand 605 may be touching the touch input area on the touch-sensitive touchscreen surface of the display 210, hovering over the input area of ​​the display 210, pointing at the input area of ​​the display 210, making gestures toward the input area of ​​the display, making gestures around the input area of ​​the display 210, or any combination thereof. The input area of ​​the display 210 may be the selectable area 540 in Figure 5, or another selectable area of ​​the display image 350. This input to the input area of ​​the display may be used to select a selectable area of ​​the display image 350, such as the selectable area 540 of the display image 350. The input area includes an area of ​​a display 210 positioned on the camera 220 through which incident light reaching the camera 220 passes. The input area may be positioned on the camera 220, adjacent to the camera 220, and / or within a predetermined radius of the camera 220.

[0100]

[0114] Figure 6 shows a timeline 620 of image frames captured by camera 220. The timeline 620 of image frames captured by camera 220 includes a first image frame 610A, a second image frame 610B, a third image frame 610C, a fourth image frame 610D, and a fifth image frame 610E. In some examples, image frames 610A-E in timeline 620 may be image frames captured sequentially by camera 220. In some examples, one or more additional image frames may exist between any two of the image frames in timeline 620. Image frames 610A-E in timeline 620 depict a scene similar to the scene depicted in display image 350, but possibly including occlusion and / or other forms of deformation.

[0101]

[0115] In timeline 620, the first image frame 610A depicts a scene in which a person is in the foreground in front of a background that includes buildings, grass, and trees. The first image frame 610A is unobstructed. The second image frame 610B is captured by camera 220 after the first image frame 610A. The second image frame 610B is partially obscured by occlusion 630A, which appears as a black or dark area covering a portion of the second image frame 610B. Occlusion 630A can be caused, for example, when the hand 605 is moving and the extended fingers of the hand 605 approach the view of camera 220. The hand 605 blocks some of the light from the scene that would otherwise pass through display 210 and reach camera 220. The third image frame 610C is captured by camera 220 after the second image frame 610B. The third image frame 610C is completely obscured by occlusion 630B and therefore completely covered in black or dark. Occlusion 630B can be caused, for example, when the hand 605 is moving, and the extended fingers of the hand 605 completely obstruct the view of the camera 220. The hand 605 blocks most or all of the light from the scene that would otherwise pass through the display 210 and reach the camera 220. The fourth image frame 610D is captured by the camera 220 after the third image frame 610C. The fourth image frame 610D is partially obscured by occlusion 630C, which appears as a black or dark area covering the portion of the fourth image frame 610D. The hand 605 blocks some of the light from the scene that would otherwise pass through the display 210 and reach the camera 220. Occlusion 630C can be caused, for example, when the hand 605 is moving, and the extended fingers of the hand 605 leave the view of the camera 220. The fifth image frame 610E is captured by camera 220 after the fourth image frame 610D.The fifth image frame 610E depicts a similar scene to the first image frame 610A, but some minor changes have occurred in the scene due to the passage of time between the captures of the first image frame 610A and the fifth image frame 610E. For example, the scene depicted in the fifth image frame 610E includes birds in the air that are not present in the first image frame 610A.

[0102]

[0116] When the second image frame 610B, the third image frame 610C, and / or the fourth image frame 610D are displayed on the display 210 as preview images, occlusions 630A-C will be displayed on the display 210. In some cases, the user may find the presence of occlusions 630A-C in the preview images to be unhelpful. In some examples, for the duration that the camera 220 is at least partially occluded, the mobile device 205 displays the last unoccluded image frame from before the occlusion. In the timeline 620, the last unoccluded image frame from before the occlusion is the first image frame 610A, indicated by arrow 640. In some examples, for the duration that the camera 220 is completely occluded (as in the occlusion 630B of the third image frame 610C), the mobile device 205 displays the last partially unoccluded image frame from before the full occlusion 630B. In timeline 620, the last partially unobstructed image frame before full occlusion 630B is the second image frame 610B. In some examples, the mobile device 205 displays the last image frame during the duration in which less than a predetermined maximum threshold percentage of the camera 220's view is obscured, while more than a predetermined maximum threshold percentage of the camera 220's view is obscured.

[0103]

[0117] In some cases, touch input or other input may be received while the camera 220 is at least partially obscured. In some examples, the mobile device 205 may receive touch input in response to an extended finger of hand 605 touching the touch input area on the touch-sensitive touchscreen surface of the display 210 at a time after the capture of the first image frame 610A and before the capture of the fifth image frame 610E. The mobile device 205 may select a selectable area of ​​the image, such as the selectable area 540 in Figure 5, based on the location on the display 210 where the touch input was received, which at least partially overlaps with the selectable area 540. The mobile device 205 may instruct the camera 220 to set one or more settings for one or more properties in order to adjust the application of one or more properties of the camera 220 based on the portion of the shooting scene depicted in the selected area in the next unobscured image frame after the occlusion has been removed. The next unobscured image frame after the occlusion is the fifth image frame 610E in the timeline 620. Therefore, the mobile device 205 may instruct the camera 220 to set one or more settings for one or more properties in order to adjust the application of one or more properties of the camera 220 based on the portion of the shooting scene depicted in the selected area in the last unobstructed image frame before occlusion.

[0104]

[0118] In one exemplary case, while camera 220 is at least partially obscured, the mobile device 205 may still display the first image frame 610A, and the user may tap a selectable area of ​​the first image frame 610A depicting the user's face with the user's hand 605. The mobile device 205 receives touch input on the touch-sensitive touchscreen surface of display 210 in the portion of the display that shows the selectable area of ​​the first image frame 610A. Upon detecting this touch input, the mobile device 205 identifies that the coordinates of the touch input coincide with the coordinates of display 210, and then displays the area of ​​the first image frame 610A depicting the user's face. The mobile device 205 instructs camera 220 to adjust its focus to improve the focus on the user's face for the next unobstructed image frame after the occlusion, a fifth image frame 610E. The mobile device 205 shifts from a first focus setting that optimizes focus on another object in the scene to a second focus setting that optimizes focus on the user's face. When the camera 220 captures the fifth image frame 610E, the camera 220's focus may already be set to the second focus setting that optimizes focus on the user's face.

[0105]

[0119] Figure 7 is a front view of a mobile device 205 having multiple under-display cameras 220A-220B and under-display sensors 260A-260B, where the display 210 shows an interface area 770 for controlling the multiple under-display cameras 220A-220B and under-display sensors 260A-260B. The legend 290 in Figure 2A is also included in Figure 7. Like the mobile device 205 in Figure 2D, the mobile device 205 in Figure 7 includes a first under-display camera 220A and a second under-display camera 220B. The first under-display camera 220A and the second under-display camera 220B are each any type of camera and may have any of the camera properties discussed herein with respect to camera 220, the first under-display camera 220A in Figure 2D, the second under-display camera 220B in Figure 2D, image capture device 105A, image processing device 105B, image capture and processing system 100, any other camera discussed herein, or any combination thereof. The first under-display camera 220A may also be referred to as the first camera 220A. The second under-display camera 220B may also be referred to as the second camera 220B.

[0106]

[0120] As with the mobile device 205 in Figures 3A, 3B, 3C, 4, 5, and 6, the mobile device 205 in Figure 7 is shown using its display 210 to display a display image 350. The display image 350 may be based on an image captured by a first camera 220A or a second camera 220B. The display image 350 may be temporarily displayed on the display 210 as a preview image to show the user of the mobile device 205 the field of view of the first camera 220A or the field of view of the second camera 220B.

[0107]

[0121] The second camera 220B in Figure 7 appears in the same position as the camera 220 in Figure 5. The display 210 shows a circular indicator 510B around the second camera 220B and within the selectable area 540B on the second camera 220B. The second indicator 510B in Figure 7 is circular and appears in the same way as the indicator 510 in Figure 5. The second selectable area 540B on the second camera 220B in Figure 7 is also circular and appears to have the same shape, size, and position as the selectable area 540 on the camera 220 in Figure 5. In other examples, the second camera 220B, the second indicator 510B, and / or the second selectable area 540B may differ from those in Figure 5.

[0108]

[0122] The first camera 220A in Figure 7 appears near the top of the display 210, similar to the first camera 220A in Figure 2D, camera 220 in Figure 3A, camera 220 in Figure 3B, camera 220 in Figure 3C, and camera 220 in Figure 4. The display 210 is a rounded rectangle around the first camera 220A, displaying the first indicator 510A within the first selectable area 540A on the first camera 220A. The first selectable area 540A on the first camera 220A in Figure 7 is a large, rounded rectangle that includes most of the sky, as depicted in the display image 350. In other examples, the second camera 220B, the second indicator 510B, and / or the second selectable area 540B may differ from those in Figure 5.

[0109]

[0123] The interface area 770 appears as a shaded, semi-transparent area overlaid on the image displayed on the display 210. The interface area 770 includes an image capture UI element 320, a first area selection UI element 720A, a second area selection UI element 720B, and a sensor control UI element 740. The first area selection UI element 720A and the second area selection UI element 720B in Figure 7 function similarly to the area selection UI element 520 in Figure 5. Similarly, the first area selection hardware button 730A and the second area selection hardware button 730B in Figure 7 function similarly to the area selection hardware button 530 in Figure 5. When the mobile device 205 receives input at the first area selection UI element 720A, it selects the first selectable area 540A. When the mobile device 205 receives input at the first area selection hardware button 730A, it selects the first selectable area 540A. A user who wishes to choose to have the sky drawn in the display image 350 can provide input to the first area selection UI element 720A and / or to the first area selection hardware button 730A to do so. When the mobile device 205 receives input in the second area selection UI element 720B, it selects the second selectable area 540B. When the mobile device 205 receives input in the second area selection hardware button 730B, it selects the second selectable area 540B. A user who wishes to have a human face drawn in the display image 350 can provide input to the second area selection UI element 720B and / or to the second area selection hardware button 730B to do so.

[0110]

[0124] In some cases, one of the first camera 220A or the second camera 220B may be disabled for a specific purpose or software application. In some cases, one of the first camera 220A or the second camera 220B may be activated for a specific purpose or software application. For example, if the mobile device 205 is running video conferencing software that is conducting a video conference, the first camera 220A may be disabled for use in the video conference, and the second camera 220B may be activated for use in the video conference. During a video conference, the user of the mobile device 205 is video conferencing with a second user. The user of the mobile device 205 is generally looking at the face of the second user displayed on the display 210. The face of the second user may be displayed approximately in the center of the display 210. During such a video conference, the user of the mobile device 205 will naturally be looking towards the center of the display 210 because the user is looking at the face of the other user. Because the second camera 220B is closer to the center of the display 210 than the first camera 220A, a user looking at the faces of other users displayed on the display 210 is also looking directly at the second camera 220B or at a point closer to the second camera 220B than to the first camera 220A. While a user is looking at the faces of other users displayed on the display 210, the image captured by the second camera 220B therefore depicts the user making eye contact with the second camera 220B or at a point close to the second camera 220B. By using the second camera 220B instead of the first camera 220A during a video conference, the image of the user making eye contact, captured by the second camera 220B, is sent to the second user's device and displayed on the second user's device. When the image captured by the second camera 220B is displayed on the second user's device, and then the second user views the image displayed on their device, it appears as if they are making eye contact with the second user.In video conferencing, there is generally no need to touch the center of the display 210, hover over it, or otherwise interact with it; therefore, the likelihood of occlusion of the second camera 220B is low during video conferencing. Thus, using the second camera 220B for video conferencing improves the occurrence of eye contact between the user and the second user. This occurrence of eye contact is generally not achievable with more offset cameras, such as the first camera 220A, except through image manipulation that corrects the rendering of the user's eyes. Such image manipulation can look unnatural and produce an undesirable uncanny valley effect.

[0111]

[0125] On the other hand, a software application that anticipates and / or expects user selection of an area of ​​the display image 350 may use the first camera 220A and disable the second camera 220B. The first camera 220A is closer to the top surface of the mobile device 205 than the second camera 220B. Therefore, the first camera 220A is less likely to be obstructed than the second camera 220B by the user holding the mobile device 205 and the bottom surface of the mobile device 205 being closer to the user's body than the top surface of the mobile device 205. For example, a software application that allows a user to draw on or otherwise edit a selfie of themselves may use the first camera 220A and disable the second camera 220B, since such a software application anticipates and / or expects input for selecting an area of ​​the display image 350.

[0112]

[0126] In some cases, the mobile device 205 may switch which of the cameras is active and which is disabled based on camera occlusion. For example, if the first camera 220A is active and the mobile device 205 detects an occlusion similar to one of the occlusions 630A-C that is blocking light from being received by the first camera 220A, the mobile device 205 may disable the first camera 220A and activate the second camera 220B. Similarly, if the second camera 220B is active and the mobile device 205 detects an occlusion similar to one of the occlusions 630A-C that is blocking light from being received by the second camera 220B, the mobile device 205 may disable the second camera 220B and activate the first camera 220A.

[0113]

[0127] Similar to the under-display sensor 260 of the mobile device 205 in Figure 2D, the mobile device 205 in Figure 7 includes a first under-display sensor 260A and a second under-display sensor 260B. The first under-display sensor 260A and the second under-display sensor 260B are each of any type of sensor and may have any of the sensor properties discussed herein with respect to the under-display sensor 260, any other camera or sensor discussed herein, or any combination thereof. The mobile device 205 can trigger use and / or control of one or both of sensors 260A and / or sensor 260B when it receives input in the sensor control UI element 740 and / or when it receives input in the sensor control hardware button 775. Figure 7 shows a single sensor control UI element 740 and a single sensor control hardware button 775, but in some cases the mobile device 205 may include a separate sensor control UI element 740 for each of the sensors 260A to 260B, and / or a separate sensor control hardware button 775 for each of the sensors 260A to 260B. When the mobile device 205 receives input in the sensor control UI element 740 and / or sensor control hardware button 775 corresponding to the first sensor 260A, it can select a selectable area of ​​the display image 350 on the first sensor 260A. When the mobile device 205 receives input in the sensor control UI element 740 and / or sensor control hardware button 775 corresponding to the second sensor 260B, it can select a selectable area of ​​the display image 350 on the second sensor 260B.

[0114]

[0128] The mobile device 205 may activate the first sensor 260A and the second sensor 260B simultaneously or within the same time window so that the mobile device 205 can collect more sensor data and obtain a more complete and accurate understanding of the environment. For example, if the first sensor 260A is a RADAR sensor and the second sensor 260B is a LIDAR sensor, the mobile device 205 can activate both sensors 260A-260B and use the sensor measurement data from both sensors 260A-260B and / or from cameras 220A-220B to obtain detailed and accurate depth information of the scene depicted in the display image 350. One of the different sensors 260A-260B may be activated and disabled at different times, for example, based on what software application is running on the mobile device 205. One software application may trigger the mobile device 205 to activate the first sensor 260A but disable the second sensor 260B. Another software application may trigger the mobile device 205 to activate the second sensor 260B but disable the first sensor 260A. Another software application may trigger the mobile device 205 to activate both the first sensor 260A and the second sensor 260B. Another software application may trigger the mobile device 205 to disable both the first sensor 260A and the second sensor 260B. In some situations, the mobile device 205 may detect occlusion of one of the sensors 260A-260B, in which case the mobile device 205 may disable the occluded sensor and activate the other (unoccluded) sensor. For example, the mobile device 205 may detect occlusion of one of the sensors 260A-260B when the sensor returns a sensor measurement indicating the detection of an object less than a threshold distance from the sensor.In some cases, one of the sensors 260A-260B may be located within the threshold distance of one of the cameras 220A-220B. If the mobile device 205 detects occlusion of such a camera (for example, as in occlusions 630A-C in Figure 6), the mobile device 205 can assume that the sensor is located within the threshold distance of the camera. For example, in Figure 7, sensors 260A-260B may be within the threshold distance of the first camera 220A. If the mobile device 205 detects occlusion of the first camera 220A, the mobile device 205 can assume that sensors 260A-260B are also occluded. This type of occlusion detection may be useful when one or more of the sensors 260A-260B are of a sensor type such as a microphone, for which detecting occlusion can be difficult.

[0115]

[0129] Figure 8 is a flowchart illustrating an image processing technique 800. The image processing technique 800 shown in the flowchart of Figure 8 can be implemented by a device. The device may be a mobile device 205, an image capture and processing system 100, an image capture device 105A, an image processing device 105B, one or more network servers of a cloud service, a computing system 1000, or any combination thereof.

[0116]

[0130] In operation 805, the device receives an image captured by camera 220. Camera 220 captures an image based on incident light 245 received by camera 220. Camera 220 is positioned relative to the display 210 to receive incident light 245 that passes through a portion of the display 210 before reaching camera 220. The portion of the display 210 may be a light-passing area of ​​the display 210, for example, in the light-passing area 285. Reaching the camera may include hitting the image sensor 130 of camera 220. The camera may be positioned adjacent to the display along an axis perpendicular to the flat surface of the display, such as the Z-axis in the legend 290 of Figure 2A in the context of the mobile device 205 in Figure 2A.

[0117]

[0131] The display 210 may be a touchscreen display. The device can use the touch-sensitive surface of the touchscreen display to detect touch input or touch gesture input received on the touch-sensitive surface of the touchscreen display. The display 210 may include one or more sensors 260. One or more sensors 260 may include one or more active depth sensors. The device can use one or more active depth sensors to detect hover input, hover pointing input, or hover gesture input on the display 210.

[0118]

[0132] In operation 810, the device displays a display image 350 based on an image on the display 210. In some cases, the device generates a display image 350 based on an image captured by the camera 220 before displaying the display image 350 on the display 220. Generating the display image 350 may include correcting the size of the image using at least one of scaling and / or cropping. Scaling may include upscaling, downscaling, and / or stretching. Correcting the size of the image may include correcting the size of the image so that the display image 350 is displayed on the display 210 outside the indicator area. Generating the display image 350 may also include processing the image by correcting at least one of the following, for example, luminance, contrast, color saturation, tone, white balance, black balance, levels, gamma correction, gain, color correction, noise filter, sharpness, blur, and red-eye correction.

[0119]

[0133] In operation 815, the device displays an indicator overlaid on the display image 350 in the indicator area of ​​the display 210 while the display 210 is displaying the display image 350. The indicator area includes at least a subset of the display. Examples of indicators in operation 815 include indicator 310A, indicator 310B, indicator 310C, indicator arrow 340, indicator 410, and indicator 510, and / or combinations thereof. The indicator area of ​​the display may be at least as large as the camera lens. The indicators may be at least as large as the camera lens.

[0120]

[0134] In some cases, the device may automatically display an indicator overlaid on the display image 350 in the indicator area. In some cases, the device receives input, and in response to receiving the input, the device displays an indicator overlaid on the display image 350 in the indicator area. The input may include at least one of the following: touching the touch-sensitive touchscreen surface of the display 210, predefined gestures across the touch-sensitive touchscreen surface of the display 210, hovering over the display 210, predefined gestures on the display 210, voice input of speaking a predefined phrase recorded by a microphone, or a combination thereof.

[0121]

[0135] The indicator may include a shape around at least a portion of the display 210. Displaying the indicator may include displaying a color on multiple pixels in the indicator area. The color may be black, white, gray, red, green, blue, yellow, or any mixture of these colors. Displaying the indicator may include displaying a pattern or secondary image separate from the image and display image 350 on multiple pixels in the indicator area. The indicator may be animated. For example, the indicator may blink or pulsate.

[0122]

[0136] While display 210 is displaying display image 350, displaying an indicator overlaid on display image 350 in the indicator area may include modifying display image 360 ​​to merge the indicator with display image 350.

[0123]

[0137] The device can display an image capture UI element 320 on the image capture user interface (UI) area on the display 210. The device can receive input in the image capture UI area. The input may be touch input, hover input, gesture input, or any other input type discussed herein. In response to receiving input in the image capture UI area, the device can receive a second image captured by the camera 220. The device can store the second image in a non-temporary computer-readable storage medium.

[0124]

[0138] The device can receive input in an area selection user interface (UI) area. The input may be touch input, hover input, gesture input, or at least one of any other input types discussed herein. In response to receiving input in the area selection UI area, the device may select a selection area of ​​the display image 350. Selectable areas 540, 540A, and 540B are examples of selection areas. The area selection UI area is separate from the selection area of ​​the display image. The selection area of ​​the display image may include at least a portion of the indicator area. In some cases, the device displays an area selection UI element 320 in the area selection UI area. In some cases, the device generates the area selection UI element 320 to draw a copy of at least a portion of the selection area.

[0125]

[0139] The device may, in response to the selection of a selection area, set image capture parameters to specific settings determined based on the selected area of ​​the displayed image 350. The image capture parameters may include at least one of focus or exposure parameters. The exposure parameters may include at least one of exposure time, analog gain, digital gain, aperture size, ISO, or a combination thereof. The device may, in response to the selection of a selection area, set image processing properties to specific settings determined based on the selected area of ​​the displayed image. The image processing properties may include at least one of brightness, contrast, color saturation, tone, white balance, black balance, black level compensation, levels, gamma correction, gain, color correction, noise filter, sharpness, blur, red-eye correction, or a combination thereof.

[0126]

[0140] The device can receive a second image captured by camera 220 after the first image capture. The device can determine that the second image is obscured by occlusion, such as occlusions 630A-C. In response to determining that the second image is obscured by occlusion, the device can display a display image 350 on display 210. For example, since the second image is obscured by occlusion, the device can choose to display the display image 350 again on display 210 instead of displaying the second image or a second display image based on the second image. In some cases, the device can receive an input, where occlusion is associated with the reception of the input. The input may be at least one of touch input, hover input, gesture input, or any other input type discussed herein. The device can select a selection area of ​​the display image based on the input. The device can determine a setting based on the selection area of ​​the display image and apply the setting. While the setting is applied, and after the capture of the second image, the device can receive a third image captured by camera 220.

[0127]

[0141] In some cases, the device can accommodate multiple cameras. In some examples, the device can receive a second image captured by a second camera. The second camera can capture a second image based on secondary incident light received by the second camera. The second camera may be positioned relative to the display 210 to receive secondary incident light that passes through a second portion of the display 210 (e.g., a second light-passing area of ​​the display 210) before reaching the second camera. Reaching the second camera may include striking the second image sensor of the second camera. In some examples, the device can determine that the second image is obscured by occlusion and, in response to this determination, can display the display image 350 on the display 210. In some examples, the device can determine depth information corresponding to one or more objects depicted in the image and the second image by processing the image and the second image, for example using stereoscopic image processing.

[0128]

[0142] Figure 9 is a flowchart illustrating an image processing technique 900. The image processing technique 900 shown in the flowchart of Figure 9 can be implemented by a device. The device may be a mobile device 205, an image capture and processing system 100, an image capture device 105A, an image processing device 105B, one or more network servers of a cloud service, a computing system 1000, or any combination thereof.

[0129]

[0143] In operation 905, the device receives an image captured by camera 220. Camera 220 captures an image based on incident light 245 received by camera 220. Camera 220 is positioned relative to the touchscreen display 210 to receive incident light 245 that passes through the light-passing area 285 of the touchscreen display 210 before reaching camera 220. Reaching the camera may include hitting the image sensor 130 of camera 220. The light-passing area 285 of the touchscreen display 210 in operation 905 may be an example of a portion of the display in operation 805.

[0130]

[0144] In operation 910, the device displays an image-based display image 350 on the touchscreen display 210.

[0131]

[0145] In operation 915, the device displays area selection user interface (UI) elements on the touchscreen display 210. The area selection UI elements in operation 915 may include, for example, area selection UI element 520 in Figure 5, area selection UI element 720A in Figure 7, or area selection UI element 720B in Figure 7.

[0132]

[0146] In operation 920, the device receives touch input to the touchscreen display 210 in an area selection input area that overlaps with an area selection UI element. In some cases, the touch input may be supplemented by or replaced by a hover input.

[0133]

[0147] In operation 925, the device selects a selection area of ​​the display image 350 in response to receiving touch input. The selection area may be a selectable area that includes at least a portion of the light-transmitting area, such as the selectable area 540 in Figure 5, the selectable area 540A in Figure 7, or the selectable area 540B in Figure 7.

[0134]

[0148] In operation 930, the device, in response to the selection of a selection area, sets the camera properties of camera 220 to a setting determined based on the selected area of ​​the display image 350. The camera properties of camera 220 include at least one of image capture properties or image processing properties. The image capture properties may include at least one of focus and exposure parameters. The exposure parameters may include at least one of exposure time, analog gain, digital gain, aperture size, ISO, or a combination thereof. The image processing properties may include at least one of brightness, contrast, color saturation, tone, white balance, black balance, black level compensation, level, gamma correction, gain, color correction, noise filter, sharpness, blur, red-eye correction, or a combination thereof.

[0135]

[0149] In some cases, at least a subset of the image processing techniques 800 and 900 shown in the flowcharts of Figures 8 and 9 may be performed remotely by one or more network servers of a cloud service. In some examples, the processes described herein (e.g., image processing techniques 800 and 900 and / or other processes described herein) may be performed by computing devices or apparatus. In one example, image processing techniques 800 and 900 may be performed by the image capture device 105A in Figure 1. In another example, image processing techniques 800 and 900 may be performed by the image processing device 105B in Figure 1. Image processing techniques 800 and 900 may also be performed by the image capture and processing system 100 in Figure 1. Image processing techniques 800 and 900 may also be performed by any of the mobile devices 205 in Figures 2A, 2B, 2C, 3A, 3B, 3C, 4, 5, 6, and / or 7. Image processing techniques 800 and 900 may be implemented by a computing device having the architecture of the computing system 1000 shown in Figure 10. The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, a computing device for an autonomous vehicle or autonomous vehicle, a robotic device, a television, and / or any other computing device having the resource capacity to implement the processes described herein, including image processing techniques 800 and 900.In some cases, a computing device or apparatus may include a variety of components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0136]

[0150] Components of a computing device can be implemented in a circuit. For example, a component may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits) to perform the various operations described herein, and may also include and / or be implemented using computer software, firmware, or any combination thereof, as well as electronic circuits or other electronic hardware.

[0137]

[0151] The processes shown in the conceptual diagrams and flowcharts of Figures 8 and 9 are organized as logical flowcharts, and their operations represent sequences of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer executable instruction stored in one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which operations are described is not intended to be interpreted as limiting, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0138]

[0152] Furthermore, the processes shown by the conceptual diagrams and flowcharts in Figures 8-9 and / or other processes described herein may be carried out under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively by hardware, or by a combination thereof, on one or more processors. As stated above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-temporary.

[0139]

[0153] Figures 2A–2C, 3A–3C, 4, 5, 6, and 7 all show the mobile device 205, but it should be understood that the mobile device 205 is merely an example of a device that includes and / or implements the technologies discussed herein. The technologies and techniques discussed herein, including image processing techniques 800 and 900, may be implemented by the mobile device 205, the image capture and processing system 100, the image capture device 105A, the image processing device 105B, one or more network servers of the cloud service, the computing system 1000, or any combination thereof.

[0140]

[0154] Figure 10 shows an example of a system for implementing several aspects of the present technology. In particular, Figure 10 shows an example of a computing system 1000 in which any computing device, remote computing system, camera, or system component constituting an internal computing system communicates with each other using connection 1005, which can be any of those components. Connection 1005 may be a physical connection using a bus, such as in a chipset architecture, or a direct connection to processor 1010. Connection 1005 may also be a virtual connection, a networked connection, or a logical connection.

[0141]

[0155] In some embodiments, the computing system 1000 is a distributed system in which the functions described herein can be distributed across one data center, multiple data centers, a peer network, and so on. In some embodiments, one or more of the system components described represent many such components, each performing some or all of the functions described for that component. In some embodiments, the components may be physical or virtual devices.

[0142]

[0156] An exemplary system 1000 includes at least one processing unit (CPU or processor) 1010 and connections 1005 that connect various system components to the processor 1010, including system memory 1015 such as read-only memory (ROM) 1020 and random access memory (RAM) 1025. The computing system 1000 may include a high-speed memory cache 1012 that is directly connected to the processor 1010, very close to it, or integrated as part of it.

[0143]

[0157] Processor 1010 may include an arbitrary general-purpose processor, hardware or software services such as services 1032, 1034, and 1036 stored in memory device 1030 and configured to control processor 1010, and a dedicated processor in which software instructions are incorporated into the actual processor design. Processor 1010 can be a fully self-contained computing system that includes multiple cores or processors, buses, memory controllers, caches, etc. A multicore processor can be symmetrical or asymmetrical.

[0144]

[0158] To enable user interaction, the computing system 1000 includes an input device 1045 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, and voice. The computing system 1000 may also include an output device 1035 that may be one or more of several output mechanisms. In some cases, a multimodal system may allow a user to provide multiple types of inputs and outputs for communication with the computing system 1000. The computing system 1000 may include a communication interface 1040 that can generally control and manage user inputs and system outputs.Communication interfaces include audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple® Lightning® ports / plugs, Ethernet® ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs, Bluetooth® wireless signal transmission, Bluetooth Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, Radio Frequency Identification (RFID) wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, and Wireless Local Area Network (WLA) wireless signal transmission. N) Wired and / or wired or wireless communications can be implemented or facilitated, including receiving and / or transmitting wired and / or wireless communications using transceivers that utilize signal transmission, visible light communications (VLC), worldwide interoperability for microwave access (WiMAX®), infrared (IR) communications wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof. The communication interface 1040 may also include one or more GNSS receivers or transceivers used to determine the position of the computing system 1000 based on the reception of one or more signals from one or more satellites associated with one or more Global Navigation Satellite Systems (GNSS) systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russia-based Global Navigation Satellite System (GLONASS), the China-based BeiDou Navigation Satellite System (BDS), and the Europe-based Galileo GNSS.There are no restrictions on operating on any specific hardware configuration; therefore, the basic features described here can be easily replaced by improved hardware or firmware configurations as they are developed.

[0145]

[0159] Storage device 1030 includes magnetic cassettes, flash memory cards, solid memory devices, digital multipurpose disks, cartridges, floppy disks, flexible disks, hard disks, magnetic tapes, magnetic strips / stripees, any other magnetic storage media, flash memory, memristor memory, any other solid memory, compact disc read-only memory (CD-ROM) optical discs, rewritable compact disc (CD) optical discs, digital video discs (DVD) optical discs, Blu-ray® discs (BDD) optical discs, holographic optical discs, other optical media, Secure Digital (SD) cards, microSecure Digital (microSD) cards, and Memory This may include non-volatile and / or non-temporary and / or computer-readable memory devices, such as Stick® cards, smart card chips, EMV chips, subscriber identification module (SIM) cards, mini / micro / nano / pico SIM cards, other integrated circuit (IC) chips / cards, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electro-erasable programmable read-only memory (EEPROM®), flash EPROM (FLASHEPROM), cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM® / ReRAM), phase-change memory (PCM), spin-transfer torque RAM (STT-RAM), other memory chips or cartridges, and / or combinations thereof, which may be hard disks or other types of computer-readable media capable of storing data accessible by a computer.

[0146]

[0160] The storage device 1030 may include software services, servers, and other services that, when the code defining such software is executed by the processor 1010, cause the system to perform functions. In some embodiments, a hardware service that performs a particular function may include software components stored on a computer-readable medium with respect to the necessary hardware components, such as the processor 1010, connection 1005, and output device 1035, in order to perform that function.

[0147]

[0161] As used herein, the term “computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or transporting instructions and / or data. Computer-readable medium may include non-transient media capable of storing data and not containing carrier waves and / or transient electronic signals that propagate wirelessly or via wired connections. Examples of non-transient media include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as compact discs (CDs) or digital multipurpose discs (DVDs), flash memory, memory, or memory devices. Computer-readable medium may store code and / or machine-executable instructions thereon that may represent procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted using any appropriate means, including memory sharing, message passing, token passing, network transmission, etc.

[0148]

[0162] In some embodiments, computer-readable storage devices, media, and memory may include cable or wireless signals, such as bitstreams. However, as stated, non-transient computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0149]

[0163] Specific details are provided in the above description in order to provide a complete understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that embodiments may be practiced without these specific details. For the sake of clarity of description, in some cases the art may be presented as including individual functional blocks, which include devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments.

[0150]

[0164] Individual embodiments may be described above as processes or methods depicted as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts can describe operations as sequential processes, many operations may occur in parallel or simultaneously. Furthermore, the order of operations can be rearranged. When its operations are completed, the process terminates, but it may have additional steps not shown in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, its termination may correspond to the return value of the function to a called function or main function.

[0151]

[0165] The processes and methods described above may be implemented using computer-executable instructions that are stored or otherwise available from a computer-readable medium. Such instructions may include instructions and data that cause a general-purpose computer, a dedicated computer, or a processing device to perform certain functions or groups of functions, or otherwise configure a general-purpose computer, a dedicated computer, or a processing device to perform certain functions or groups of functions. The portion of computer resources used may be accessible over a network. Computer-executable instructions may include, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created in the methods described above include magnetic or optical disks, flash memory, USB devices with non-volatile memory, and networked storage devices.

[0152]

[0166] Devices implementing the processes and methods described herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the required tasks may be stored in computer-readable or machine-readable media. The processor may perform the required tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mount devices, and standalone devices. The functions described herein may also be embodied in peripherals or add-in cards. Such functions may also be implemented, as a further example, on different processes running on circuit boards within different chips or within a single device.

[0153]

[0167] Instructions, a medium for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described herein.

[0154]

[0168] While the above description has described aspects of this application with reference to specific embodiments thereof, those skilled in the art will recognize that this application is not limited thereto. Therefore, although exemplary embodiments of this application are described in detail herein, it should be understood that, except as limited by the prior art, the inventive concept may be embodied and employed in various ways, and the appended claims should be interpreted to include such variations. The various features and aspects of the applications described above may be used individually or together. Furthermore, the embodiments may be used in any number of environments and applications other than those described herein, without departing from the broader spirit and scope of this specification. Therefore, this specification and the drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be carried out in a different order than described.

[0155]

[0169] Those skilled in the art will understand that the symbols or terms less than ("<") and greater than (">") used herein may be replaced with the symbols less than ("≦") and greater than ("≧"), respectively, without departing from the scope of this specification.

[0156]

[0170] Where a component is described as "configured to perform certain operations," such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor, or other suitable electronic circuits) to perform the operations, or by any combination thereof.

[0157]

[0171] The phrase "connected" refers to any component that is physically connected to another component, either directly or indirectly, and / or any component that communicates with another component, either directly or indirectly (for example, connected to another component via a wired or wireless connection and / or other preferred communication interface).

[0158]

[0172] The wording of a claim that includes "at least one of" a set and / or "one or more" of a set indicates that one member of a set or multiple members of a set (in any combination) satisfy the claim. For example, the wording of a claim that includes "at least one of A and B" means A, B, or A and B. In another example, the wording of a claim that includes "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The wording of "at least one of" a set and / or "one or more" of a set does not limit the set to items enumerated in the set. For example, the wording of a claim that includes "at least one of A and B" could mean A, B, or A and B, and could further include items not enumerated in the set A and B.

[0159]

[0173] The various exemplary logic blocks, modules, circuits, and algorithmic steps described in relation to the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly demonstrate this hardware- and software compatibility, various exemplary components, blocks, modules, circuits, and steps have generally been described above in terms of their function. Whether such functions are implemented as hardware or software depends on the particular application and the design constraints imposed on the overall system. A person skilled in the art may implement the described functions in different ways for each particular application, but such implementation decisions should not be construed as causing a departure from the scope of this application.

[0160]

[0174] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices with multiple applications, including applications in wireless communication device handsets and other devices. Any feature described as a module or component may be implemented together in an integrated logic device, or separately as individual but interoperable logic devices. When implemented in software, the technique may be at least partially realized by a computer-readable data storage medium having program code including instructions, wherein when the instructions are executed, they perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. Computer-readable media may comprise memory or data storage media such as random access memory (RAM) including synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH® memory, and magnetic or optical data storage media. The technique may be at least partially realized by computer-readable communication media, such as propagating signals or radio waves, that carry or transmit program code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer.

[0161]

[0175] The program code may be executed by a processor which may include one or more processors such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other uniform integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors working with a DSP core, or any other such configuration. Accordingly, the term “processor” as used herein may refer to any of the above structures, any combination thereof, or any other structure or device suitable for implementing the techniques described herein. In addition, in some embodiments, the functions described herein may be provided in dedicated software or hardware modules configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0162]

[0176] Exemplary aspects of this disclosure include:

[0163]

[0177] Embodiment 1: An apparatus for image processing, the apparatus comprising a memory and one or more processors coupled to the memory, wherein the one or more processors are configured to receive an image captured by a camera based on incident light received by the camera, display a display image based on the image on a display, the camera being positioned relative to the display to receive incident light passing through a portion of the display before reaching the camera, and display an indicator overlaid on the display image in an indicator area of ​​the display while the display is displaying the display image, wherein the indicator area includes at least a subset of a portion of the display.

[0164]

[0178] Embodiment 2: The apparatus of Embodiment 1, wherein the apparatus is at least one of a mobile device, a wireless communication device, and a camera device.

[0165]

[0179] Embodiment 3: The apparatus according to Embodiment 1 or 2, wherein the apparatus includes at least one of a camera and a display.

[0166]

[0180] Embodiment 4: The apparatus according to any one of Embodiments 1 to 3, wherein at least one of the indicator and the indicator area of ​​the display is at least as large as the area of ​​the camera lens.

[0167]

[0181] Embodiment 5: The apparatus of any one of Embodiments 1 to 4, wherein one or more processors are further configured to receive inputs, and an indicator is displayed on an indicator area in response to the reception of an input.

[0168]

[0182] Embodiment 6: The apparatus of Embodiment 5, wherein the input includes at least one of the following: touching the touch-sensitive touchscreen surface of the display, a predefined gesture across the touch-sensitive touchscreen surface of the display, hovering over the display, a predefined gesture on the display, and voice input of speaking a predefined phrase recorded by a microphone.

[0169]

[0183] Embodiment 7: An apparatus of any one of Embodiments 1 to 6, wherein one or more processors are further configured to generate a display image by modifying the size of the image using at least one of scaling and cropping.

[0170]

[0184] Embodiment 8: The apparatus of Embodiment 7, wherein resizing an image includes resizing an image so that the displayed image is shown on a display other than the indicator area.

[0171]

[0185] Embodiment 9: The apparatus of any embodiment 1 to 8, wherein the indicator includes a shape around at least a portion of the display.

[0172]

[0186] Embodiment 10: An apparatus according to any of Embodiments 1 to 9, wherein the indicator is animated.

[0173]

[0187] Embodiment 11: An apparatus according to any one of Embodiments 1 to 10, wherein while the display is showing a display image, the display of an indicator overlaid on the display image in an indicator area includes modifying the display image so that the indicator merges with the display image.

[0174]

[0188] Embodiment 12: An apparatus according to any of Embodiments 1 to 11, wherein the display is a touchscreen.

[0175]

[0189] Embodiment 13: Any device of Embodiments 1 to 12, further configured to display image capture UI elements on an image capture user interface (UI) area on a display; receive input in the image capture UI area; receive a second image captured by a camera in response to receiving input in the image capture UI area, wherein the input is one of touch input, hover input, and gesture input; and store the second image in a non-temporary computer-readable storage medium.

[0176]

[0190] Embodiment 14: Any device of Embodiments 1 to 13, further configured to receive input in an area selection user interface (UI) area, wherein the input is one of touch input, hover input, and gesture input, and in response to receiving input in the area selection UI area, select a selection area for a display image, wherein the area selection UI area is separate from the selection area for a display image.

[0177]

[0191] Embodiment 15: The apparatus of Embodiment 14, wherein the selection area of ​​the display image includes at least a portion of the indicator area.

[0178]

[0192] Embodiment 16: The apparatus of Embodiment 14 or 15, wherein one or more processors are further configured to set image capture parameters associated with a camera to specific settings determined based on the selected area of ​​the displayed image in response to the selection of a selection area, wherein the image capture parameters include at least one of focus and exposure parameters.

[0179]

[0193] Embodiment 17: An apparatus of any embodiment 14 to 16, wherein one or more processors are further configured to set an image processing property to a specific setting determined based on the selected area of ​​a displayed image in response to the selection of a selection area, wherein the image processing property is one of luminance, contrast, color saturation, tone, white balance, black balance, black level compensation, level, gamma correction, gain, color correction, noise filter, sharpness, blur, and red-eye correction.

[0180]

[0194] Embodiment 18: An apparatus according to any one of Embodiments 14 to 17, wherein one or more processors are further configured to display area selection UI elements in an area selection UI area.

[0181]

[0195] Embodiment 19: The apparatus of Embodiment 18, further configured to generate area selection UI elements such that one or more processors draw copies of at least a portion of the selection area.

[0182]

[0196] Embodiment 20: An apparatus of any embodiment 1 to 19, further configured to receive a second image captured by a camera after the capture of an image, determine that the second image is obscured by occlusion, and, in response to the determination that the second image is obscured by occlusion, display the displayed image on a display.

[0183]

[0197] Embodiment 21: Apparatus of Embodiment 20, wherein one or more processors are further configured to receive an input, wherein occlusion is associated with receiving an input, wherein the input is one of touch input, hover input, and gesture input, to select a selection area of ​​a display image based on the input, to determine a setting based on the selection area of ​​the display image, to apply the setting, and to receive a third image captured by a camera while the setting is applied and after the capture of a second image.

[0184]

[0198] Embodiment 22: An apparatus of any embodiment 1 to 21, wherein one or more processors are further configured to receive a second image captured by a second camera based on secondary incident light received by a second camera, wherein the second camera is positioned relative to a display to receive secondary incident light that passes through a second portion of the display before reaching the second camera.

[0185]

[0199] Embodiment 23: The apparatus of Embodiment 22, wherein one or more processors are further configured to determine that a second image is obscured by occlusion, and in response to the determination that the second image is obscured by occlusion, a display image is displayed.

[0186]

[0200] Embodiment 24: The apparatus of Embodiment 22 or 23, wherein one or more processors are further configured to generate a display image based on an image and a second image.

[0187]

[0201] Embodiment 25: Any apparatus according to Embodiments 1 to 24, wherein one or more processors are further configured to process an image and a second image to determine depth information corresponding to one or more objects depicted in the image and the second image.

[0188]

[0202] Embodiment 26: A processing method comprising: receiving an image captured by a camera based on incident light received by the camera; displaying a display image based on the image on a display, wherein the camera is positioned relative to the display to receive incident light that passes through a portion area of ​​the display before reaching the camera; and displaying an indicator overlaid on the display image in an indicator area of ​​the display while the display is displaying the display image, wherein the indicator area includes at least a subset of a portion of the display.

[0189]

[0203] Embodiment 27: The method of Embodiment 26, wherein the method is carried out by a mobile device having a camera and a display.

[0190]

[0204] Embodiment 28: Any method of Embodiments 26 to 27, wherein at least one of the indicator and the indicator area of ​​the display is at least as large as the area of ​​the camera lens.

[0191]

[0205] Embodiment 29: Any method of Embodiments 26 to 28, further comprising receiving an input, wherein an indicator is displayed as an overlay on an indicator area in response to the reception of the input.

[0192]

[0206] Embodiment 30: The method of Embodiment 29, wherein the input includes at least one of touching the touch-sensitive touchscreen surface of the display, a predefined gesture across the touch-sensitive touchscreen surface of the display, hovering over the display, a predefined gesture on the display, or voice input of speaking a predefined phrase recorded by a microphone.

[0193]

[0207] Embodiment 31: A method of any of Embodiments 26 to 30, further comprising generating a display image by resizing the image using at least one of scaling and cropping, wherein resizing the image includes resizing the image so that the display image is displayed on a display other than the indicator area.

[0194]

[0208] Embodiment 32: Any method of Embodiments 26 to 31, wherein the indicator includes a shape around at least a portion of the display.

[0195]

[0209] Embodiment 33: Any method of Embodiments 26 to 32, wherein while the display is showing a display image, the display of an indicator overlaid on the display image in an indicator area includes modifying the display image to merge the indicator with the display image.

[0196]

[0210] Embodiment 34: Any method of Embodiments 26 to 33, wherein the display is a touchscreen.

[0197]

[0211] Embodiment 35: Any method of Embodiments 26 to 34, further comprising: displaying an image capture UI element on an image capture user interface (UI) area of ​​a displayed image; receiving input in the image capture UI area; receiving a second image captured by a camera in response to receiving input in the image capture UI area, wherein the input is one of touch input, hover input, and gesture input; and storing the second image in a non-temporary computer-readable storage medium.

[0198]

[0212] Embodiment 36: Any method of Embodiments 26 to 35, further comprising: receiving input in an area selection user interface (UI) area, wherein the input is one of touch input, hover input, and gesture input; selecting a selection area of ​​a display image in response to receiving input in the area selection UI area; wherein the area selection UI area is separate from the selection area of ​​a display image.

[0199]

[0213] Embodiment 37: The method of Embodiment 36, wherein the selection area of ​​the display image includes at least a portion of the indicator area.

[0200]

[0214] Embodiment 38: The method of Embodiment 37, further comprising setting image capture parameters to a specific setting determined based on the selected area of ​​the displayed image in response to the selection of a selection area, wherein the image capture parameters include at least one of focus and exposure parameters.

[0201]

[0215] Aspect 39: The method of aspect 37 or 38, further comprising setting an image processing property to a specific setting determined based on the selected area of ​​the displayed image in response to the selection of a selection area, wherein the image processing property is one of white balance, black level compensation, gamma correction, gain, color correction, color saturation, noise filter, contrast control, brightness control, tone mapping, sharpness, blur, and red-eye correction.

[0202]

[0216] Embodiment 40: Any method of Embodiments 26 to 39, further comprising receiving a second image captured by a camera after the capture of an image, determining that the second image is obscured by occlusion, and displaying the displayed image on a display in response to the determination that the second image is obscured by occlusion.

[0203]

[0217] Embodiment 41: The method of Embodiment 40, further comprising receiving an input, wherein occlusion is associated with receiving an input, wherein the input is one of touch input, hover input, and gesture input, selecting a selection area of ​​a display image based on the input, determining a setting based on the selection area of ​​the display image, applying the setting, and receiving a third image captured by a camera while the setting is applied and after the capture of a second image.

[0204]

[0218] Embodiment 42: Any method of Embodiments 26 to 41, further comprising receiving a second image captured by a second camera based on secondary incident light received by a second camera, wherein the second camera is positioned relative to the display to receive secondary incident light that passes through a second portion of the display before reaching the second camera.

[0205]

[0219] Embodiment 43: Any method of Embodiments 26 to 42, further comprising determining that a second image is obscured by occlusion, wherein a display image is displayed in response to the determination that the second image is obscured by occlusion.

[0206]

[0220] Embodiment 44: Any method of Embodiments 26 to 43, further comprising processing an image and a second image to determine depth information corresponding to one or more objects depicted in the image and the second image.

[0207]

[0221] Embodiment 45: A non-temporary computer-readable medium storing instructions that cause one or more processors to perform the operations described in any of Embodiments 1 to 44 when executed by one or more processors.

[0208]

[0222] Embodiment 46: An apparatus comprising one or more means for performing the operation described in any of Embodiments 1 to 44. The invention described in the original claims of this application is listed below. [C1] A device for image processing, Memory and The system comprises one or more processors coupled to the memory, and the one or more processors are Receiving an image captured by a camera based on incident light received by the camera, wherein the camera is positioned relative to the display to receive the incident light that passes through a portion of the display before reaching the camera, and displaying a display image based on the image on the display. While the display is displaying the display image, the indicator area of ​​the display displays an indicator overlaid on the display image, wherein the indicator area includes at least a subset of the portion of the display. A device configured to perform the following actions. [C2] The apparatus according to C1, wherein the apparatus is at least one of a mobile device, a wireless communication device, and a camera device. [C3] The apparatus according to C1, wherein the apparatus includes at least one of the camera and the display. [C4] The apparatus according to C1, wherein at least one of the indicator and the indicator area of ​​the display is at least as large as the area of ​​the camera lens. [C5] The one or more processors described above are: Further configured to receive input, wherein the indicator is displayed on the indicator area in response to the reception of the input. The apparatus described in C1. [C6] The apparatus according to C5, wherein the input includes at least one of a touch on the touch-sensitive touchscreen surface of the display, a predefined gesture across the touch-sensitive touchscreen surface of the display, a hover over the display, a predefined gesture on the display, or a voice input of speaking a predefined phrase recorded by a microphone. [C7] The one or more processors described above are: The display image is generated by modifying the size of the image using at least one of scaling and cropping. The apparatus described in C1, further configured as follows. [C8] The apparatus according to C7, wherein modifying the size of the image includes modifying the size of the image so that the display image is displayed on the display other than the indicator area. [C9] The apparatus according to C1, wherein the indicator includes a shape around at least a portion of the portion of the display. [C10] The apparatus according to C1, wherein the aforementioned indicator is animated. [C11] The apparatus according to C1, wherein, while the display is showing the display image, displaying the indicator overlaid on the display image in the indicator area includes modifying the display image so that the indicator is merged with the display image. [C12] The apparatus as described in C1, wherein the display is a touchscreen. [C13] The one or more processors described above are: Displaying image capture UI elements on the image capture user interface (UI) area on the aforementioned display, The image capture UI area receives input, and the input is one of touch input, hover input, and gesture input. In response to receiving the input in the aforementioned image capture UI area, the system receives a second image captured by the camera, The second image is stored in a non-temporary computer-readable storage medium, The apparatus described in C1, further configured to perform the following actions. [C14] The one or more processors described above are: Receiving input in an area selection user interface (UI) area, wherein the input is one of touch input, hover input, and gesture input. In response to receiving the input in the area selection UI area, the selection area of ​​the display image is selected, and herein, the area selection UI area is separate from the selection area of ​​the display image. The apparatus described in C1, further configured to perform the following actions. [C15] The apparatus according to C14, wherein the selected area of ​​the display image includes at least a portion of the indicator area. [C16] The one or more processors described above are: In response to the selection of the selection area, the camera is further configured to set the image capture parameters associated with the camera to a specific setting determined based on the selection area of ​​the displayed image, wherein the image capture parameters include at least one of the focus and exposure parameters. The device described in C14. [C17] The one or more processors described above are: In response to the selection of the selection area, the image processing property is further configured to set to a specific setting determined based on the selected area of ​​the displayed image, wherein the image processing property is one of the following: brightness, contrast, color saturation, tone, white balance, black balance, black level compensation, level, gamma correction, gain, color correction, noise filter, sharpness, blur, and red-eye correction. The device described in C14. [C18] The one or more processors described above are: Display area selection UI elements in the aforementioned area selection UI area. The apparatus described in C14, further configured as follows. [C19] The one or more processors described above are: Generate the area selection UI element to draw a copy of at least a portion of the selected area. The apparatus described in C18, further configured as follows. [C20] The one or more processors described above are: After capturing the aforementioned image, a second image captured by the camera is received, It is determined that the second image is obscured by occlusion, In response to the determination that the second image is obscured by the occlusion, the display image is shown on the display, The apparatus described in C1, further configured to perform the following actions. [C21] The one or more processors described above are: Receiving an input, wherein the occlusion is associated with the reception of the input, wherein the input is one of touch input, hover input, and gesture input. Selecting a selection area of ​​the display image based on the input, The setting is determined based on the selected area of ​​the displayed image, Applying the above settings, While the above setting is applied, and after the capture of the second image, the third image captured by the camera is received, The apparatus described in C20, further configured to perform the following actions. [C22] The one or more processors described above are: The system is further configured to receive a second image captured by the second camera based on secondary incident light received by the second camera, wherein the second camera is positioned relative to the display to receive the secondary incident light passing through the second portion of the display before reaching the second camera. The apparatus described in C1. [C23] The one or more processors described above are: The system is further configured to determine that the second image is obscured by occlusion, and in response to the determination that the second image is obscured by occlusion, the display image is displayed. The device described in C22. [C24] The one or more processors described above are: The display image is generated based on the aforementioned image and the second image. The apparatus described in C22, further configured as follows. [C25] The one or more processors described above are: By processing the aforementioned image and the second image, depth information corresponding to one or more objects depicted in the aforementioned image and the second image is determined. The apparatus described in C22, further configured as follows. [C26] A method of processing, Receiving an image captured by the camera based on incident light received by the camera, wherein the camera is positioned relative to the display to receive the incident light passing through a portion area of ​​the display before reaching the camera. Displaying a display image based on the image on the aforementioned display, While the display is displaying the display image, the indicator area of ​​the display displays an indicator overlaid on the display image, wherein the indicator area includes at least a subset of the portion of the display. A method that includes [a certain feature]. [C27] The method according to C26, wherein the method is carried out by a mobile device comprising the camera and the display. [C28] The method according to C26, wherein at least one of the indicator and the indicator area of ​​the display is at least as large as the area of ​​the camera lens. [C29] The system further comprises receiving an input, wherein the indicator is displayed as an overlay on the indicator area in response to the reception of the input. The method described in C26. [C30] The method according to C29, wherein the input includes at least one of a touch on the touch-sensitive touchscreen surface of the display, a predefined gesture across the touch-sensitive touchscreen surface of the display, a hover over the display, a predefined gesture on the display, or a voice input of speaking a predefined phrase recorded by a microphone. [C31] The present invention further comprises generating the display image by modifying the size of the image using at least one of scaling and cropping, wherein modifying the size of the image includes modifying the size of the image so that the display image is displayed on the display other than the indicator area. The method described in C26. [C32] The method according to C26, wherein the indicator includes a shape around at least a portion of the display. [C33] The method of C26, wherein while the display is displaying the display image, displaying the indicator overlaid on the display image in the indicator area includes modifying the display image to merge the indicator with the display image. [C34] The method according to C26, wherein the display is a touchscreen. [C35] Displaying image capture UI elements on the image capture user interface (UI) area of ​​the aforementioned displayed image, The image capture UI area receives input, and the input is one of touch input, hover input, and gesture input. In response to receiving the input in the aforementioned image capture UI area, the system receives a second image captured by the camera, The second image is stored in a non-temporary computer-readable storage medium, A method for C26 that further incorporates these features. [C36] Receiving input in an area selection user interface (UI) area, wherein the input is one of touch input, hover input, and gesture input. In response to receiving the input in the area selection UI area, the selection area of ​​the display image is selected, and herein, the area selection UI area is separate from the selection area of ​​the display image. A method for C26 that further incorporates these features. [C37] The method according to C36, wherein the selected area of ​​the display image includes at least a portion of the indicator area. [C38] The further comprising setting the image capture parameters to a specific setting determined based on the selected area of ​​the displayed image in response to the selection of the selected area, wherein the image capture parameters include at least one of focus and exposure parameters. Method C37. [C39] The system further comprises setting an image processing property to a specific setting determined based on the selected area of ​​the displayed image in response to the selection of the selected area, wherein the image processing property is one of the following: white balance, black level compensation, gamma correction, gain, color correction, color saturation, noise filter, contrast control, brightness control, tone mapping, sharpness, blur, and red-eye correction. Method C37. [C40] After capturing the aforementioned image, a second image captured by the camera is received, It is determined that the second image is obscured by occlusion, In response to the determination that the second image is obscured by the occlusion, the display image is shown on the display, A method for C26 that further incorporates these features. [C41] Receiving an input, wherein the occlusion is associated with the reception of the input, wherein the input is one of touch input, hover input, and gesture input. Selecting a selection area of ​​the display image based on the input, The setting is determined based on the selected area of ​​the displayed image, Applying the above settings, While the above setting is applied, and after the capture of the second image, the third image captured by the camera is received, A method for C40 that further enhances this feature. [C42] The system further comprises receiving a second image captured by the second camera based on secondary incident light received by the second camera, wherein the second camera is positioned relative to the display to receive the secondary incident light passing through the second portion of the display before reaching the second camera. The method described in C26. [C43] The system further comprises determining that the second image is obscured by occlusion, wherein, in response to the determination that the second image is obscured by occlusion, the display image is displayed. Methods used in C42. [C44] By processing the aforementioned image and the second image, depth information corresponding to one or more objects depicted in the aforementioned image and the second image is determined. A method for C42 that further incorporates these features. [C45] A non-temporary computer-readable medium storing instructions, wherein when an instruction is executed by one or more processors, the one or more processors... Receiving an image captured by the camera based on incident light received by the camera, wherein the camera is positioned relative to the display to receive the incident light that passes through a portion of the display before reaching the camera, and displaying a display image based on the image on the display. While the display is displaying the display image, the indicator area of ​​the display displays an indicator overlaid on the display image, wherein the indicator area includes at least a subset of the portion of the display. A non-temporary computer-readable medium that enables the operation of [the process].

Claims

1. A device for image processing, Memory and The system comprises one or more processors coupled to the memory, and the one or more processors are The camera receives an image captured by the camera based on incident light received by the camera, wherein the camera is positioned relative to the display to receive the incident light that passes through a portion of the display before reaching the camera. Displaying a display image based on the image on the aforementioned display, The system generates an indicator that shows at least a subset of the portion of the display through which the incident light from the scene passes when the incident light moves toward the camera, and the indicator is generated to be hidden by default, so as to be overlaid on the display image using a blending effect. Receiving input and, In response to the reception of the aforementioned input, while the display is displaying the aforementioned display image, the indicator overlaid on the aforementioned display image is displayed in the indicator area of ​​the display, A device configured to perform the following actions.

2. The apparatus according to claim 1, wherein the apparatus is at least one of a mobile device, a wireless communication device, and a camera device.

3. The apparatus according to claim 1, wherein the apparatus includes at least one of the camera and the display.

4. The apparatus according to claim 1, wherein at least one of the indicator and the indicator area of ​​the display is at least as large as the area of ​​the camera lens.

5. The apparatus according to claim 1, wherein the input includes at least one of a touch on the touch-sensitive touchscreen surface of the display, a predefined gesture across the touch-sensitive touchscreen surface of the display, a hover over the display, a predefined gesture on the display, or a voice input of speaking a predefined phrase recorded by a microphone.

6. The apparatus according to claim 1, wherein the indicator includes a shape around at least a portion of the portion of the display.

7. The apparatus according to claim 1, wherein the indicator is animated.

8. The apparatus according to claim 1, wherein the display is a touchscreen.

9. A method of processing, Receiving an image captured by the camera based on incident light received by the camera, wherein the camera is positioned relative to the display to receive the incident light passing through a portion area of ​​the display before reaching the camera. Displaying a display image based on the image on the aforementioned display, The system generates an indicator that shows at least a subset of the portion of the display through which the incident light from the scene passes when the incident light moves toward the camera, and the indicator is generated to be hidden by default, so as to be overlaid on the display image using a blending effect. Receiving input and, In response to the reception of the aforementioned input, while the display is displaying the aforementioned display image, the indicator overlaid on the aforementioned display image is displayed in the indicator area of ​​the display, A method that includes [a certain feature].

10. The method according to claim 9, wherein the input includes at least one of a touch on the touch-sensitive touchscreen surface of the display, a predefined gesture across the touch-sensitive touchscreen surface of the display, a hover over the display, a predefined gesture on the display, or a voice input of speaking a predefined phrase recorded by a microphone.

11. The present invention further comprises generating the display image by modifying the size of the image using at least one of scaling and cropping, wherein modifying the size of the image includes modifying the size of the image so that the display image is displayed on the display other than the indicator area. The method according to claim 9.

12. The method of claim 9, wherein, while the display is displaying the display image, displaying the indicator overlaid on the display image in the indicator area includes modifying the display image so that the indicator is merged with the display image.

13. Displaying image capture UI elements on the image capture user interface (UI) area of ​​the aforementioned displayed image, The image capture UI area receives input, and the input is one of touch input, hover input, and gesture input. In response to receiving the input in the aforementioned image capture UI area, the system receives a second image captured by the camera, The second image is stored in a non-temporary computer-readable storage medium, The method according to claim 9, further comprising:

14. Receiving a second image captured by the second camera based on secondary incident light received by the second camera, wherein the second camera is positioned relative to the display to receive the secondary incident light passing through the second portion of the display before reaching the second camera. Optionally, depth information corresponding to one or more objects depicted in the image and the second image is determined by processing the image and the second image. The method according to claim 9, further comprising:

15. A non-temporary computer-readable medium storing instructions, wherein when an instruction is executed by one or more processors, the one or more processors... Receiving an image captured by the camera based on incident light received by the camera, wherein the camera is positioned relative to the display to receive the incident light that passes through a portion of the display before reaching the camera. Displaying a display image based on the image on the aforementioned display, The system generates an indicator that shows at least a subset of the portion of the display through which the incident light from the scene passes when the incident light moves toward the camera, and the indicator is generated to be hidden by default, so as to be overlaid on the display image using a blending effect. Receiving input and, In response to the reception of the aforementioned input, while the display is displaying the aforementioned display image, the indicator overlaid on the aforementioned display image is displayed in the indicator area of ​​the display, A non-temporary computer-readable medium that enables the operation of [the process].

Images