Under-screen camera and sensor control

Abstract

An under-display camera is positioned under a display of a mobile device. The under-display camera captures an image using light that passes through a portion of the display. The mobile device displays a display image on the display, the display image being based on the image. The mobile device displays an indicator overlaid on the display image on an indicator area of the display that overlaps with the portion of the display. The indicator can identify a location of the camera. If a recently captured image includes an occlusion, the mobile device can compensate for the occlusion of the camera by continuing to display a previous display image. The mobile device can provide alternative ways for a user to select an area of the display image to avoid the camera occlusion, e.g., using a hardware button and / or a touch screen interface element.

Description

Technical Field

[0001] This application relates to image capture and image processing. More specifically, this application relates to techniques and processes for receiving user input to control an under-display camera and / or an under-display sensor and to compensate for occlusion of the under-display camera and / or the under-display sensor. Background Technology

[0002] Some mobile devices include a front-facing camera that faces the user while the device is held in hand. The front-facing camera can be adjacent to the mobile device's display. For example, this allows the mobile device to use the front-facing camera and an adjacent display for video conferencing. Because mobile devices are typically small and portable, the front-facing camera is usually located within a bezel around the display. Some mobile devices include displays with notches or cutouts, where the display cannot show any content, and the front-facing camera is located within the notch or cutout. When the front-facing camera is located within a bezel, notch, or cutout, the user can usually see where the front-facing camera is. However, bezels, notches, and cutouts occupy valuable space on the front surface of the mobile device and prevent the mobile device from having a display that occupies the entire front surface of the mobile device. Summary of the Invention

[0003] The systems and techniques described herein are for using under-display cameras and / or under-display sensors located beneath the display of a device (e.g., a mobile device or other device). The device may display an indicator on the display indicating the location of the under-display camera and / or under-display sensor. The device may compensate for occlusion of the under-display camera and / or under-display sensor. For example, such occlusion may be caused by a user or input device touching the display in a manner that obscures the under-display camera and / or under-display sensor, a user or input device hovering above the display in a manner that obscures the under-display camera and / or under-display sensor, or a user or input device being sufficiently close to the display for other reasons to obscure the under-display camera and / or under-display sensor. The device may control the under-display camera and / or under-display sensor based on input. The device may include user interface (UI) elements that allow selection of an image area without touching, hovering, or otherwise obscuring the image area, for example, such input at the image area that would obscure the under-display camera and / or under-display sensor.

[0004] In one example, an apparatus for image processing is provided. The apparatus includes a memory. The apparatus includes one or more processors coupled to the memory and configured to perform a method. The method includes receiving an image captured by a camera based on incident light received by a camera. The camera is positioned relative to a display to receive incident light that has passed through a portion of the display before reaching the camera. The method includes displaying an image-based display image on the display. The method also includes displaying an indicator overlaid on the display image at an indicator area of ​​the display while the display is displaying the display image. The indicator area includes at least a subset of a portion of the display.

[0005] In another example, an apparatus for image processing is provided. The apparatus includes one or more memory units storing instructions. The apparatus includes one or more processors executing the instructions, wherein execution of the instructions by the one or more processors causes the one or more processors to perform a method. The method includes receiving an image captured by a camera based on incident light received by the camera. The camera is positioned relative to a display to receive incident light that has passed through a portion of the display before reaching the camera. The method includes displaying an image-based display image on the display. The method also includes displaying an indicator overlaid on the display image at an indicator area of ​​the display while the display is displaying the display image. The indicator area includes at least a subset of a portion of the display.

[0006] In another example, an image processing method is provided. The method includes receiving image data captured by an image sensor. The method includes receiving the image captured by the camera based on incident light received by the camera. The camera is positioned relative to a display to receive incident light that passes through a portion of the display before reaching the camera. The method includes displaying an image-based display image on the display. The method includes displaying an indicator overlaid on the display image at an indicator area of ​​the display while the display is displaying the display image. The indicator area includes at least a subset of the display portion.

[0007] In another example, a non-transitory computer-readable storage medium on which a program is embodied is provided. The program is executable by a processor to perform an image processing method. The method includes receiving an image captured by a camera based on incident light received by the camera. The camera is positioned relative to a display to receive incident light that has passed through a portion of the display before reaching the camera. The method includes displaying an image-based display image on the display. The method also includes displaying an indicator overlaid on the display image at an indicator area of ​​the display while the display is displaying the display image. The indicator area comprises at least a subset of a portion of the display.

[0008] In another example, an apparatus for image processing is provided. The apparatus includes a memory. The apparatus includes a unit for receiving an image captured by a camera based on incident light received by the camera. The camera is positioned relative to a display to receive incident light that passes through a portion of the display before reaching the camera. The apparatus includes a unit for displaying an image-based display image on the display. The apparatus includes a unit for displaying an indicator overlaid on the display image at an indicator area of ​​the display while the display is displaying the display image. The indicator area includes at least a subset of a portion of the display.

[0009] In another example, an image processing method is provided. The method includes receiving an image captured by a camera. The camera is located below a display area and captures the image using light passing through the display. The method includes displaying the image on the display. The method includes setting image processing attributes to specific settings based on the region determined by the display. The image processing attributes are one of the following: white balance, black level compensation, gamma correction, gain, color correction, color saturation, noise filter, contrast control, brightness control, tone mapping, sharpness, blur, or red-eye correction.

[0010] In another example, an image processing method is provided. The method includes receiving an image captured by a camera. The camera captures the image based on incident light received by the camera. The camera is positioned relative to a touchscreen display to receive incident light passing through a portion of the touchscreen display before reaching the camera. The method includes displaying an image-based display on the touchscreen. The method includes displaying a region selection user interface (UI) element on the touchscreen. The method includes receiving touch input to the touchscreen display at a region selection input area overlapping with the region selection UI element. The method includes selecting a selection area of ​​the display image in response to receiving the touch input. The method includes setting camera properties of the camera to settings determined based on the selection area of ​​the display image in response to the selection of the selection area, wherein the camera properties of the camera include at least one of image capture properties or image processing properties.

[0011] In some respects, the indicator area of ​​the display is at least as large as the camera lens.

[0012] In some aspects, the methods, apparatus, and computer-readable media described above further include: receiving input, wherein an indicator is displayed as an overlay on an indicator area in response to the receipt of input. In some aspects, the input includes at least one of the following: touching a touch-sensitive touchscreen surface of the display, a predefined gesture across the touch-sensitive touchscreen surface of the display, hovering over the display, a predefined gesture above the display, or voice input uttering a predefined phrase recorded by a microphone.

[0013] In some aspects, the methods, apparatus, and computer-readable media described above further include generating a display image by modifying the size of an image using at least one of scaling and cropping. In some aspects, modifying the image size includes modifying the image size such that the display image is displayed on a monitor but not on an indicator area.

[0014] In some aspects, the above-described methods, apparatus, and computer-readable media further include: generating a display image by processing an image, wherein processing the image includes modifying at least one of brightness, contrast, color saturation, hue, white balance, black balance, level, gamma correction, gain, color correction, noise filter, sharpness, blur, and red-eye correction.

[0015] In some aspects, the camera is positioned adjacent to the display along an axis perpendicular to the plane surface of the display. In some aspects, the indicator includes a shape surrounding at least a portion of that part of the display. In some aspects, the display indicator includes causing a plurality of pixels in the indicator area to display a color. In some aspects, the color is black. In some aspects, the indicator is animated.

[0016] In some aspects, displaying an indicator overlaid on the display image at the indicator area while the display is displaying the display image includes modifying the display image to merge the indicator with the display image. In some aspects, the display is a touchscreen.

[0017] In some aspects, the methods, apparatus, and computer-readable media described above further include: displaying image capture user interface (UI) elements on an image capture user interface (UI) area on a display; receiving input at the image capture UI area, wherein the input is one of touch input, hover input, and gesture input; receiving a second image captured by the camera in response to receiving the input at the image capture UI area; and storing the second image to a non-transitory computer-readable storage medium.

[0018] In some aspects, the methods, apparatus, and computer-readable media described above further include: receiving input at a region selection user interface (UI) area, wherein the input is one of touch input, hover input, and gesture input; and selecting a selection area of ​​the display image in response to receiving the input at the region selection UI area, wherein the region selection UI area is different from the selection area of ​​the display image. In some aspects, the selection area of ​​the display image includes at least a portion of an indicator area. In some aspects, the methods, apparatus, and computer-readable media described above further include: displaying a region selection UI element at the region selection UI area. In some aspects, the methods, apparatus, and computer-readable media described above further include: generating a copy of the region selection UI element to depict at least a portion of the selection area.

[0019] In some aspects, the methods, apparatus, and computer-readable media described above further include: in response to the selection of the selected region, setting image capture parameters to specific settings determined based on the selected region of the displayed image, wherein the image capture parameters include at least one of focus and exposure parameters. In some aspects, the methods, apparatus, and computer-readable media described above further include: in response to the selection of the selected region, setting image processing attributes to specific settings determined based on the selected region of the displayed image, wherein the image processing attributes are one of white balance, black level compensation, gamma correction, gain, color correction, color saturation, noise filter, contrast control, brightness control, tone mapping, sharpness, blur, and red-eye correction.

[0020] In some aspects, the methods, apparatus, and computer-readable media described above further include: receiving a second image captured by the camera after capturing the image; determining that the second image is occluded by an occluder; and displaying the display image on the display in response to determining that the second image is occluded by the occluder. In some aspects, the methods, apparatus, and computer-readable media described above further include: receiving input, wherein the occluder is associated with the reception of the input, wherein the input is one of touch input, hover input, and gesture input; selecting a selection area of ​​the display image based on the input; determining a setting based on the selection area of ​​the display image; applying the setting; and receiving a third image captured by the camera while applying the setting and after capturing the second image.

[0021] In some aspects, the methods, apparatus, and computer-readable media described above further include: receiving a second image captured by the second camera based on secondary incident light received by the second camera, wherein the second camera is positioned relative to the display to receive the secondary incident light passing through a second portion of the display before reaching the second camera. In some aspects, the methods, apparatus, and computer-readable media described above further include: determining that the second image is occluded by an occluder, wherein the display image is displayed in response to determining that the second image is occluded by the occluder. In some aspects, the methods, apparatus, and computer-readable media described above further include: determining depth information corresponding to one or more objects depicted in the image and the second image by processing the image and the second image.

[0022] In some aspects, the device includes a camera, a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wireless communication device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other devices. In some aspects, one or more processors include an image signal processor (ISP). In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device includes an image sensor for capturing image data. In some aspects, the device also includes a display for displaying images, one or more notifications and / or other displayable data associated with image processing.

[0023] This overview is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the appropriate portions of this patent specification, any or all of the accompanying drawings, and each claim.

[0024] The foregoing, along with other features and embodiments, will become more apparent from the following description, claims, and drawings. Attached Figure Description

[0025] The exemplary embodiments of this application will now be described in detail with reference to the accompanying drawings:

[0026] Figure 1 This is a block diagram illustrating an example of the architecture of an image capture and processing device.

[0027] Figure 2A It is an exploded front view of a mobile device with a display, an under-display camera, and an under-display speaker;

[0028] Figure 2B yes Figure 2A An unexposed front view of the mobile device;

[0029] Figure 2C yes Figure 2A A side sectional view of the mobile device;

[0030] Figure 2D yes Figure 2A A front view of a mobile device, which has a first under-display camera, a second under-display camera, and an under-display sensor;

[0031] Figure 3A This is a front view of a mobile device with an under-display camera, where the display shows a black-bordered indicator above the camera;

[0032] Figure 3B This is a front view of a mobile device with an under-display camera, where the display shows a black notch indicator above the camera;

[0033] Figure 3C This is a front view of a mobile device with an under-display camera, where the display shows a black circular indicator above the camera;

[0034] Figure 4 This is a front view of a mobile device with an under-display camera, where the display shows a triangular indicator around the area above the camera.

[0035] Figure 5 This is a front view of a mobile device with an under-display camera, where the display shows a circular indicator above the camera and area selection user interface (UI) elements for selecting a selectable area above the camera;

[0036] Figure 6 It is a front view of the mobile device that compensates for the obstruction of the under-screen camera by the user's hand during touch, hover, or gesture input;

[0037] Figure 7 This is a front view of a mobile device with multiple under-display cameras and under-display sensors, where the display shows the interface area used to control the multiple under-display cameras and under-display sensors;

[0038] Figure 8 This is a flowchart illustrating an example of image processing techniques;

[0039] Figure 9 This is a flowchart illustrating another example of image processing techniques; and

[0040] Figure 10 This is a diagram illustrating an example of a system used to implement certain aspects of this technology. Detailed Implementation

[0041] Certain aspects and embodiments of this disclosure are provided below. It will be apparent to those skilled in the art that some of these aspects and embodiments can be applied independently and some can be combined. In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of embodiments of this application. However, it will be apparent that various embodiments can be practiced without these specific details. The accompanying drawings and description are not intended to be limiting.

[0042] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing the exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0043] An image capture device (e.g., a camera) is a device that uses an image sensor to receive light and capture image frames (e.g., still images or video frames). The terms "image," "image frame," and "frame" are used interchangeably herein. An image capture device typically includes at least one lens that receives light from a scene and directs the light to the image sensor of the image capture device. The light received by the lens can be controlled by an aperture controlled by one or more control mechanisms. The light received by the lens is then received by the image sensor after passing through the lens. One or more control mechanisms can control exposure, focus, and / or zoom based on information from the image sensor and / or based on information from an image processor (e.g., a host or application process and / or an image signal processor). In some examples, one or more control mechanisms include a motor or other control mechanism that moves the lens of the image capture device to a target lens position.

[0044] As described in more detail below, the systems and techniques described herein are for using one or more under-display cameras and / or one or more under-display sensors located beneath the display of a device (e.g., a mobile device or other device). Some image capture devices include one or more front-facing cameras on the front surface of the image capture device. In some examples, the one or more front-facing cameras face the same direction as the display screen of the image capture device. In a video conference, as an illustrative example, one or more front-facing cameras capture images (e.g., video frames) of a user, while the display screen faces the user and displays images (e.g., video frames) of one or more other users connected to the video conference and / or the user themselves. The user can capture an image of himself or her using one or more front-facing cameras facing the user (referred to as a "selfie"). As the user moves the image capture device to a position for capturing images, the user can align the captured image with the frame and align the user's depiction in the image by viewing the displayed image frame on the user-facing display screen. The displayed image frame may be based on image frames captured by one or more front-facing cameras and / or temporarily stored in an image buffer, and thus can serve as a preview of the field of view of one or more front-facing cameras.

[0045] In some cases, one or more front-facing cameras may be positioned adjacent to the display screen in a direction parallel to the plane of the display. For example, one or more front-facing cameras may be located within a bezel surrounding the display screen. In some cases, the display screen may include a notch or cutout in which the display screen does not display anything. A cutout is sometimes referred to as a “hole” or “punch-hole” in the display screen. Each of one or more front-facing cameras may be positioned within such a bezel, notch, or cutout. One advantage of placing the front-facing camera within a bezel, notch, or cutout is that the user understands the position of the front-facing camera on the front surface of the image capture device. In some cases, the bezel, notch, or cutout may be one of the only locations on the front surface of the device that are not covered by the display screen. The presence of a bezel, notch, or cutout on the front surface of the device can be considered an indicator that the front-facing camera is located within the bezel, notch, or cutout. However, the presence of bezels, notches, and / or cutouts in the device prevents the display screen from occupying the entire surface of the image capture device. This side effect of bezels, notches, and / or cutouts can cause problems, especially on devices with small front surfaces, such as mobile phones. Using smaller displays can lead to accessibility issues. For example, users with poor or impaired vision may have difficulty seeing visual elements displayed on a screen. If the screen is a touchscreen, users with poor vision may have difficulty seeing and successfully interacting with touch-based user interface elements displayed on the screen. Positioning the camera within a bezel, notch, and / or cutout can also cause gaze mismatch problems. Gaze mismatch occurs when a user is looking at the monitor instead of the camera while the camera is capturing an image of the user. This results in the captured image depicting the user's eyes looking to one side. This is often a problem in video conferencing because users typically look at the monitor to see the second user they are talking to, and in doing so, they are not looking at the camera. Therefore, during a video conference, neither user appears to be making eye contact with the other.

[0046] A device may include a camera located beneath a display screen. Such a camera may be referred to as an under-display camera. The under-display camera can capture images using light passing through the display screen beneath it. In some examples, the under-display camera can capture images using light passing through the display screen while the display screen is in use and displaying visual content. In some examples, the under-display camera can capture images using light passing through the display screen while the display screen is off and not displaying any visual content. In some examples, the under-display camera can capture images using light passing through the display screen during a brief period when the display screen is flickering off and otherwise on and displaying visual content.

[0047] A device with one or more under-display cameras may include a display screen that faces the same direction as the one or more under-display cameras and occupies the entire front surface of the device without any bezels, notches, or cutouts. However, devices with under-display cameras can present problems. For example, if the camera is located below the display screen, the user may not know where the camera is. In one example, a user who wants to look at the front-facing camera while capturing an image may not know where to look on the device, which is common in the case of selfies or group photos. A user interacting with the touchscreen while the under-display camera is active may also obscure the camera with their finger or stylus by touching the portion of the touchscreen located above the under-display camera. For example, the device may include a touchscreen user interface that allows users to touch an area of ​​an image to select that area. In some cases, a user may accidentally obscure the camera while touching the display screen to attempt to select an area of ​​an image.

[0048] The issues with under-display cameras are not limited to the front-facing display and / or front-facing camera. Some devices may have multiple displays and / or multiple under-display cameras on different surfaces. For example, some devices may have a front surface and a back surface, and may have displays with under-display cameras on both the front and back surfaces. Some devices may have foldable displays, or may have a fold or hinge between two separate displays. These displays, and the under-display camera beneath each of them, may face different directions at different times. When a foldable device is folded closed, some under-display cameras may be obscured and disabled by the device.

[0049] Figure 1 This is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing scene images (e.g., an image of scene 110). The image capture and processing system 100 can capture individual images (or photographs) and / or can capture video including multiple images (or video frames) in a specific sequence. A lens 115 of the system 100 faces scene 110 and receives light from scene 110. The lens 115 directs the light to an image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130.

[0050] One or more control mechanisms 120 may control exposure, focus, white balance, black balance, and / or zoom based on information from image sensor 130 and / or image processor 150. One or more control mechanisms 120 may include multiple mechanisms and components; for example, control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. One or more control mechanisms 120 may also include additional control mechanisms besides those shown, such as controls for analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0051] The focus control mechanism 125B of the control mechanism 120 can obtain the focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the position of the lens 115 relative to the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can move the lens 115 closer to or further away from the image sensor 130 by an actuating motor or servo system (or other lens mechanism), thereby adjusting the focus. In some cases, the system 100 may include additional lenses, such as one or more microlenses on each photodiode of the image sensor 130. Each microlens can direct light received from the lens 115 to the corresponding photodiode before the light reaches the photodiode. The focus setting can be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus setting can be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may be referred to as the image capture setting and / or the image processing setting.

[0052] The exposure control mechanism 125A of the control mechanism 120 can obtain the exposure settings. In some cases, the exposure control mechanism 125A stores the exposure settings in a memory register. Based on these exposure settings, the exposure control mechanism 125A can control the aperture size (e.g., aperture size or f / stop), the duration of the aperture opening (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied to the image sensor 130, or any combination thereof. The exposure settings may be referred to as image capture settings and / or image processing settings.

[0053] The zoom control mechanism 125C of the control mechanism 120 can obtain zoom settings. In some examples, the zoom control mechanism 125C stores the zoom settings in a memory register. Based on the zoom settings, the zoom control mechanism 125C can control the focal length of an assembly (lens assembly) including lens elements such as lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by actuating one or more motors or servo systems (or other lens mechanisms) to move one or more lenses relative to each other. The zoom settings may be referred to as image capture settings and / or image processing settings. In some examples, the lens assembly may include a parfocal zoom lens or a varifocal zoom lens. In some examples, the lens assembly may include a focusing lens (which in some cases may be lens 115) that first receives light from scene 110, and then the light passes through an afocal zoom system between the focusing lens (e.g., lens 115) and the image sensor 130 before reaching the image sensor 130. In some cases, a focusless zoom system may include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference between them), with a negative (e.g., diverging, concave) lens between them. In some cases, the zoom control mechanism 125C moves one or more lenses in the focusless zoom system, such as a negative lens and one or two positive lenses.

[0054] Image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image produced by image sensor 130. In some cases, different photodiodes may be covered by different color filters, and thus light matching the color of the color filter covering the photodiode can be measured. For example, Bayer color filters include red, blue, and green color filters, and each pixel of the image is generated based on red light data from at least one photodiode covered by the red color filter, blue light data from at least one photodiode covered by the blue color filter, and green light data from at least one photodiode covered by the green color filter. Other types of color filters may be used in place of or attached to red, blue, and / or green color filters using yellow, magenta, and / or cyan (also known as "emerald") color filters. Some image sensors (e.g., image sensor 130) may not have color filters at all, instead using different photodiodes throughout the pixel array (in some cases, vertically stacked). Different photodiodes throughout the pixel array may have different spectral sensitivity profiles, and thus respond to light of different wavelengths. Monochrome image sensors may also lack color filters, thus lacking color depth.

[0055] In some cases, image sensor 130 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which can be used for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier to amplify the analog signal output from the photodiodes and / or include an analog-to-digital converter (ADC) to convert the analog signal output from the photodiodes (and / or the analog signal output amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more control mechanisms 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide-semiconductor (CMOS), an N-type metal-oxide-semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0056] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more other types of processors 1010 discussed with respect to the computing system 1000. The host processor 152 may be a digital signal processor (DSP) and / or other types of processor. In some embodiments, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-a-chip or SoC) including the host processor 152 and ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G, or LTE, 5G, etc.), memory, and connectivity components (e.g., Bluetooth). TMThis includes components such as the Global Positioning System (GPS), any combination thereof, and / or other components. I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as Inter-Integrated Circuit 2 (I2C) interface, Inter-Integrated Circuit 3 (I3C) interface, Serial Peripheral Interface (SPI) interface, Serial General Purpose Input / Output (GPIO) interface, Mobile Industry Processor Interface (MIPI) (e.g., MIPICSI-2 physical (PHY) layer port or interface), Advanced High Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an illustrative example, host processor 152 may communicate with image sensor 130 using the I2C port, and ISP 154 may communicate with image sensor 130 using the MIPI port.

[0057] Image processor 150 can perform a variety of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, input reception, output management, memory management, or some combination thereof. Image processor 150 can store image frames and / or processed images in random access memory (RAM) 140 / 1020, read-only memory (ROM) 145 / 1025, cache, memory cells, another storage device, or some combination thereof.

[0058] Various input / output (I / O) devices 160 can be connected to the image processor 150. I / O devices 160 may include a display screen, keyboard, keypad, touchscreen, touchpad, touch-sensitive surface, printer, any other output device 1035, any other input device 1045, or some combination thereof. In some cases, text input to the image processing device 105B can be performed via the physical keyboard or keypad of the I / O device 160, or via the virtual keyboard or keypad of the touchscreen of the I / O device 160. I / O 160 may include one or more ports, jacks, or other connectors that enable wired connections between the system 100 and one or more peripheral devices, through which the system 100 can receive data from and / or send data to one or more peripheral devices. I / O 160 may include one or more wireless transceivers that enable wireless connections between the system 100 and one or more peripheral devices, through which the system 100 can receive data from and / or send data to one or more peripheral devices. Peripheral devices may include any type of I / O device 160 previously discussed, and once they are coupled to ports, jacks, wireless transceivers or other wired and / or wireless connectors, they can be considered I / O devices 160 in themselves.

[0059] In some cases, the image capture and processing system 100 may be a single device. In other cases, the image capture and processing system 100 may be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some embodiments, the image capture device 105A and the image processing device 105B may be wirelessly coupled together, for example, via one or more cables, wires, or other electrical connectors and / or via one or more wireless transceivers. In some embodiments, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0060] like Figure 1 As shown, the vertical dashed line will Figure 1 The image capture and processing system 100 is divided into two parts, namely image capture device 105A and image processing device 105B. Image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. Image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), RAM 140, ROM 145, and I / O 160. In some cases, certain components shown in image capture device 105A (e.g., ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0061] Image capture and processing system 100 may include electronic devices such as mobile or landline handsets (e.g., smartphones, cellular phones, etc.), desktop computers, laptop or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game terminals, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some examples, image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some embodiments, image capture device 105A and image processing device 105B may be different devices. For example, image capture device 105A may include a camera device and image processing device 105B may include a computing device, such as a mobile phone, desktop computer, or other computing device.

[0062] Although the image capture and processing system 100 is shown to include certain components, those skilled in the art will recognize that the image capture and processing system 100 may include more than [other components]. Figure 1 The components shown are further components. Components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, components of the image capture and processing system 100 may include electronic circuitry or other electronic hardware and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits) and / or may include computer software, firmware, or any combination thereof and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image capture and processing system 100.

[0063] I / O 156 and / or I / O 160 may include a connection to a display. The connection to the display included in I / O 156 and / or I / O 160 may include an output connection through which image processor 150 transmits images captured and processed by image sensor 130 to the display and / or display controller. The display controller receives the image and causes the display to display the image.

[0064] In some implementations, the display may be a touchscreen. Connections to the display included in I / O 156 and / or I / O 160 may include input connections through which image processor 150 can receive input from the touchscreen and / or touchscreen controller. For example, the touchscreen may receive touch input, and the touchscreen controller may send information about the touch input to image processor 150. Touch input may be an instance where a portion of the touchscreen detects a touch by a user's finger, a stylus, a pointer, a pen, a pencil, or another input device. Information about the touch input may include, for example, the coordinates of the touch input on the touchscreen. Touch input may be received at the touchscreen while it is displaying an image. Information about the touch input may identify a portion of an image being displayed on the touchscreen at the location where the touch input was received. This portion of the image may include, for example, an area spanning one or more pixels of the image. Touch input may be received at the touchscreen while it is displaying a user interface (UI). The UI may include one or more UI elements, such as virtual buttons or sliders. Information about the touch input may identify a portion of the UI (e.g., UI elements) displayed on the touchscreen at the location where the touch input was received. A touchscreen can be a resistive touchscreen, a capacitive touchscreen, a surface acoustic wave (SAW) touchscreen, an infrared (IR) touchscreen, an optical imaging touchscreen, a dispersive signal touchscreen, a piezoelectric touchscreen, an acoustic pulse recognition touchscreen, other types of touchscreens discussed herein, or some combination thereof.

[0065] In some implementations, the display may include one or more other input components. For example, the display may include one or more active depth sensors located near or below the display. The active depth sensor may be oriented in the same direction as the display. The active depth sensor may include a stereo camera. The active depth sensor may be any type of sensor discussed herein with respect to sensor 260. Connections to the display included in I / O 156 and / or I / O 160 may include input connections through which image processor 150 may receive input from the active depth sensor. The input from the active depth sensor may be processed by image processor 150 or another processor to determine the distance from the sensor to an object, which may be referred to as depth. If the active depth sensor includes a camera, this depth determination may be performed using stereo image processing, as discussed further herein. Thus, a device with an active depth sensor can determine the three-dimensional coordinates of objects in the environment surrounding the device. Image processor 150 (or another processor) can process sensor measurements from an active depth sensor to determine whether a user's hand, finger, or pointing device (e.g., a stylus) is hovering over, pointing at, gesturing toward, or gesturing around a user interface (UI) element displayed on the display. If image processor 150 (or another processor) detects that a user (or pointing device) is hovering over, pointing at, gesturing toward, or gesturing around a particular UI element, image capture and processing system 100 can trigger a function corresponding to that UI element. Hover-based, pointing-based, or gesture-based interactions with a display may be more hygienic than touchscreen-based interactions, especially for image capture and processing system 100, which is expected to be interacted with by many different people. For example, these non-touch inputs may be more hygienic for image capture and processing systems 100 included in point-of-sale (POS) terminals, ATMs, restaurant kiosks, information kiosks, or other kiosks.

[0066] At least a subset of the image capture and processing system 100 may be located below the display. In some examples, at least the image sensor 130 and the lens 115 are located below the display. In one example, the image capture device 105A is located below the display. In some examples, the image processing device 105B is located below the display. The term "under-display camera" may refer to at least a subset of the image capture and processing system 100, at least a subset of the image capture device 105A, at least a subset of the image processing device 105B, or some combination thereof, that is at least partially located below the display. Components or devices located below the display may be referred to as under-display components or devices.

[0067] Figure 2A This is an exploded front view of a mobile device 205 having a display 210, an under-display camera 220, and an under-display speaker 225. Figure 290 is included. Figure 2A In it, it shows the representation Figure 2A The device has three axes in different spatial dimensions. These axes are labeled as the X-axis, Y-axis, and Z-axis. The X-axis and Y-axis are parallel to the planar surface of the display 210. The Z-axis is perpendicular to the planar surface of the display 210. The mobile device 205 is shown as having a body 215 including a front surface 255, which... Figure 2A The center is white. The main body 215 also includes the outer shell 250, which... Figure 2A Diagonal stripes and shading are used. As shown by the dashed lines and arrows, the display 210 is positioned along the Z-axis in front of the front surface 255 of the body 215 and will be placed on the front surface 255 of the body 215. The front surface 255 of the body 215 includes the camera 220 and the speaker 225. Placing the display 210 on the front surface of the body 215 includes placing the display 210 on and / or within the camera 220 and speaker 225 along the Z-axis. Placing the display 210 on the front surface of the body 215 along the Z-axis so that the surface of the display 210 covers the camera 220 and speaker 225.

[0068] The main body 215 also includes a processor 270, a memory 275, and a battery 280. The camera 220 and the speaker 225 are connected to the processor 270 and the memory 275 via one or more data connection lines. In some cases, the processor 270 may include an image processor 150, an ISP 154, a host processor 152, or some combination thereof. The processor 270 can process image data captured by the image sensor of the camera 220. The memory 275 can store images captured by the camera 220 and / or processed by the processor 270. The memory 275 can also store audio to be output by the speaker 225. The processor 270 can process the audio to be output by the speaker 225. The camera 220, speaker 225, processor 270, and / or memory 275 receive power from the battery 280 via one or more data lines.

[0069] Camera 220 can be any type of camera discussed herein. For example, camera 220 may include at least a subset of image capture device 105A, at least a subset of image processing device 105B, at least a subset of image capture and processing system 100, at least a subset of any other camera device or system discussed herein, or combinations thereof. Camera 220 can use any type of lens. For example, the camera can use a conventional lens, a wide-angle lens, a fisheye lens, a telephoto lens, a zoom lens, or combinations thereof. Camera 220 can detect light within a specific spectrum. The spectrum can be a subset of the electromagnetic spectrum. For example, the spectrum can be or may include the visible light (VL) spectrum, the infrared (IR) spectrum, the near-infrared (NIR) spectrum, the ultraviolet (UV) spectrum, another subset of the electromagnetic spectrum, or combinations thereof. The NIR spectrum may refer to a subset of the IR spectrum adjacent to the VL spectrum. The image sensor of camera 220 can be a complementary metal-oxide-semiconductor (CMOS) image sensor, a charge-coupled device (CCD) image sensor, other types of image sensors, or combinations thereof. The image sensor of camera 220 can be an image sensor of any size.

[0070] Figure 2B yes Figure 2A An unexploded front view of the mobile device 205. The camera 220 is located below the display 210 and is... Figure 2B The monitor 210 is covered. Figure 2A Legend 290 is also included. Figure 2B The camera 220 can be referred to as an under-display camera. The speaker 225 is located below the display 210 along the Z-axis and is... Figure 2B The display 210 covers the device. A speaker 225, which may be referred to as an under-display speaker, is positioned near the top of the mobile device 205 along the Y-axis and outputs audio to the display 210, allowing audio waves to propagate through the display 210. A user holding the mobile device 205 near their ear can hear the audio output by the speaker 225 through the display 210. The speaker 225 and camera 220 may be adjacent to the display 210 along the Z-axis. Because the camera 220 is completely covered by the display 210, any incident light reaching the camera 220 passes through the light passage area 285 of the display 210 before reaching the camera 220. The light passage area 285 is shown using a dashed line and is at least as large as the lens of the camera 220. In some devices, the camera may only be partially covered by the display 210; in this case, some incident light reaching such a camera may pass through the area 285 of the display 210, while other incident light reaching such a camera may pass through another display, a non-display surface, or no surface at all (e.g., air or another fluid). In some cases, the light-passing area 285 of the display 210 may be simply referred to as a part of the display 210, or an area of ​​the display 210.

[0071] For simplicity, processor 270, memory 275 and battery 280 are not included. Figure 2B As shown in the diagram. Similarly, the data and power cables between the camera 220, speaker 225, processor 270, memory 275, and battery 280 are not connected. Figure 2B As shown in the image. It should be understood that these components can still exist. Figure 2B In the mobile device 205, its configuration is similar to Figure 2A The configuration shown is similar, with these components positioned below the display 210 along the Z-axis.

[0072] Display 210 is shown as being slightly smaller than the front surface 255 of the body 215 of the mobile device 205, such that display 210 is... Figure 2A and Figure 2B The various components of the mobile device 205 are clearly visible inside. The mobile device 205 therefore includes a small bezel surrounding the display 210. However, this bezel does not need to be present in the mobile device 205. In some cases, the display 210 may cover the entire vertical length and / or the entire horizontal length of the front surface 255 of the body 215 of the mobile device 205, from edge to edge, where there is no bezel around the display 210. In some cases, the display 210 may include one or more portions curved around the edge of the front surface 255 of the body 215 of the mobile device 205, such that the display 210 may even wrap around at least a portion of the side of the mobile device 210 connecting the front surface 255 of the body 215 to the rear surface 240 of the body 215 of the mobile device 205.

[0073] Figure 2C yes Figure 2A A side sectional view of mobile device 205. (Due to...) Figure 2A and Figures 2B to 2C The perspective view changes, showing only the Y and Z axes of Legend 290. Legend 295 includes... Figure 2C In. Figure 2C In a side view of the mobile device 205, the front surface 252 of the display 210 appears on the left side of the mobile device 205, while the rear surface 240 of the body 215 appears on the right side of the mobile device 205. The body 215 includes a housing 250, which is shown as a shaded area with horizontal stripes. The body 215 also includes a camera 220 and a speaker 225. A portion of the body 215 may be coupled to the camera 220 and / or the speaker 225. The front portions of the speaker 225 and the camera 220 form a portion of the front surface 255 of the body 215. The camera 220 and the speaker 225 are shown adjacent to the display 210 and / or coupled to the display 210 along the Z-axis. In some cases, there may be some space along the Z-axis between the camera 220 and the display 210 and / or between the speaker 225 and the display 210.

[0074] Multiple rays 245 are shown as arrows passing through display 210 and reaching camera 220. Image sensor 130 of camera 220 receives the light 245 passing through at least a portion of display 210. In some examples, display 210 may include microlenses in the gap regions between the pixels of the display. Display 210 may be a touchscreen. In some aspects, the gap regions, and therefore the microlenses, are also located between traces on the touch-sensitive touchscreen surface of display 210. These traces may be capacitive traces, resistive traces, or other traces, depending on the type of touchscreen. In some examples, light 245 passes through transparent and / or translucent liquid crystal, glass, and / or plastic layers of display 210 before reaching the under-display camera. In some cases, camera 220 and / or associated processor 270 may perform image processing to compensate for color deviations associated with the materials of display 210 through which light 245 passes before reaching camera 220, such as color filters. The light 245 received by the image sensor 130 of the camera 220 may be incident light that has passed through at least a portion of the display 210 before reaching the camera 220 (e.g., before impacting the image sensor 130 of the camera 220).

[0075] Several sound waves 260 are shown as arrows passing through the display 210. The sound waves 260 continue across the front surface 252 of the display 210 and propagate into the air. When near or within range of the mobile device 205, the user can hear the sound waves 260.

[0076] For simplicity, processor 270, memory 275 and battery 280 are not included. Figure 2C As shown in the diagram. Similarly, the data and power cables between the camera 220, speaker 225, processor 270, memory 275, and battery 280 are not connected. Figure 2C As shown in the image. It should be understood that these components can still exist. Figure 2C In the mobile device 205, its configuration is similar to Figure 2A The configuration shown is similar.

[0077] Figure 2D yes Figure 2A A front view of a mobile device 205, comprising a first under-display camera 220A, a second under-display camera 220B, and an under-display sensor 260. The first under-display camera 220A and the second under-display camera 220B may be referred to as first camera 220A and second camera 220B, respectively. The under-display sensor 260 may be referred to as sensor 260. Figure 2A Legend 290 is also included. Figure 2D In the middle. Mobile device 205 is an example of a device with multiple under-display cameras. Like... Figure 2A-2CCamera 220, first camera 220A and second camera 220B are located below display 210 along the Z-axis. Figure 2C In this context, the first camera 220A and the second camera 220B can be located between the display 210 and the housing 250 of the main body 215 of the mobile device 205, similar to Figure 2C The device includes a camera 220 and a speaker 225. A first camera 220A can receive incident light that passes through at least a first portion of the display 210 before reaching the first camera 220A. A second camera 220B can receive incident light that passes through at least a second portion of the display 210 before reaching the second camera 220B. The incident light reaching the cameras may include incident light that impacts the camera's image sensor. The first camera 220A and the second camera 220B can capture images based on receiving this incident light at their respective image sensors. The first camera 220A is closer to a first side of the mobile device 205, while the second camera 220B is closer to a second side of the mobile device 205. Figure 2D In the example mobile device 205 shown, the first side is the top of the mobile device 205, and the second side is the bottom of the bottom of the mobile device. In other examples, the first side and / or the second side can be either the left or the right side.

[0078] Since the first camera 220A and the second camera 220B are positioned at a certain distance, they can be used as stereo cameras. Within the same time window, the first camera 220A can capture a first image and the second camera 220B can capture a second image. The time window can be a predetermined duration, such as one picosecond or more, one nanosecond or more, one millisecond or more, one second or more, or a combination thereof. The mobile device 205 or another device receiving the first and second images from the mobile device 205 can use stereo image processing to process the first and second images to determine depth information corresponding to the first and second images. The depth information can refer to the depth of different objects depicted in the first and second images. Depth can refer to the distance between the subject and the mobile device 205. The greater the distance between the subject and the mobile device 205, the greater the depth of the object. The closer the distance between the subject and the mobile device 205, the lower the depth of the object. Stereo image processing can also be called stereo image processing, multi-image processing, binocular image processing, stereo camera processing, stereo camera processing, multi-camera processing, binocular camera processing, or a combination thereof.

[0079] Stereo image processing can use stereo parallax, binocular parallax, stereo vision, stereo triangulation, triangulation, stereo photogrammetry, photogrammetry, 3D mapping, 3D modeling, or combinations thereof to determine depth information corresponding to at least a subset of pixels in a first image and / or at least a subset of pixels in a second image. In some examples, stereo image processing may include: detecting a first depiction of a feature (e.g., a corner or edge) in the first image, and detecting a second depiction of the same feature in the second image. Stereo image processing can determine the depth of a feature by determining the distance between the location of the first depiction of the feature in the first image and the location of the second depiction of the feature in the second image. This distance may be referred to as stereo parallax, stereoscopic parallax, or binocular parallax. The greater the stereo parallax, the lower the depth of the feature, and the closer the feature is to the mobile device 205. The lower the stereo parallax, the higher the depth of the feature, and the farther the feature is from the mobile device 205.

[0080] For example, both the first and second images can depict people in front of the background, similar to... Figure 3A The display image 350 is displayed on the display 210. The mobile device 205 can use stereoscopic image processing to process the first and second images to identify pixels depicting a person that correspond to low depth values ​​because the person is close to the mobile device 205. The mobile device 205 can also use stereoscopic image processing to process the first and second images to identify pixels depicting a background that correspond to high depth values ​​because the background is far from the mobile device 205.

[0081] Mobile device 205 may include two or more under-display cameras below display 210. For example, mobile device 205 may include three, four, five, six, seven, eight, nine, ten, or more under-display cameras. A mobile device 205 with N under-display cameras below display 210 can capture at least N images of a scene within the same time window. Each of the N images of the scene can be captured by a different under-display camera among the N under-display cameras below display 210. Stereo image processing or multi-image processing can be used to process all N images to determine the depth information of features and / or pixels depicted in at least a subset of the N images. If N is greater than two, the determined depth information can be more accurate and detailed than when N is two.

[0082] In some examples, the first camera 220A and the second camera 220B share at least one component, component type, feature, function, or a combination thereof. For example, the first camera 220A may use the same type of lens as the second camera 220B, the first camera 220A may detect light within the same spectrum as the second camera 220B, the first camera 220A may use the same type of image sensor as the second camera 220B, the first camera 220A may use an image sensor of the same size as the second camera 220B, or a combination thereof.

[0083] In some examples, the first camera 220A differs from the second camera 220B by using at least one different component, component type, feature, function, or combination thereof. For example, the first camera 220A may use a first lens, while the second camera 220B may use a second lens. The first lens may have a wider angle of view than the second lens, for example, if the first lens is a wide-angle lens or a fisheye lens. The wider angle of view of the first lens can give the first camera 220A a wider field of view than the second camera 220B. The first lens may be a telephoto lens, while the second lens is not a telephoto lens, thus providing the first camera 220A with a greater focal distance than the second camera 220B. The first lens may be a zoom lens, while the second lens is not a zoom lens, thus providing the first camera 220A with greater zoom than the second camera 220B. In some examples, the image sensor of the first camera 220A detects light within a first spectrum, while the image sensor of the second camera 220B detects light within a second spectrum. Each spectrum may refer to a different subset of the entire electromagnetic spectrum. For example, the first spectrum may be the visible light (VL) spectrum, while the second spectrum may be the infrared (IR) spectrum. The second spectrum can be a near-infrared (NIR) spectrum, which is a subset of the IR spectrum adjacent to the VL spectrum. The second spectrum can also be an ultraviolet (UV) spectrum. In some examples, the first image sensor of the first camera 220A can be an image sensor of a different type than the second image sensor of the second camera 220B. For example, the first image sensor can be a complementary metal-oxide-semiconductor (CMOS) image sensor, while the second image sensor is a charge-coupled device (CCD) image sensor. The size of the first image sensor can differ from the size of the second image sensor, such that one is larger than the other. The first camera 220A and the second camera 220B can differ in any combination of the methods discussed herein.

[0084] Sensor 260 can be any type of sensor. For example, sensor 260 may include at least one of the following: a radio detection and ranging (RADAR) sensor, a light detection and ranging (LIDAR) sensor, an electromagnetic detection and ranging (EMDAR) sensor, a multispectral imaging detection and active reflection (MIDAR) sensor, a sound detection and ranging (SODAR) sensor, a sound navigation and ranging (SONAR) sensor, a laser rangefinder, a time-of-flight (TOF) transmitter / sensor, a structured light transmitter / sensor, a camera with an coded aperture, an active depth sensor, a distance sensor, a microphone, an accelerometer, a gyroscope, a Global Navigation Satellite System (GNSS) receiver, a Global Positioning System (GPS) receiver, another sensor discussed herein, or a combination thereof. Figure 2A-2C Like camera 220, sensor 260 is located below display 210. Figure 2C In this context, sensor 260 can be located between display 210 and housing 250 of the main body 215 of mobile device 205, similar to Figure 2C The sensor 260 includes a camera 220 and a speaker 225. The sensor 260 can receive incident light, sound, radio waves, microwaves, IR waves (e.g., NIR waves), UV waves, electromagnetic (EM) radiation, and / or other signals that pass through at least a portion of the display 210 before impacting the sensor 260. In some cases, the sensor 260 can output light, sound, electromagnetic (EM) radiation, and / or other signals that, after being output by the sensor 260, pass through at least a portion of the display 210 and are directed towards the environment. In some cases, the signal output by the sensor 260 can be reflected from objects in the environment, bounce back to the sensor 260, and be received by the sensor 260.

[0085] Figure 3A This is a front view of a mobile device 205 with an under-display camera 220, where a display 210 shows a black-bordered indicator 310A in the indicator area. The term "indicator area" can refer to the area in which the indicator (such as the black-bordered indicator 310A or other indicators) is displayed. The indicator area may include at least a subset of the light-passing area 290 above the camera 220. Figure 2A Legend 290 is also included. Figure 3AIn the mobile device 205, camera 220 captures one or more images. In some examples, camera 220 may capture images continuously and may temporarily store those images in an image buffer. Mobile device 205 is shown displaying display image 350 on display 210. In some examples, display image 350 is one of the images captured by camera 220 and / or stored in the image buffer on display 210. In some examples, display image 350 is based on one of the images captured by camera 220 and / or stored in the image buffer on display 210. In some examples, display image 350 is a modified variant of one of the images captured by camera 220 and / or stored in the image buffer on display 210. For example, mobile device 205 can generate display image 350 by modifying an image captured by camera 220 using the following modifications: cropping, enlarging, shrinking, stretching, tilting, distorting, warping, adjusting brightness, adjusting contrast, adjusting color saturation, adjusting tone mapping, adjusting white balance, adjusting black balance, adjusting histogram, adjusting levels, correcting red-eye, overlaying text, overlaying user interface elements, overlaying graphics, overlaying another image, overlaying elements of another image, merging an image with another image (e.g., generating an HDR image), any other image processing modifications discussed herein, or combinations thereof. In one example, display image 350 could be a version of an image captured by camera 220 that has been scaled so that most of the image is visible on display 210 based on the resolution of display 210; that is, cropped to fit the size of display 210, and its brightness, contrast, and saturation adjusted to display well on display 210. Display image 350 can be used as a preview image of the field of view of camera 220 by previewing display image 350, which is based on an image recently captured by camera 220 (e.g., an image recently captured by camera 220). Figure 3A The image 350 displayed on the monitor 210 is a selfie taken by the user in front of a background. The background includes buildings, trees, grass, and other plants.

[0086] Because the camera 220 is located below the display 210, the user of the mobile device 205 may not know where the camera 220 is relative to the front surface 255 of the display 210. It is desirable to capture images simultaneously (e.g., while they are being displayed on the display 210). Figure 3A During the display of a 350-pixel selfie image, the user looking at the camera might not know where to look. To solve this problem, Figure 3AMobile device 205 displays a border-shaped indicator 310A overlaid on the displayed image 350 within the indicator area of ​​display 210. The border-shaped indicator 310 has a rounded rectangular shape, outlining the image area of ​​camera 220 located below display 210 along the z-axis while display image 350 is displayed on display 210. Indicator 310 is shown as a black shaded area. Mobile device 205 displays indicator 310A by making multiple pixels grouped together in the indicator area appear black. Depending on the type of display 210, display 210 can indicate which pixels are black by turning off the power to the pixels. A user of mobile device 205 who wants to look at camera 220 knows to look within the indicator area where the border-shaped indicator 310A is displayed. Figure 3A The rounded rectangular shape of the indicator 310A mimics the border. Because the pixels of the display 210 are displayed as black in the indicator 310A, any interference to the light 245 that reaches the camera 220 after passing through the display 210 is minimized.

[0087] Indicator 310A is large enough that a user might not be able to discern which part of indicator 310A is positioned above camera 220 along the Z-axis. To help the user identify the position of camera 220, the indicator area may include an additional indicator for identifying the position of camera 220 below display 210, which can overlap indicator 310A on display 210. For example, two indicator arrows 340 pointing to camera 220 are shown overlapping. Figure 3A Above the bordered indicator 310A in the indicator area on the display 210. The indicator arrow may be displayed in a different color than the rest of the indicator area 310. Other types of additional indicators may overlap the indicator 310A or otherwise cover the indicator area on the display 210, such as indicator shapes. Indicator shapes may outline an area smaller than the indicator 310A within the indicator area. For example, an indicator shape may outline the position of a camera 220 positioned below the smaller area along the Z-axis. Examples of indicator shapes include... Figure 4 Indicator 410 and Figure 5 Indicator 510.

[0088] Mobile device 205 includes an image capture hardware button 330. Mobile device 205 can receive image capture input when the image capture hardware button 330 is pressed. In some cases, the image capture hardware button 330 can be replaced and / or supplemented by another hardware user interface that receives image capture input, such as a switch, a touch-sensitive surface separate from display 210, a fingerprint sensor, another type of biometric scanner, a microphone, a wheel, a knob, a slider, an accelerometer, a gyroscope, or some combination thereof. Mobile device 205 also includes an image capture user interface (UI) element 320, which is displayed on the UI area of ​​display 210, overlaying the displayed image 350. Mobile device 205 can receive image capture input when a touch area at the UI area of ​​display 210 is received on the touch-sensitive touchscreen surface of display 210.

[0089] In some examples, receiving the image capture input triggers the camera 220 of the mobile device 205 to capture a second image, and also triggers the mobile device 205 to receive and store the second image from the camera 220. In some examples, receiving the image capture input triggers the transfer of the image already captured by the camera 220 of the mobile device 205 from temporary storage in the image buffer to long-term non-temporary storage in the memory 175.

[0090] Figure 3B This is a front view of a mobile device 205 with an under-display camera 220, where a display 210 shows a black notch-shaped indicator 310B above the camera 220. The mobile device 205 displays the black notch-shaped indicator 310B in the indicator area located above the camera 220 along the Z-axis. Figure 2A Legend 290 is also included. Figure 3B middle. Figure 3B The mobile device 205 is similar to Figure 3A Mobile device 205. Figure 3A The border-shaped indicator 310A is made of Figure 3B The notch-shaped indicator 310B is used instead. The notch-shaped indicator 310B can occupy less part of the display 210 than the bezel-shaped indicator 310A, thus allowing the displayed image 350 to fill more part of the display 210.

[0091] Figure 3C This is a front view of a mobile device 205 with an under-display camera 220, where a display 210 shows a black circular indicator 310C above the camera 220. The mobile device 205 displays the black circular indicator 310C in an indicator area located above the camera 220 along the Z-axis. Figure 2A Legend 290 is also included. Figure 3C middle. Figure 3C The mobile device 205 is similar to Figure 3A and Figure 3B Mobile device 205. Figure 3A The border-shaped indicator 310A is made of Figure 3C The circular indicator 310C is used instead. The circular indicator 310C can occupy less part of the display 210 than the bezel-shaped indicator 310A or the notch-shaped indicator 310B, thus allowing the displayed image 350 to fill more part of the display 210.

[0092] Indicators 310A, 310B and 310C in Figure 3A , Figure 3B and Figure 3C The pixels shown are either filled with black pixels or are turned off. In some examples, indicators 310A, 310B, and / or 310C may be filled with pixels of different colors, such as white, gray, red, green, blue, yellow, or some mixture of these colors. The color filled by indicators 310A, 310B, and / or 310C can have any hue, any brightness, and any opacity. In some examples, indicators 310A, 310B, and 310C may be filled with patterns, textures, gradients, or even another image different from the display image 350 over which indicators 310A, 310B, and 310C are overlaid. In some examples, indicators 310A, 310B, and 310C may be at least partially translucent or transparent.

[0093] In some examples, indicators 310A, 310B, and 310C may be consistently overlaid on the displayed image 350 for a period of time, such as during a period when a photo capture software application is running on the mobile device 205 and / or during a period when the under-display camera 220 is activated. In some examples, indicators 310A, 310B, and 310C may periodically blink on and off during this period of time. In some examples, indicators 310A, 310B, and 310C periodically fade in and out during this period of time. In some examples, indicators may be displayed, highlighted, emphasized, hidden, or some combination thereof in response to user input. User input includes at least one of the following: tapping on the display, hovering over the display, pre-configured gestures across the display, user preferences, or voice input from the user. In some examples, at least a subset of indicators 310A, 310B, or 310C are animated, for example, with boundaries that include periodic expansion, contraction, pulsation, or some combination thereof.

[0094] In some examples, mobile device 205 generates display image 350 such that it includes indicators 310A, 310B, and / or 310C to display indicators 310A, 310B, and / or 310C overlaid on display image 350. Camera 220 may capture an image, which mobile device 205 may modify as described above to generate display image 350. Mobile device 205 may further modify display image 350 to merge display image 350 with indicators 310A, 310B, and / or indicators 310C. In some examples, indicators 310A, 310B, and / or 310C overlay display image 350, but display image 350 remains separate from indicators 310A, 310B, and / or 310C. In some examples, mobile device 205 uses blending effects to overlay indicators 310A, 310B, and / or 310C on display image 350. Blending effects can include, for example, dissolve, multiply, darken, burn, linear burn, brighten, screen effect, dodge, linear dodge, overlay, soft light, hard light, glare, needle light, hard blend, difference, exclusion, subtraction, segmentation, threshold, hue blend, saturation blend, color blend, brightness blend, brighter color comparison, darker color comparison, or some combination thereof.

[0095] Figure 4 This is a front view of a mobile device 205 with an under-display camera 220, wherein the display 210 displays a triangular indicator 410 around the area above the camera 220. Figure 2A Legend 290 is also included. Figure 4 Middle. Similar to Figure 3A , Figure 3B and Figure 3C Mobile device 205, Figure 4 The mobile device 205 is illustrated as displaying a display image 350 using its display 210. The display image 350 may be based on an image captured by the camera 220. The display image 350 may be temporarily displayed on the display 210 as a preview image to indicate to the user of the mobile device 205 how the scene from which the camera 220 receives light is framed within the camera 220's field of view. The mobile device 205 displays an indicator 410 overlaid on the display image 350. The indicator 410 includes an indicator shape that outlines an area located above the camera 220. Figure 4The example indicator 410 shown is a triangle. Indicator 410 can be any shape. For example, indicator 410 can be a circle, semicircle, square, triangle, rectangle, ellipse, trapezoid, parallelogram, rhombus, kite shape, quadrilateral, pentagon, hexagon, heptagon, octagon, nonagon, decagon, polygon with more than 10 sides, or other shapes.

[0096] Attached to the triangular region outline, indicator 410 in Figure 4 The diagram shows a shadow pattern filled in. This shadow pattern can represent a shadowed area within the indicator shape of indicator 410 or an area otherwise filled, similar to the black shadow of indicators 310A, 310B, and 310C. The shadowed area within the outlined indicator shape of indicator 410 can have any combination of the characteristics and properties discussed with respect to indicators 310A, 310B, and 310C. For example, the shadowed area within the outlined indicator shape of indicator 410 can be patterned and semi-transparent. In some cases, the outlined indicator shape of indicator 410 serves as the boundary or edge of the shadowed area. In some cases, the shadowed area of ​​indicator 410 and the outlined indicator shape of indicator 410 cover overlapping but distinct areas. In some examples, the shadowed area can be removed from indicator 410, or it can periodically blink on and off. Indicator 410 may be a unique indicator of the position of camera 220 below display 210, or it may be used in conjunction with other indicators such as indicators 310A-C and / or indicator arrows 340.

[0097] The lines constituting the outlined indicator shape of indicator 410 can be of any color, hue, brightness, transparency, and / or thickness. The lines constituting the outlined indicator shape of indicator 410 can be solid colors, can be patterned, can include gradients, and / or can form an image 410 different from the display image 350 covered by the outlined indicator shape. The lines constituting the outlined indicator shape of indicator 410 can include solid lines, dashed lines, dotted lines, dotted-dash lines, dotted-dashed lines, any other type of line, or combinations thereof. The lines constituting the outlined indicator shape of indicator 410 can include single lines, double lines, triple lines, quadruple lines, more than four lines, or combinations thereof. In some examples, the outlined indicator shape of indicator 410 consistently covers the display image 350 for a period of time, such as during a period when a photo capture software application is running on mobile device 205 and / or during a period when the under-display camera 220 is activated. In some examples, the outlined indicator shape of indicator 410 periodically flashes on and off during this period. In some examples, the outlined indicator shape of indicator 410 fades in and out periodically during the time period. In some examples, at least a subset of the outlined indicator shape of indicator 410 is animated, such as periodically expanding, contracting, pulsating, or some combination thereof.

[0098] In some examples, the mobile device 205 displays the indicator 410 overlaid on the display image 350 by modifying the display image 350 to include the indicator 410. The modified display image may be a copy of the image merged with the indicator 410. In some examples, the mobile device 205 uses blending effects to blend the indicator 410 with the display image 350 it overlays, such as any blending effects discussed regarding indicator areas 310A, 310B, and 310C.

[0099] Figure 4 The mobile device 205 also includes image capture UI elements 320 and image capture hardware buttons 330. These elements can be similar to... Figure 3A , Figure 3B and Figure 3C The image capture UI element 320 and the image capture hardware button 330 are functional.

[0100] In some cases, indicator 410 or any such indicator may permanently overlay the display image 350 on the display in an area through which camera 220 receives incident light. In other cases, indicator 410 or any such indicator may be hidden by default and may overlay the display image 350 on the display in response to input. Input may be a touch on the touch-sensitive surface of the display, a predefined gesture across the touch-sensitive surface of the display, a hovering gesture above the display, or voice input narrating a predefined phrase recorded by a microphone. For example, a user of mobile device 205 may touch the touch-sensitive surface of display 210, triggering mobile device 205 to display indicator 410. The user may speak the phrase “Where is my camera?” or a similar phrase, which can serve as voice input. The microphone of mobile device 205 (or coupled to mobile device 205) can record the phrase spoken by the user as voice input, and mobile device 205 (or a remote voice-to-text interpretation server) can recognize that the voice input is a predefined phrase that triggers the display of indicator 410, thus triggering mobile device 205 to display indicator 410.

[0101] Figure 5 This is a front view of a mobile device 205 with an under-display camera 220, wherein a display 210 displays a circular indicator 510 and a region selection user interface (UI) element 520 above the camera 220 to display a selectable region 540 above the camera 220. Figure 2A Legend 290 is also included. Figure 5 Middle. Similar to Figure 3A , Figure 3B , Figure 3C and Figure 4 Mobile device 205, Figure 5 The mobile device 205 is illustrated as displaying a display image 350 using its display 210. The display image 350 may be based on an image captured by the camera 220. The display image 350 may be temporarily displayed on the display 210 as a preview image to indicate to the user of the mobile device 205 how the scene from which the camera 220 receives light is framed within the camera 220's field of view. The mobile device 205 displays a circular indicator 510 overlaid on the display image 350. The circular indicator 510 outlines the area located above the camera 220 and is not filled in.

[0102] In some software applications and associated user interfaces running on mobile device 205, selecting certain portions of an image may be useful. Image capture software applications and their associated user interfaces can be configured to allow selection of areas of image 350. For example, image capture software applications can be configured to select a selectable area 540 of image 350 in response to touch input or touch gesture input within or around a selectable area 540 of the touchscreen displaying image 350. In some cases, image capture software applications can be configured to select a selectable area 540 of image 350 in response to hover input, point input, or hover gesture input, where the user hovers over (along the Z-axis) and / or makes a gesture (along the Z-axis) over the selectable area 540. In some cases, image capture software applications can be configured to select a selectable area 540 of image 350 in response to voice input uttering a predefined phrase recorded by a microphone. For example, a user of mobile device 205 can say the phrase “select an area above the camera” or a similar phrase, which can be used as voice input. The microphone of mobile device 205 (or coupled to mobile device 205) can record the phrase spoken by the user as voice input, and mobile device 205 (or remote voice-to-text interpretation server) can recognize that the voice input is a predefined phrase that triggers the selection of selectable region 540, triggering mobile device 205 to select selectable region 540.

[0103] In some examples, the selection of a region in the preview image causes the mobile device 205 to instruct the camera 220 to perform autofocus to adjust the camera 220's focus to improve focus on a portion of the scene depicted in the selected selectable region 540 of the displayed image 350. This provides an intuitive interface that allows the user of the mobile device to modify the camera's focus, for example, from focusing on an object in the scene's background to focusing on an object in the scene's foreground, and vice versa. For example, in Figure 4The image shown is a display image 350 on display 210, where a selectable area 540 depicts a portion of a face. A user can touch the touch-sensitive surface of display 210 at the selectable area 540, or hover over the surface of display 210 at the selectable area 540, point at the selectable area 540, gesture toward or around the selectable area 540, or otherwise provide input at the selectable area 540. This input can be referred to as touch input, hover input, point input, gesture input, or other types of input based on the type of interaction with display 210. The mobile device 205, upon detecting this input, recognizes the coordinates of the input as matching the coordinates of the area of ​​display 210 on which the preview image depicting the face is displayed. Mobile device 205 instructs camera 220 to adjust its focus to improve focus on the face, for example, by shifting from a first focus setting optimized for focusing on another object in the scene to a second focus setting optimized for focusing on the face. While camera 220's focus is set to the second focus setting, camera 220 can then capture a second image. Due to the change in focus, the second image may be more focused on the face than the displayed image 350. The face may appear sharper and clearer in the second image than in the displayed image 350.

[0104] In some examples, the selection of a region or area of ​​the displayed image 350 (e.g., selectable region 540) can cause the mobile device 205 to instruct the camera 220 to set one or more settings associated with one or more attributes of the camera 220 to improve the application of one or more attributes based on a portion of the captured scene depicted in the selected region or area of ​​the displayed image 350. For example, one or more attributes may include white balance, black level compensation, gamma correction, gain, color correction, color saturation, noise filter, contrast control, brightness control, tone mapping, sharpness, blur, red-eye correction, or some combination thereof. While one or more settings associated with one or more attributes of the camera 220 are set, the camera 220 may capture a second image. The camera 220 may then capture a second image while setting one or more settings associated with one or more attributes of the camera 220. Assuming the selected region or area is selectable region 540, because one or more settings are set based on the selection of selectable region 540, the second image may use settings that improve the depiction of a face compared to the depiction of a face in the displayed image 350. The face may appear clearer and sharper in the second image than in the displayed image 350.

[0105] Because camera 220 is located below display 210 along the Z-axis, the reception of certain inputs (such as inputs for selecting certain areas of the displayed image 350) obstructs the view of camera 220. Such inputs may include touch input, touch gesture input, hover input, point input, hover gesture input, or any other type of input discussed herein. For example, when the under-display camera 220 is active, a user providing touch input or touch gesture input by interacting with the touch-sensitive touchscreen surface of display 210 may at least partially obstruct the view of camera 220 by touching the portion of display 210 above the under-display camera 220 with the user's finger, stylus, hand, arm, or pointer. Even if the touch input is received in a specific area of ​​display 210 that is not directly above camera 220, a portion of the user's arm or hand may reach above camera 220 and thus at least partially obstruct the view of camera 220 to reach that specific area of ​​display 210. Similarly, users can provide hover input, pointing input, or hover gesture input by hovering their hand and / or pointing the device above display 210, pointing at display 210, and / or gesturing above display 210. Such hover input, pointing input, and hover gesture input can also block camera 220 by obstructing the area of ​​the display from light that would otherwise pass through before reaching camera 220.

[0106] In some examples, occlusion of the field of view of camera 220 may refer to camera 220 being turned away from the target field of view (FOV). Mobile device 205 may determine the target FOV based on the software application it is running. For example, if mobile device 205 is running a video conferencing software application, it may determine that the target FOV is any FOV of camera 220 that includes a depiction of the user's face. If mobile device 205 is moved (e.g., accidentally dropped), causing the FOV of camera 220 to no longer conform to the predetermined criteria of the target FOV, because the FOV of camera 220 no longer includes a depiction of the user's face, mobile device 205 may consider any time during which the FOV of camera 220 does not match the target FOV as a time during which camera 220 is occluded.

[0107] Figure 5 The mobile device 205 includes a region selection hardware button 530. The mobile device 205 can receive region selection input when the region selection hardware button 530 is pressed. In some cases, the region selection hardware button 530 can be replaced and / or supplemented by another hardware user interface that receives image capture input. In response to the receipt of region selection input, the mobile device 205 can select a selectable region 540 of the display image 350 displayed on the display 210.

[0108] Mobile device 205 also includes a region selection user interface (UI) element 520, which is displayed on the UI area of ​​display 210, overlaying the displayed image 350. Mobile device 205 can receive region selection input at the region selection user UI element 520 of display 210. Region selection input can include touch input, touch gesture input, hover input, hover pointing input, hover gesture input, or any other type of input discussed herein. For example, touch input or touch gesture input can include a touch receiving touch area at the region selection user UI element 520 of display 210 on the touch-sensitive touchscreen surface of display 210. Hover input can include a user hovering over the region selection user UI element 520 of display 210 (in the Z-axis). In some cases, voice input can be used. In some cases, image capture software applications can be configured to select a selectable region 540 of displayed image 350 in response to voice input uttering a predefined phrase recorded by a microphone, as discussed herein.

[0109] As described above, using region selection input from the region selection hardware button 530 and / or the region selection UI element 520 to select a selectable region 540 of the displayed image 350 can cause the mobile device 205 to instruct the camera 220 to set one or more settings associated with one or more attributes of the camera 220 to improve the application of one or more attributes based on a portion of the shooting scene depicted in the selectable region 540 of the displayed image 350. For example, using region selection input to select the selectable region 540 of the displayed image 350 can cause the mobile device 205 to instruct the camera 220 to adjust its focus to improve focusing on a portion of the shooting scene depicted in the selectable region 540 of the displayed image 350. Because the user can select the selectable region 540 of the displayed image 350 by pressing the region selection hardware button 530 and / or by providing input at the region selection UI element 520, the user no longer needs to obscure the camera 220 to select the selectable region 540. Although Figure 5 The selectable area 540 shown includes the portion of the display 210 located directly above the camera 220, but the selectable area does not necessarily need to include the portion of the display 210 located directly above the camera 220. In some cases, the selectable area 540 for displaying the image 350 may include a portion of the image 350 that the user may need to reach above the camera 220 to touch, hover over, point at, gesture toward, or otherwise provide input to in this way.

[0110] To clarify the selectable area 540 of the display image 350 when input is received at the area selection UI element 520. Figure 5Example area selection UI element 520 may include the word "select" on a visual copy of the selectable area 540 of the image. Alternatively, an input indicator (e.g., an arrow indicator, a hand indicator, a palm indicator, etc.) may be provided to guide the user to touch the selectable area. The visual copy of the selectable area 540 displaying the image 350 may be modified (e.g., reduced, enlarged, scaled, cropped, stretched, squeezed, tilted, distorted, or a combination thereof) to fit the size of the area selection UI element 520. In some examples, the area selection UI element 520 may act as a virtual button and may select the entire selectable area 540 when any part of the area selection UI element 520 receives input such as touch, hover, pointing, gesture, any input type or combination thereof discussed herein. In some examples, the visual copy of the selectable area 540 of the image 350 displayed in the area selection UI element 520 may act as an offset version of the selectable area 540. A user can select a specific portion of the selectable region 540 of the display image 350 by providing input to a visual copy of that selectable region 540 within the region selection UI element 520. In some examples, the region selection UI element 520 can control a cursor. A user can move the cursor by swiping, sliding, or otherwise moving touch input at the region selection UI element 520. A similar swipe or slide gesture can be used to move the cursor while hovering over the region selection UI element 520. Selection of the selectable region 540 can be performed by moving the cursor to the selectable region 540 of the display image 350 and tapping the region selection UI element 520, pressing a button such as the region selection hardware button 530, using a pointing gesture, or triggering another hardware interface element. In some examples, the mobile device 205 can also allow the user to select the selectable region 540 of the display image 350 by drawing a shape around the selectable region 540 using the cursor. In some examples, the mobile device 205 may also allow the user to select the selectable area 540 for displaying the image 350 by drawing a shape around the selectable area 540 using touch input, hover input, gesture input, or another form of input around the selectable area 540.

[0111] Figure 5 The mobile device 205 also includes image capture UI elements 320 and image capture hardware buttons 330. These elements can be similar to... Figure 3A , Figure 3B , Figure 3C and Figure 4The image capture UI element 320 and image capture hardware button 330 function. While the image capture hardware button 330 and region selection hardware button 530 are shown along one side of the mobile device 205, one or both can be positioned along different parts of the mobile device 205. For example, one or both can be positioned along the rear, top, or bottom of the mobile device 205. Any other hardware interface element used instead of the image capture hardware button 330 and / or region selection hardware button 530 can similarly be positioned along the rear, top, or bottom of the mobile device 205.

[0112] Figure 6 This is a front view of the mobile device 205 that compensates for the occlusion of the under-display camera 220 by the user's hand 605 during touch, hover, or gesture input. Figure 2A Legend 290 is also included. Figure 6 Middle. Similar to Figure 3A , Figure 3B , Figure 3C , Figure 4 and Figure 5 Mobile device 205, Figure 6 The mobile device 205 is illustrated to display a display image 350 using its display 210. The display image 350 may be based on an image captured by the camera 220. The display image 350 may be temporarily displayed on the display 210 as a preview image to indicate to the user of the mobile device 205 how the scene from which the camera 220 receives light is framed within the field of view of the camera 220.

[0113] exist Figure 6 Hand 605 is shown as obscuring the under-display camera 220 of mobile device 205. The finger of hand 605 extends outwards, with the fingertip positioned above camera 220, blocking light from the scene from reaching camera 220. The extended finger of hand 605 can touch the touch input area of ​​the touch-sensitive touchscreen surface of display 210, hover over the input area of ​​display 210, point towards the input area of ​​display 210, gesture toward the input area of ​​display 210, gesture around the input area of ​​display 210, or some combination thereof. The input area of ​​display 210 can be... Figure 5 The selectable area 540, or another selectable area for displaying image 350. This input to the display's input area can be used to select a selectable area for displaying image 350, such as selectable area 540 for displaying image 350. The input area includes the area of ​​display 210 above camera 220 through which incident light reaching camera 220 passes. The input area may be located above camera 220, adjacent to camera 220, and / or within a predetermined radius of camera 220.

[0114] exist Figure 6The image timeline 620 captured by camera 220 is shown. The timeline 620 includes a first image frame 610A, a second image frame 610B, a third image frame 610C, a fourth image frame 610D, and a fifth image frame 610E. In some examples, image frames 610A-E in the timeline 620 may be image frames captured consecutively by camera 220. In some examples, one or more additional image frames may exist between any two image frames in the timeline 620. Image frames 610A-E in the timeline 620 depict scenes similar to those depicted in display image 350, but in some cases include occlusion and / or other variations.

[0115] In timeline 620, the first image frame 610A depicts a scene before a person in the foreground is depicted against a background of buildings, grass, and trees. The first image frame 610A is not occluded. The second image frame 610B is captured by camera 220 after the first image frame 610A. The second image frame 610B is partially occluded by an occluder 630A, which appears as a black area or dark area covering a portion of the second image frame 610B. The occluder 630A may be caused, for example, by a portion of the extended finger of hand 605 approaching the field of view of camera 220 as hand 605 moves. Hand 605 blocks some light from the scene that would otherwise pass through display 210 and reach camera 220. The third image frame 610C is captured by camera 220 after the second image frame 610B. The third image frame 610C is completely occluded by the occluder 630B and is therefore completely covered by black or darkness. Obstruction 630B may be caused, for example, by an extended finger of hand 605 completely obstructing the field of view of camera 220 as hand 605 moves. Hand 605 blocks most or all of the light from the scene that would otherwise pass through display 210 and reach camera 220. Fourth image frame 610D is captured by camera 220 after third image frame 610C. Fourth image frame 610D is partially obstructed by obstruction 630C, which manifests as a black area or dark area covering a portion of fourth image frame 610D. Hand 605 blocks some light from the scene that would otherwise pass through display 210 and reach camera 220. For example, obstruction 630C may be caused by a portion of an extended finger of hand 605 leaving the field of view of camera 220 as hand 605 moves. Fifth image frame 610E is captured by camera 220 after fourth image frame 610D. The fifth image frame 610E depicts a scene similar to the first image frame 610A, although some minor changes occur in the scene due to the passage of time between the capture of the first image frame 610A and the fifth image frame 610E. For example, the scene depicted in the fifth image frame 610E includes birds in the sky that are not present in the first image frame 610A.

[0116] Displaying the second image frame 610B, the third image frame 610C, and / or the fourth image frame 610D as preview images on display 210 will result in the display of obstructions 630A-C on display 210. In some cases, the user may find the presence of obstructions 630A-C in the preview image unhelpful. In some examples, during the duration for which camera 220 is at least partially obstructed, mobile device 205 displays the last unobstructed image frame preceding the obstruction. In timeline 620, the last unobstructed image frame preceding the obstruction is the first image frame 610A. This is indicated by arrow 640. In some examples, during the duration for which camera 220 is completely obstructed (such as during the obstruction 630B of the third image frame 610C), mobile device 205 displays the last partially obstructed image frame preceding the fully obstructed 630B. In timeline 620, the last partially obstructed image frame preceding the fully obstructed 630B is the second image frame 610B. In some examples, the mobile device 205 displays the last image frame in which the field of view of camera 220 is less than a predetermined maximum threshold percentage is occluded for a period of time, while the field of view of camera 220 is greater than a predetermined maximum threshold percentage is occluded.

[0117] In some cases, touch input or other input can be received while the camera 220 is at least partially obscured. In some examples, the mobile device 205 can receive touch input in response to an extended finger of hand 605 touching a touch input area on the touch-sensitive touchscreen surface of display 210 after the capture of the first image frame 610A and before the capture of the fifth image frame 610E. The mobile device 205 can select a selectable area of ​​the image based on the location of the received touch input on display 210 at least partially overlapping with the selectable area 540, for example... Figure 5 The selectable area 540. The mobile device 205 can instruct the camera 220 to set one or more settings 220 for one or more camera attributes to adjust the application of one or more attributes based on a portion of the captured scene depicted in the selected area of ​​the next unobstructed image frame after the removal of the occlusion. In the timeline 620, the next unobstructed image frame after the occlusion is the fifth image frame 610E. Therefore, the mobile device 205 can instruct the camera 220 to set one or more settings for one or more camera attributes to adjust the application of one or more attributes based on a portion of the captured scene depicted in the selected area of ​​the last unobstructed image frame before the occlusion.

[0118] In an illustrative example, while camera 220 is at least partially obscured, mobile device 205 can still display a first image frame 610A, and a user can tap a selectable area of ​​the first image frame 610A depicting the user's face with their hand 605. Mobile device 205 receives touch input on the touch-sensitive touchscreen surface of display 210 at a portion of the selectable area displaying the first image frame 610A. Detecting this touch input, mobile device 205 recognizes that the coordinates of the touch input match the coordinates of display 210, displaying the area of ​​the first image frame 610A depicting the user's face on the display. Mobile device 205 instructs camera 220 to adjust its focus to improve focus on the user's face for the next unobstructed image frame (fifth image frame 610E) after the obstruction. Mobile device 205 shifts from a first focus setting optimized for focusing on another object in the scene to a second focus setting optimized for focusing on the user's face. Once camera 220 captures the fifth image frame 610E, the focus of camera 220 may have already been set to the second focus setting optimized for focusing on the user's face.

[0119] Figure 7 This is a front view of a mobile device 205 having multiple under-display cameras 220A-220B and under-display sensors 260A-260B, wherein a display 210 shows an interface area 770 for controlling the multiple under-display cameras 220A-220B and under-display sensors 260A-260B. Figure 2A Legend 290 is also included. Figure 7 Middle. Similar to Figure 2D Mobile device 205, Figure 7 The mobile device 205 includes a first under-display camera 220A and a second under-display camera 220B. The first under-display camera 220A and the second under-display camera 220B can each be any type of camera and have any camera attributes, as described herein regarding camera 220, Figure 2D The first under-screen camera 220A Figure 2D The second under-screen camera 220B, image capture device 105A, image processing device 105B, image capture and processing system 100, any other camera discussed herein, or any combination thereof. The first under-screen camera 220A may also be referred to as the first camera 220A. The second under-screen camera 220B may also be referred to as the second camera 220B.

[0120] Similar to Figure 3A , Figure 3B , Figure 3C , Figure 4 , Figure 5 and Figure 6 Mobile device 205, Figure 7The mobile device 205 is illustrated to display a display image 350 using its display 210. The display image 350 may be based on an image captured by a first camera 220A or a second camera 220B. The display image 350 may be temporarily displayed on the display 210 as a preview image to indicate to the user of the mobile device 205 the field of view of the first camera 220A or the field of view of the second camera 220B.

[0121] Figure 7 The second camera 220B performs in conjunction with Figure 5 The same position as camera 220B. Display 210 displays a circular indicator 510B within a selectable area 540B around and above the second camera 220B. Figure 7 The second indicator 510B is circular and is similar to... Figure 5 The indicator is the same as 510. Figure 7 The second selectable area 540B above the second camera 220B is also circular and exhibits characteristics similar to... Figure 5 The second camera 220B has the same shape, size, and position as the selectable area 540 above the camera 220. In other examples, the second camera 220B, the second indicator 510B, and / or the second selectable area 540B may differ from the second camera 220B. Figure 5 Those in it.

[0122] Figure 7 The first camera 220A is positioned near the top of the display 210, similar to... Figure 2D The first camera in the series, 220A, Figure 3A Camera 220 in Figure 3B Camera 220 in Figure 3C Camera 220 and Figure 4 Camera 220. Display 210 displays a first indicator 510A around the first camera 220A, the first indicator 510A being a rounded rectangle and located within a first selectable area 540A above the first camera 220A. Figure 7 The first selectable area 540A above the first camera 220A is a large rounded rectangle that includes most of the sky depicted in the display image 350. In other examples, the second camera 220B, the second indicator 510B, and / or the second selectable area 540B may be... Figure 5 Those differences.

[0123] The interface area 770 is represented as a semi-transparent shadow area covering the image displayed on the display 210. The interface area 770 includes an image capture UI element 320, a first area selection UI element 720A, a second area selection UI element 720B, and a sensor control UI element 740. Figure 7The function of selecting UI element 720A in the first area and UI element 720B in the second area is similar. Figure 5 The area selection UI element is 520. Similarly, Figure 7 The functions of the first area selection hardware button 730A and the second area selection hardware button 730B are similar. Figure 5 The mobile device 205 selects a first selectable area 540A upon receiving input at the first area selection UI element 720A. The mobile device 205 selects a first selectable area 540A upon receiving input at the first area selection hardware button 730A. A user wishing to select an area such as the sky depicted in the displayed image 350 can provide input to the first area selection UI element 720A and / or provide input to the first area selection hardware button 730A to do so. The mobile device 205 selects a second selectable area 540B upon receiving input at the second area selection UI element 720B. The mobile device 205 selects a second selectable area 540B upon receiving input at the second area selection hardware button 730B. A user wishing to select an area depicted in the displayed image 350 can provide input to the second area selection UI element 720B and / or provide input to the second area selection hardware button 730B to do so.

[0124] In some cases, either the first camera 220A or the second camera 220B can be disabled for a specific purpose or software application. In other cases, either the first camera 220A or the second camera 220B can be activated for a specific purpose or software application. For example, if the mobile device 205 is running video conferencing software for video conferencing, the first camera 220A can be disabled for video conferencing, and the second camera 220B can be activated for video conferencing. In the video conference, the user of the mobile device 205 is having a video conference with a second user. The user of the mobile device 205 is typically looking at the face of the second user displayed on the display 210. The second user's face may be roughly displayed in the center of the display 210. During such a video conference, the user of the mobile device 205 can naturally look towards the center of the display 210 because they are looking at another user's face. Because the second camera 220B is closer to the center of the display 210 than the first camera 220A, the user looking at the face of another user displayed on the display 210 also looks directly at the second camera 220B or at a point closer to the second camera 220B than the first camera 220A. Therefore, the image captured by the second camera 220B while the user is looking at their face displayed on monitor 210 depicts the user making eye contact with the second camera 220B, or with a point close to the second camera 220B. By using the second camera 220B instead of the first camera 220A in the video conference, the image of the user making eye contact, captured by the second camera 220B, is sent to and displayed on the second user's device. The image captured by the second camera 220B when displayed on the second user's device subsequently appears to be making eye contact with the second user when the second user views the image displayed on their device. In video conferencing, it is generally not necessary to touch the center of monitor 210, hover over the center of monitor 210, or otherwise interact with the center of monitor 210, so obstruction by the second camera 220B is unlikely during video conferencing. Therefore, using the second camera 220B in video conferencing improves video conferencing by improving the representation of eye contact between the user and the second user. This representation of eye contact is generally not achievable with a more offset camera, such as the first camera 220A, except through image manipulation by modifying the depiction of the user's eyes. Such image manipulation can appear unnatural and produce the uncanny valley effect.

[0125] On the other hand, software applications that anticipate and / or expect users to select an area for displaying image 350 may use the first camera 220A and disable the second camera 220B. The first camera 220A is closer to the top side of the mobile device 205 than the second camera 220B. Therefore, the first camera 220A is less likely to be obscured by a user holding the mobile device 205, where the bottom side of the mobile device 205 is closer to the user's body than the top side. For example, software applications in which users draw or otherwise edit their own selfies may use the first camera 220A and disable the second camera 220B because input to select an area for displaying image 350 is anticipated and / or expected in such software applications.

[0126] In some cases, mobile device 205 can switch which camera is active and which is disabled based on camera occlusion. For example, if first camera 220A is active and mobile device 205 detects an obstruction similar to one of the obstructions 630A-C that blocks light received by first camera 220A, then mobile device 205 can disable first camera 220A and activate second camera 220B. Similarly, if second camera 220B is active and mobile device 205 detects an obstruction similar to one of the obstructions 630A-C that blocks light received by second camera 220B, then mobile device 205 can disable second camera 220B and activate first camera 220A.

[0127] Similar to Figure 2D The under-display sensor 260 of the mobile device 205, Figure 7 The mobile device 205 includes a first under-display sensor 260A and a second under-display sensor 260B. Both the first under-display sensor 260A and the second under-display sensor 260B can be any type of sensor and have any sensor properties discussed herein with respect to under-display sensor 260, any other camera or sensor discussed herein, or any combination thereof. The mobile device 205 can trigger use and / or control of one or both of sensors 260A and / or sensors 260B upon receiving input at sensor control UI element 740 and / or upon receiving input at sensor control hardware button 775. Although in Figure 7The diagram shows a single sensor control UI element 740 and a single sensor control hardware button 775. However, in some cases, the mobile device 205 may include a separate sensor control UI element 740 and / or a separate sensor control hardware button 775 for each of the sensors 260A-260B. When input is received at the sensor control UI element 740 and / or the sensor control hardware button 775 corresponding to the first sensor 260A, the mobile device 205 can select to display an image 350 over a selectable area of ​​the first sensor 260A. When input is received at the sensor control UI element 740 and / or the sensor control hardware button 775 corresponding to the second sensor 260B, the mobile device 205 can select to display an image 350 over a selectable area of ​​the second sensor 260B.

[0128] Mobile device 205 can activate the first sensor 260A and the second sensor 260B simultaneously or within the same time window, allowing mobile device 205 to collect more sensor data and gain a more complete and accurate understanding of the environment. For example, if the first sensor 260A is a RADAR sensor and the second sensor 260B is a LIDAR sensor, mobile device 205 can activate both sensors 260A-260B and can use sensor measurement data from both sensors 260A-260B and / or from cameras 220A-220B to obtain detailed and accurate depth information of the scene depicted in the displayed image 350. Different sensors among sensors 260A-260B can be activated and disabled at different times, for example, based on which software application is running on mobile device 205. One software application can trigger mobile device 205 to activate the first sensor 260A but disable the second sensor 260B. Another software application can trigger mobile device 205 to activate the second sensor 260B but disable the first sensor 260A. Another software application can trigger mobile device 205 to activate both the first sensor 260A and the second sensor 260B. Another software application can trigger mobile device 205 to disable both the first sensor 260A and the second sensor 260B. In some cases, mobile device 205 can detect occlusion of one of sensors 260A-260B, where mobile device 205 can disable the occluded sensor and activate the other (unoccluded) sensor. For example, mobile device 205 can detect occlusion of one of sensors 260A-260B when a sensor returns a sensor measurement indicating that an object has been detected at a distance less than a threshold distance from the sensor. In some cases, the sensors in sensors 260A-260B may be within a threshold distance of the cameras in cameras 220A-220B. If mobile device 205 detects such camera occlusion (e.g., as...), Figure 6If the obstruction is in 630A-C, then the mobile device 205 can assume that the sensor is within a threshold distance of the camera. For example, in Figure 7 In this configuration, sensors 260A-260B are within a threshold distance of the first camera 220A. If the mobile device 205 detects occlusion of the first camera 220A, the mobile device 205 can assume that sensors 260A-260B are also occluded. This type of occlusion detection can be useful when one or more of sensors 260A-260B are sensor types that may be difficult to detect occlusion of (e.g., microphones).

[0129] Figure 8 This is a flowchart illustrating image processing technology 800. Figure 8 The flowchart illustrates that the image processing technology 800 can be performed by a device. This device can be a mobile device 205, an image capture and processing system 100, an image capture device 105A, an image processing device 105B, one or more web servers for cloud services, a computing system 1000, or some combination thereof.

[0130] In operation 805, the device receives an image captured by camera 220. Camera 220 captures the image based on incident light 245 received by camera 220. Camera 220 is positioned relative to display 210 to receive incident light 245 passing through a portion of display 210 before it reaches camera 220. This portion of display 210 may be, for example, a light-passing area of ​​display 210, such as in light-passing area 285. The light reaching the camera may include an image sensor 130 impacting camera 220. The camera may be along an axis perpendicular to the planar surface of the display (e.g., in...). Figure 2A In the context of mobile device 205 Figure 2A The Z-axis of Figure 290 is positioned near the display.

[0131] Display 210 may be a touchscreen display. The device can use the touch-sensitive surface of the touchscreen display to detect touch input or touch gesture input received at the touch-sensitive surface of the touchscreen display. Display 210 may include one or more sensors 260. The one or more sensors 260 may include one or more active depth sensors. The device can use one or more active depth sensors to detect hover input, hover pointing input, or hover gesture input above display 210.

[0132] In operation 810, the device displays an image-based display image 350 on display 210. In some cases, the device generates the display image 350 based on an image captured by camera 220 before displaying the display image 350 on display 220. Generating the display image 350 may include modifying the size of the image using at least one of scaling and / or cropping. Scaling may include enlarging, shrinking, and / or stretching. Modifying the size of the image may include modifying the size of the image so that the display image 350 is displayed on display 210 but not on the indicator area. Generating the display image 350 may also include processing the image, for example by modifying at least one of brightness, contrast, color saturation, hue, white balance, black balance, level, gamma correction, gain, color correction, noise filter, sharpness, blur, and red-eye correction.

[0133] In operation 815, while display 210 is displaying display image 350, the device displays an indicator overlaid on display image 350 at an indicator area of ​​display 210. The indicator area includes at least a subset of the display. Examples of indicators in operation 815 include indicators 310A, 310B, 310C, indicator arrow 340, indicator 410, and indicator 510 and / or combinations thereof. The indicator area of ​​the display may be at least as large as the lens of a camera.

[0134] In some cases, the device may automatically display an indicator overlaid on the display image 350 at the indicator area. In other cases, the device receives input, and in response to the receipt of input, the device displays an indicator overlaid on the display image 350 at the indicator area. The input may include at least one of the following: touching the touch-sensitive touchscreen surface of the display 210, a predefined gesture across the touch-sensitive touchscreen surface of the display 210, hovering over the display 210, a predefined gesture above the display 210, voice input uttering a predefined phrase recorded by a microphone, or a combination thereof.

[0135] The indicator may include a shape surrounding at least a portion of the display 210. The display indicator may include multiple pixels in the indicator area displaying a color. The color may be black, white, gray, red, green, blue, yellow, or some mixture of these colors. The display indicator may include multiple pixels in the indicator area displaying a pattern or secondary image that is different from the image and different from the displayed image 350. The indicator may be animated. For example, the indicator may blink or pulse.

[0136] Displaying an indicator overlaid on the display image 350 in the indicator area while the display 210 is displaying the display image 350 may include modifying the display image 360 ​​to merge the indicator with the display image 350.

[0137] The device may display an image capture user interface (UI) element 320 on the image capture UI area of ​​the display 210. The device may receive input at the image capture UI area. The input may be touch input, hover input, gesture input, or one of any other input types discussed herein. In response to receiving input at the image capture UI area, the device may receive a second image captured by the camera 220. The device may store the second image to a non-transitory computer-readable storage medium.

[0138] The device may receive input at a region selection user interface (UI) area. The input may be touch input, hover input, gesture input, or at least one of any other input types discussed herein. The device may select a region of the displayed image 350 in response to receiving input at the region selection UI area. Selectable regions 540, 540A, and 540B are examples of selection regions. The region selection UI area differs from the selection region of the displayed image. The selection region of the displayed image may include at least a portion of an indicator area. In some cases, the device displays a region selection UI element 320 at the region selection UI area. In some cases, the device generates a copy of the region selection UI element 320 to depict at least a portion of the selection region.

[0139] The device can set image capture parameters to specific settings determined based on the selected area of ​​the displayed image 350 in response to the selection of a selected area. Image capture parameters may include at least one of focus or exposure parameters. Exposure parameters may include at least one of exposure time, analog gain, digital gain, aperture size, ISO, or combinations thereof. The device can set image processing attributes to specific settings determined based on the selected area of ​​the displayed image in response to the selection of a selected area. Image processing attributes may include at least one of brightness, contrast, color saturation, hue, white balance, black balance, black level compensation, level, gamma correction, gain, color correction, noise filter, sharpness, blur, red-eye correction, or combinations thereof.

[0140] After capturing an image, the device can receive a second image captured by camera 220. The device can determine that the second image is occluded by an occluder (e.g., occluder 630A-C). The device can display display image 350 on display 210 in response to determining that the second image is occluded by an occluder. For example, because the second image is occluded by an occluder, the device can choose to display display image 350 again on display 210 instead of displaying the second image or a second display image based on the second image. In some cases, the device can receive input, where occlusion is associated with the reception of input. The input can be touch input, hover input, gesture input, or at least one of any other input types discussed herein. The device can select a selection area for displaying the image based on the input. The device can determine a setting based on the selection area for displaying the image and can apply that setting. The device can receive a third image captured by camera 220 while the setting is being applied and after capturing the second image.

[0141] In some cases, the device may receive signals from multiple cameras. In some examples, the device may receive a second image captured by a second camera. The second camera may capture the second image based on secondary incident light received by the second camera. The second camera may be positioned relative to display 210 to receive secondary incident light that passes through a second portion of display 210 (e.g., a second light passage area of ​​display 210) before reaching the second camera. Reaching the second camera may include a second image sensor that impacts the second camera. In some examples, the device may determine that the second image is occluded by an occlusion and may display a display image 350 on display 210 in response to determining that the second image is occluded by an occlusion. In some examples, the device may determine depth information corresponding to one or more objects depicted in the image and the second image by processing the image and the second image (e.g., using stereoscopic image processing).

[0142] Figure 9 This is a flowchart illustrating image processing technology 900. Figure 9 The image processing technology 900 illustrated in the flowchart can be performed by a device. This device can be a mobile device 205, an image capture and processing system 100, an image capture device 105A, an image processing device 105B, one or more web servers for cloud services, a computing system 1000, or some combination thereof.

[0143] In operation 905, the device receives an image captured by camera 220. Camera 220 captures the image based on incident light 245 received by camera 220. Camera 220 is positioned relative to touchscreen display 210 to receive incident light 245 passing through light passage area 285 of touchscreen display 210 before reaching camera 220. The light passing area 285 of touchscreen display 210 in operation 905 can be an example of a portion of the display in operation 805.

[0144] During operation 910, the device displays an image-based display image 350 on the touchscreen display 210.

[0145] In operation 915, the device displays region selection user interface (UI) elements on the touchscreen display 210. The region selection UI elements in operation 915 may include, for example... Figure 5 520 UI elements for area selection Figure 7 Select UI elements in the region 720A or Figure 7 The area selection UI element 720B.

[0146] In operation 920, the device receives touch input to the touchscreen display 210 at the region selection input area that overlaps with the region selection UI element. In some cases, touch input can be supplemented or replaced by hover input.

[0147] In operation 925, the device selects a selection area for displaying image 350 in response to receiving touch input. The selection area may be a selectable region that includes at least a portion of the light-passing area, for example... Figure 5 Selectable area 540 Figure 7 Selectable area 540A or Figure 7 Selectable area 540B.

[0148] In operation 930, the device, in response to the selection of a selected area, sets the camera attributes of camera 220 to settings determined based on the selected area of ​​the displayed image 350. The camera attributes of camera 220 include at least one of image capture attributes or image processing attributes. Image capture attributes may include at least one of focus and exposure parameters. Exposure parameters may include at least one of exposure time, analog gain, digital gain, aperture size, ISO, or combinations thereof. Image processing attributes may include at least one of brightness, contrast, color saturation, hue, white balance, black balance, black level compensation, level, gamma correction, gain, color correction, noise filter, sharpness, blur, red-eye correction, or combinations thereof.

[0149] In some cases, Figure 8-9At least a subset of the image processing techniques 800 and 900 shown in the flowchart can be remotely executed by one or more web servers of a cloud service. In some examples, the processes described herein (e.g., image processing techniques 800 and 900 and / or other processes described herein) can be executed by a computing device or apparatus. In one example, image processing techniques 800 and 900 can be executed by... Figure 1 The image capture device 105A performs this. In another example, image processing techniques 800 and 900 can be performed by... Figure 1 The image processing device 105B performs the processing. Image processing techniques 800 and 900 can also be performed by... Figure 1 The image capture and processing system 100 performs the operation. Image processing techniques 800 and 900 can also be performed by... Figure 2A , Figure 2B , Figure 2C , Figure 3A , Figure 3B , Figure 3C , Figure 4 , Figure 5 , Figure 6 and / or Figure 7 The image processing technologies 800 and 900 can be executed by any of the mobile devices 205. Figure 10 The computing device executing the architecture of the computing system 1000 shown herein. The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a connected watch or smartwatch, or other wearable device), a server computer, a computing device of an autonomous vehicle or autonomous vehicle, a robotic device, a television, and / or any other computing device with resource capabilities for performing the processes described herein, including image processing techniques 800 and 900. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive Internet Protocol (IP) based data or other types of data.

[0150] Components of a computing device can be implemented in circuitry. For example, components may include electronic circuitry or other electronic hardware and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0151] Depend on Figure 8-9 The processes illustrated by conceptual diagrams and flowcharts are organized into logical flowcharts, where operations represent a series of operations that can be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which operations are described is not intended to be construed as restrictive, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0152] In addition, by Figure 8-9 The conceptual diagrams and flowcharts illustrating the processes and / or other processes described herein can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors via hardware or a combination thereof. As described above, the code can be stored, for example, on a computer-readable or machine-readable storage medium in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0153] Although Figure 2A-2C , Figures 3A-3C , Figure 4 , Figure 5 , Figure 6 and Figure 7 Mobile device 205 has been described, but it should be understood that mobile device 205 is merely one example of a device that includes the technologies and / or performs the technologies discussed herein. The technologies and processes discussed herein (including image processing technologies 800 and 900) can be performed by mobile device 205, image capture and processing system 100, image capture device 105A, image processing device 105B, one or more web servers for cloud services, computing system 1000, or some combination thereof.

[0154] Figure 10 These are diagrams illustrating examples of systems used to implement certain aspects of this technology. In particular, Figure 10 An example of a computing system 1000 is illustrated, which can be any computing device constituting, for example, an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 1005. Connection 1005 can be a physical connection using a bus, or a direct connection to processor 1010, such as in a chipset architecture. Connection 1005 can also be a virtual connection, a network connection, or a logical connection.

[0155] In some embodiments, the computing system 1000 is a distributed system, wherein the functions described herein may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more described system components represent a plurality of such components, each performing some or all of the functions described for that component. In some embodiments, a component may be a physical or virtual device.

[0156] Example system 1000 includes at least one processing unit (CPU or processor) 1010 and a connection 1005 that couples various system components to the processor 1010. These system components include system memory 1015, such as read-only memory (ROM) 1020 and random access memory (RAM) 1025. The computing system 1000 may include a cache 1012 of high-speed memory, which is directly connected to, adjacent to, or integrated into the processor 1010.

[0157] Processor 1010 may include any general-purpose processor and hardware or software services, such as services 1032, 1034, and 1036 stored in storage device 1030, which are configured to control processor 1010 and dedicated processors in which software instructions are incorporated into the actual processor design. Processor 1010 may essentially be a fully self-contained computing system, containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0158] To enable user interaction, the computing system 1000 includes an input device 1045, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. The computing system 1000 may also include an output device 1035, which can be one or more of a variety of output mechanisms. In some cases, a multimodal system allows the user to provide multiple types of input / output to communicate with the computing system 1000. The computing system 1000 may include a communication interface 1040, which typically controls and manages user input and system output. The communication interface can use wired and / or wireless transceivers to perform or facilitate the reception and / or transmission of wired or wireless communications, including using audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, etc. Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs Wireless signal transmission Low-power (BLE) wireless signal transmission Wired and / or wireless transceivers for wireless signal transmission, including radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), global microwave access interoperability (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1040 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers for determining the location of the computing system 1000 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There are no limitations on operation on any particular hardware configuration; therefore, as improved hardware or firmware configurations are developed, the basic features described here can be easily replaced with those improved configurations.

[0159] Storage device 1030 may be a non-volatile and / or non-transitory and / or computer-readable storage device and may be a hard disk or other type of computer-readable medium that can store computer-accessible data, such as magnetic tape cassettes, flash memory cards, solid-state storage devices, digital versatile disks, magnetic tape cassettes, floppy disks, flexible disks, hard disks, magnetic tapes, magnetic stripes / sheets, any other magnetic storage media, flash memory, memristor memory, any other solid-state storage, compact disc read-only memory (CD-ROM), rewritable compact disc (CD), digital video disc (DVD), Blu-ray disc (BDD), holographic disc, another optical medium, secure digital card (SD card), micro secure digital card (microSD card), memory Cards, smart card chips, EMV chips, Subscriber Identity Module (SIM) cards, mini / micro / nano / pico SIM cards, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase-change memory (PCM), spin-transfer torque RAM (STT-RAM), another memory chip or cassette tape, and / or combinations thereof.

[0160] Storage device 1030 may include software services, servers, etc., which enable the system to perform functions when processor 1010 executes code defining such software. In some embodiments, hardware services that perform specific functions may include software components stored in a computer-readable medium that, together with necessary hardware components (such as processor 1010, connection 1005, output device 1035, etc.), perform the functions.

[0161] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include, but does not include, non-transitory media on which data can be stored, carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as CDs or DVDs, flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted using any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0162] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0163] Specific details are provided in the foregoing description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that these embodiments can be practiced without these specific details. For clarity of explanation, in some cases, the technology may be presented as comprising individual functional blocks, including functional blocks containing devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0164] The various embodiments described above can be presented as processes or methods, which may be depicted as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. Although flowcharts can describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but there may be other steps not included in the diagram. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.

[0165] The processes and methods described in the examples above can be implemented using computer-executable instructions stored or otherwise obtained from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a specific function or group of functions. Part of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include hard disks or optical disks, flash memory, USB devices with non-volatile memory, network storage devices, etc.

[0166] Devices implementing the processes and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) that perform the necessary tasks may be stored on a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small personal computers, personal digital assistants, rack-mount devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. As a further example, such functionality may also be implemented on circuit boards between different chips or between different processes executing in a single device.

[0167] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means of providing the functionality described in this disclosure.

[0168] In the foregoing description, various aspects of this application have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this application is not limited thereto. Therefore, while illustrative embodiments of this application have been described in detail herein, it should be understood that the concepts of the invention can be embodied and employed in other ways, and the appended claims are intended to be construed as including such variations unless limited by prior art. Various features and aspects of the above applications can be used individually or in combination. Furthermore, without departing from the broader spirit and scope of this specification, the embodiments can be used in any number of environments and applications beyond those described herein. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. For illustrative purposes, methods are described in a particular order. It should be appreciated that, in alternative embodiments, these methods may be performed in a different order than that described.

[0169] Those skilled in the art will understand that the less than (“<”) and greater than (“>”) symbols or terms used herein may be replaced by the less than or equal to (“≤”) and greater than or equal to (“≥”) symbols, respectively, without departing from the scope of this specification.

[0170] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0171] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0172] The use of the language “at least one of” and / or “one or more of” in a set of claims, or other languages, indicates that one or more members of the set (any combination thereof) satisfy the claim. For example, the language of the claim “at least one of A and B” indicates A, B, or A and B. In another example, the language of the claim “at least one of A, B, and C” indicates A, B, C, or A and B, or A and C, or B and C, or A and B and C. The use of “at least one of” and / or “one or more of” in a set of languages ​​does not limit the set to the items listed in the set. For example, the language of the claim “at least one of A and B” can indicate A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0173] The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0174] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or multi-purpose integrated circuit devices, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as a discrete but interoperable logic device. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium including program code comprising instructions to perform one or more of the methods described above when executed. The computer-readable data storage medium can form part of a computer program product and may include packaging materials. The computer-readable medium can include memory or data storage media, such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. These technologies may also be implemented, at least in part, through a computer-readable communication medium that carries or transmits program code in the form of instructions or data structures that can be accessed, read, and / or executed by a computer, such as propagating signals or waves.

[0175] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0176] The illustrative aspects of this disclosure include:

[0177] Aspect 1: An apparatus for image processing, the apparatus comprising: a memory; and one or more processors coupled to the memory and configured to: receive an image captured by a camera based on incident light received by a camera, wherein the camera is positioned relative to a display to receive incident light passing through a portion of the display before reaching the camera; display a display image based on the image on the display; and while the display is displaying the display image, display an indicator overlaid on the display image at an indicator area of ​​the display, wherein the indicator area comprises at least a subset of the portion of the display.

[0178] Aspect 2: The apparatus of aspect 1, wherein the apparatus is at least one of a mobile device, a wireless communication device, and a camera device.

[0179] Aspect 3: A device of any one of aspects 1 to 2, wherein the device includes at least one of the camera and the display.

[0180] Aspect 4: The apparatus of any one of Aspects 1 to 3, wherein at least one of the indicator areas of the indicator and the display is at least as large as the area of ​​the lens of the camera.

[0181] Aspect 5: A device of any of Aspects 1 to 4, wherein one or more processors are further configured to: receive input, wherein the indicator is displayed on the indicator area in response to the receipt of the input.

[0182] Aspect 6: The apparatus of aspect 5, wherein the input includes at least one of the following: touching the touch-sensitive touchscreen surface of the display, a predefined gesture across the touch-sensitive touchscreen surface of the display, hovering over the display, a predefined gesture above the display, or voice input uttering a predefined phrase recorded by a microphone.

[0183] Aspect 7: An apparatus of any one of Aspects 1 to 6, wherein the one or more processors are further configured to generate the display image by modifying the size of the image using at least one of scaling and cropping.

[0184] Aspect 8: The apparatus of aspect 7, wherein modifying the size of the image includes modifying the size of the image such that the display image is displayed on the display outside the indicator area.

[0185] Aspect 9: A device of any of aspects 1 to 8, wherein the indicator includes the shape of at least a portion surrounding the portion of the display.

[0186] Aspect 10: A device for any of aspects 1 to 9, wherein the indicator is animated.

[0187] Aspect 11: The apparatus of any one of Aspects 1 to 10, wherein displaying the indicator overlaid on the display image at the indicator area while the display is displaying the display image includes modifying the display image to merge the indicator with the display image.

[0188] Aspect 12: A device of any of aspects 1 to 11, wherein the display is a touch screen.

[0189] Aspect 13: An apparatus of any of Aspects 1 to 12, wherein one or more processors are further configured to: display an image capture UI element above an image capture user interface (UI) area on the display; receive input at the image capture UI area, wherein the input is one of touch input, hover input, and gesture input; receive a second image captured by the camera in response to receiving the input at the image capture UI area; and store the second image to a non-transitory computer-readable storage medium.

[0190] Aspect 14: An apparatus of any of Aspects 1 to 13, wherein the one or more processors are further configured to: receive input at a region selection user interface (UI) region, wherein the input is one of touch input, hover input, and gesture input; and select a selection region of the display image in response to receiving the input at the region selection UI region, wherein the region selection UI region is different from the selection region of the display image.

[0191] Aspect 15: The apparatus of aspect 14, wherein the selection area of ​​the displayed image includes at least a portion of the indicator area.

[0192] Aspect 16: An apparatus of any of Aspects 14 or 15, wherein one or more processors are further configured to: in response to the selection of the selected area, set image capture parameters associated with the camera to specific settings determined based on the selected area of ​​the displayed image, wherein the image capture parameters include at least one of focus and exposure parameters.

[0193] Aspect 17: The apparatus of any of Aspects 14 to 16, wherein one or more processors are further configured to: in response to the selection of the selected region, set an image processing attribute to a specific setting determined based on the selected region of the displayed image, wherein the image processing attribute is one of brightness, contrast, color saturation, hue, white balance, black balance, black level compensation, level, gamma correction, gain, color correction, noise filter, sharpness, blur, and red-eye correction.

[0194] Aspect 18: The apparatus of any of aspects 14 to 17, wherein the one or more processors are further configured to: display a region selection UI element at a region selection UI region.

[0195] Aspect 19: The apparatus of aspect 18, wherein one or more processors are further configured to: generate a copy of a region selection UI element to depict at least a portion of the selected region.

[0196] Aspect 20: The apparatus of any one of Aspects 1 to 19, wherein the one or more processors are further configured to: receive a second image captured by the camera after capturing the image; determine that the second image is occluded by an occluder; and display the display image on the display in response to determining that the second image is occluded by the occluder.

[0197] Aspect 21: The apparatus of aspect 20, wherein one or more processors are further configured to: receive input, wherein the occlusion is associated with the reception of the input, wherein the input is one of touch input, hover input, and gesture input; select a selection area of ​​the displayed image based on the input; determine a setting based on the selection area of ​​the displayed image; apply the setting; and receive a third image captured by the camera while the setting is applied and after the second image is captured.

[0198] Aspect 22: The apparatus of any one of Aspects 1 to 21, wherein the one or more processors are further configured to: receive a second image captured by the second camera based on secondary incident light received by the second camera, wherein the second camera is positioned relative to the display to receive the secondary incident light passing through a second portion of the display before reaching the second camera.

[0199] Aspect 23: The apparatus of aspect 22, wherein one or more processors are further configured to: determine that the second image is occluded by an occluder, wherein the display image is displayed in response to determining that the second image is occluded by the occluder.

[0200] Aspect 24: An apparatus of any one of Aspects 22 or 23, wherein the one or more processors are further configured to generate the display image based on the image and the second image.

[0201] Aspect 25: An apparatus of any of Aspects 1 to 24, wherein one or more processors are further configured to: determine depth information corresponding to one or more objects depicted in the image and the second image by processing the image and the second image.

[0202] Aspect 26: A processing method comprising: receiving an image captured by a camera based on incident light received by the camera, wherein the camera is positioned relative to a display to receive incident light passing through a portion of the display before reaching the camera; displaying a display image based on the image on the display; and displaying an indicator overlaid on the display image at an indicator area of ​​the display while the display is displaying the display image, wherein the indicator area comprises at least a subset of a portion of the display.

[0203] Aspect 27: The method of aspect 26, wherein the method is performed by a mobile device including the camera and the display.

[0204] Aspect 28: The method of any one of Aspects 26 to 27, wherein at least one of the indicator regions of the indicator and the display is at least as large as the area of ​​the lens of the camera.

[0205] Aspect 29: The method of any one of aspects 26 to 28 further includes: receiving input, wherein the indicator is displayed as an overlay on the indicator area in response to the receipt of the input.

[0206] Aspect 30: A method of aspect 29, wherein the input includes at least one of the following: touching a touch-sensitive touchscreen surface of the display, a predefined gesture across the touch-sensitive touchscreen surface of the display, hovering over the display, a predefined gesture over the display, or voice input uttering a predefined phrase recorded by a microphone.

[0207] Aspect 31: The method of any one of Aspects 26 to 30 further includes: generating the display image by modifying the size of the image using at least one of scaling and cropping, wherein modifying the size of the image includes modifying the size of the image such that the display image is displayed on the display outside the indicator area.

[0208] Aspect 32: The method of any one of Aspects 26 to 31, wherein the indicator includes a shape surrounding at least a portion of the portion of the display.

[0209] Aspect 33: The method of any one of Aspects 26 to 32, wherein displaying the indicator overlaid on the display image at the indicator area while the display is displaying the display image includes modifying the display image to merge the indicator with the display image.

[0210] Aspect 34: The method of any one of Aspects 26 to 33, wherein the display is a touch screen.

[0211] Aspect 35: The method of any one of Aspects 26 to 34 further includes: displaying an image capture UI element above the image capture user interface (UI) area of ​​the displayed image; receiving input at the image capture UI area, wherein the input is one of touch input, hover input, and gesture input; receiving a second image captured by the camera in response to receiving the input at the image capture UI area; and storing the second image to a non-transitory computer-readable storage medium.

[0212] Aspect 36: The method of any one of Aspects 26 to 35 further includes: receiving input at a region selection user interface (UI) region, wherein the input is one of touch input, hover input, and gesture input; and selecting a selection region of the display image in response to receiving the input at the region selection UI region, wherein the region selection UI region is different from the selection region of the display image.

[0213] Aspect 37: The method of aspect 36, wherein the selection area of ​​the displayed image includes at least a portion of the indicator area.

[0214] Aspect 38: The method of aspect 37 further includes: in response to the selection of the selected region, setting image capture parameters to specific settings determined based on the selected region of the displayed image, wherein the image capture parameters include at least one of focus and exposure parameters.

[0215] Aspect 39: The method of any one of Aspects 37 or 38 further includes: in response to the selection of the selected region, setting an image processing attribute to a specific setting determined based on the selected region of the displayed image, wherein the image processing attribute is one of white balance, black level compensation, gamma correction, gain, color correction, color saturation, noise filter, contrast control, brightness control, tone mapping, sharpness, blur and red-eye correction.

[0216] Aspect 40: The method of any one of Aspects 26 to 39 further includes: receiving a second image captured by the camera after capturing the image; determining that the second image is occluded by an occluder; and displaying the display image on the display in response to determining that the second image is occluded by the occluder.

[0217] Aspect 41: The method of aspect 40 further includes: receiving input, wherein the occlusion is associated with receiving the input, wherein the input is one of touch input, hover input, and gesture input; selecting a selection area of ​​the displayed image based on the input; determining a setting based on the selection area of ​​the displayed image; applying the setting; and receiving a third image captured by the camera while applying the setting and after capturing the second image.

[0218] Aspect 42: The method of any one of Aspects 26 to 41 further includes: receiving a second image captured by the second camera based on secondary incident light received by the second camera, wherein the second camera is positioned relative to the display to receive the secondary incident light passing through a second portion of the display before reaching the second camera.

[0219] Aspect 43: The method of any one of Aspects 26 to 42 further includes: determining that the second image is occluded by an occluder, wherein the display image is displayed in response to determining that the second image is occluded by the occluder.

[0220] Aspect 44: The method of any one of aspects 26 to 43 further includes: determining depth information corresponding to one or more objects depicted in the image and the second image by processing the image and the second image.

[0221] Aspect 45: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform an operation according to any one of aspects 1 to 44.

[0222] Aspect 46: An apparatus comprising one or more units for performing operations according to any one of aspects 1 to 44.

Claims

1. An apparatus for image processing, the apparatus comprising: Memory; as well as One or more processors coupled to the memory and configured to: Based on incident light received by the camera from the scene, an image of the scene captured by the camera is received, wherein the camera is positioned relative to the display to receive the incident light from the scene that passes through a portion of the display before reaching the camera; Displaying a display image based on the image on the display; and Generate an indicator that indicates at least a subset of the portion of the display through which the incident light from the scene passes, the indicator being configured to be hidden by default when no user-triggered input is received; Receive user-triggered input; In response to receiving the user-triggered input, while the display is showing the display image, an indicator is displayed in the indicator area of ​​the display, overlaying the display image, to indicate to the user the position of the camera below the display.

2. The apparatus of claim 1, wherein, The device is at least one of the following: a mobile device, a wireless communication device, or a camera device.

3. The apparatus of claim 1, wherein, The device includes at least one of the following: the camera or the display.

4. The apparatus of claim 1, wherein, At least one of the indicator areas of the indicator or the display is at least as large as the area of ​​the camera lens.

5. The apparatus of claim 1, wherein, The input includes at least one of the following: touching the touch-sensitive surface of the display, a predefined gesture across the touch-sensitive surface of the display, hovering over the display, a predefined gesture above the display, or voice input uttering a predefined phrase recorded by a microphone.

6. The apparatus of claim 1, wherein, The one or more processors are further configured to: The display image is generated by modifying the size of the image using at least one of scaling or cropping before it is displayed.

7. The apparatus of claim 6, wherein, Modifying the size of the image includes resizing the image so that the display image is shown on the display outside the indicator area.

8. The apparatus of claim 1, wherein, The indicator includes the shape of at least a subset of the portion of the display through which the incident light passes.

9. The apparatus of claim 1, wherein, The indicator is animated.

10. The apparatus of claim 1, wherein, Displaying an indicator overlaid on the display image at the indicator area while the display is displaying the display image includes: modifying the display image to merge the indicator with the display image.

11. The apparatus of claim 1, wherein, The display is a touchscreen.

12. The apparatus of claim 1, wherein, The one or more processors are further configured to: Image capture UI elements are displayed on the image capture user interface (UI) area of ​​the display. Input is received at the image capture UI area, wherein the input is one of the following: touch input, hover input, or gesture input; In response to receiving the input at the image capture UI area, a second image captured by the camera is received; and The second image is stored in a non-transitory computer-readable storage medium.

13. The apparatus of claim 1, wherein, The one or more processors are further configured to: Input is received at the region selection user interface (UI) area, wherein the input is one of the following: touch input, hover input, or gesture input; and In response to receiving the input at the region selection UI region, a selection region of the display image is selected, wherein the region selection UI region is different from the selection region of the display image.

14. The apparatus of claim 13, wherein, The selection area for the displayed image includes at least a portion of the indicator area.

15. The apparatus of claim 13, wherein, The one or more processors are further configured to: In response to the selection of the selected area, the image capture parameters associated with the camera are set to specific settings determined based on the selected area of ​​the displayed image, wherein the image capture parameters include at least one of the following: focus or exposure parameters.

16. The apparatus of claim 13, wherein, The one or more processors are further configured to: In response to the selection of the selected area, image processing attributes are set to specific settings determined based on the selected area of ​​the displayed image, wherein the image processing attributes are one of the following: brightness, contrast, color saturation, hue, white balance, black balance, black level compensation, level, gamma correction, gain, color correction, noise filter, sharpness, blur or red-eye correction.

17. The apparatus of claim 13, wherein, The one or more processors are further configured to: Display the region selection UI element at the selected region.

18. The apparatus of claim 17, wherein, The one or more processors are further configured to: Generate a UI element for the selected region to depict a copy of at least a portion of the selected region.

19. The apparatus of claim 1, wherein, The one or more processors are further configured to: After capturing the first image, a second image captured by the camera is received; It is determined that the second image is obscured by an occluder; and In response to determining that the second image is obscured by the obstruction, the display image is displayed on the display.

20. The apparatus of claim 19, wherein, The one or more processors are further configured to: Receiving input, wherein the obstruction is associated with receiving the input, wherein the input is one of the following: touch input, hover input, or gesture input; Select a region of the displayed image based on the input; The settings are determined based on the selected area of ​​the displayed image; Apply the settings described above; and While the settings are applied, a third image captured by the camera is received after the second image is captured.

21. The apparatus of claim 1, wherein, The one or more processors are further configured to: A second image captured by the second camera is received based on secondary incident light received by the second camera, wherein the second camera is positioned relative to the display to receive the secondary incident light that passes through a second portion of the display before reaching the second camera.

22. The apparatus of claim 21, wherein, The one or more processors are further configured to: It is determined that the second image is obscured by an occluder, wherein the display image is displayed in response to determining that the second image is obscured by the occluder.

23. The apparatus of claim 21, wherein, The one or more processors are further configured to: The display image is generated based on the image and the second image.

24. The apparatus of claim 21, wherein, The one or more processors are further configured to: Depth information corresponding to one or more objects depicted in the image and the second image is determined by processing the image and the second image.

25. A method for image processing, the method comprising: An image of the scene captured by the camera is received based on incident light received by the camera from the scene, wherein the camera is positioned relative to the display to receive the incident light from the scene that passes through a portion of the display before reaching the camera; Displaying a display image based on the image on the display; and Generate an indicator that indicates at least a subset of the portion of the display through which the incident light from the scene passes, the indicator being configured to be hidden by default when no user-triggered input is received; Receive user-triggered input; In response to receiving the user-triggered input, while the display is showing the display image, an indicator is displayed in the indicator area of ​​the display, overlaying the display image, to indicate to the user the position of the camera below the display.

26. The method of claim 25, wherein, The method is performed by a mobile device including the camera and the display.

27. The method of claim 25, wherein, At least one of the indicator or the indicator area of ​​the display is at least as large as the area of ​​the camera lens.

28. The method of claim 25, wherein, The input includes at least one of the following: touching the touch-sensitive touchscreen surface of the display, a predefined gesture across the touch-sensitive touchscreen surface of the display, hovering over the display, a predefined gesture over the display, or voice input uttering a predefined phrase recorded by a microphone.

29. The method of claim 25, further comprising: The display image is generated by modifying the size of the image using at least one of scaling or cropping before it is displayed.

30. The method of claim 25, wherein, The indicator includes the shape of at least a subset of the portion of the display through which the incident light passes.

31. The method of claim 25, wherein, Displaying an indicator overlaid on the display image at the indicator area while the display is displaying the display image includes modifying the display image to merge the indicator with the display image.

32. The method of claim 25, wherein, The display is a touchscreen.

33. The method of claim 25, further comprising: Display image capture UI elements on the image capture user interface (UI) area where the image is displayed; Input is received at the image capture UI area, wherein the input is one of the following: touch input, hover input, or gesture input; In response to receiving the input at the image capture UI area, a second image captured by the camera is received; and The second image is stored in a non-transitory computer-readable storage medium.

34. The method of claim 25, further comprising: Input is received at the region selection user interface (UI) area, wherein the input is one of the following: touch input, hover input, or gesture input; and In response to receiving the input at the region selection UI region, a selection region of the display image is selected, wherein the region selection UI region is different from the selection region of the display image.

35. The method of claim 34, wherein, The selection area for the displayed image includes at least a portion of the indicator area.

36. The method of claim 35, further comprising: In response to the selection of the selected area, the image capture parameters are set to specific settings determined based on the selected area of ​​the displayed image, wherein the image capture parameters include at least one of the following: focus or exposure parameters.

37. The method of claim 35, further comprising: In response to the selection of the selected area, image processing attributes are set to specific settings determined based on the selected area of ​​the displayed image, wherein the image processing attributes are one of the following: white balance, black level compensation, gamma correction, gain, color correction, color saturation, noise filter, contrast control, brightness control, tone mapping, sharpness, blur or red-eye correction.

38. The method of claim 25, further comprising: After capturing the first image, a second image captured by the camera is received; It is determined that the second image is obstructed by an occluder; as well as In response to determining that the second image is obscured by the obstruction, the display image is displayed on the display.

39. The method of claim 38, further comprising: Receiving input, wherein the obstruction is associated with receiving the input, wherein the input is one of the following: touch input, hover input, or gesture input; Select a region of the displayed image based on the input; The settings are determined based on the selected area of ​​the displayed image; Apply the settings described above; and While the settings are applied, a third image captured by the camera is received after the second image is captured.

40. The method of claim 25, further comprising: A second image captured by a second camera is received based on secondary incident light received by the second camera, wherein the second camera is positioned relative to the display to receive the secondary incident light that passes through a second portion of the display before reaching the second camera.

41. The method of claim 40, further comprising: It is determined that the second image is obscured by an occluder, wherein the display image is displayed in response to determining that the second image is obscured by the occluder.

42. The method of claim 40, further comprising: Depth information corresponding to one or more objects depicted in the image and the second image is determined by processing the image and the second image.

43. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 25-42.

Images