Mitigating display interference on captured images using calibration images

Abstract

This invention relates to mitigating display interference with captured images using calibration images. A device includes a digital camera positioned below a display of the device that captures images through the display. When capturing images using the digital camera, light emitted by the display can enter the digital camera, thereby interfering with or corrupting the captured images. An enhanced output image is generated from the captured image that mitigates interference from the display. For example, the enhanced output image is generated by capturing different calibration images for each pixel type (e.g., red, green, and blue) included in the display. When subsequently capturing an image, the amount of color emitted by the pixels of the current display image on the display is determined, and one or more subtraction images are generated and subtracted from the captured image to mitigate interference from the display.

Description

Technical Field

[0001] This invention relates to using calibration images to mitigate display interference with captured images. More specifically, it relates to corresponding apparatus, methods, and client devices. Background Technology

[0002] Modern devices offer arrays of image capture technologies, such as those implemented through cameras used to capture digital images, digital video, and so on. For example, mobile devices such as mobile phones and tablets typically include camera functionality for capturing images from various angles, including front and rear cameras. However, efforts to maximize the screen area of ​​the device's display have presented challenges for the placement of cameras used for front-facing cameras. Summary of the Invention

[0003] The device according to the invention includes: an illumination display; a digital camera disposed below the illumination display, wherein the field of view of the digital camera passes through the illumination display; a display interference mitigation system coupled to the digital camera to generate an enhanced output image from an input image captured by the digital camera to mitigate interference from the illumination display; and a storage device for storing the enhanced output image.

[0004] The method according to the invention includes: capturing a first image by means of a digital camera mounted below an illumination display in a client device, wherein the field of view of the digital camera passes through the illumination display; generating an enhanced output image from the first image by mitigating interference from the illumination display; and storing the enhanced output image.

[0005] A client device according to the present invention includes: a processor implemented in hardware; a lighting display; a digital camera disposed below the lighting display, wherein the field of view of the digital camera passes through the lighting display; and a computer-readable storage medium storing a plurality of instructions that, in response to execution of the processor, cause the processor to perform the following actions: capturing a first image by the digital camera; generating an enhanced output image from the first image by mitigating interference from the lighting display; and storing the enhanced output image. Attached Figure Description

[0006] The implementation of a technique for mitigating display interference to a captured image using a calibration image is described with reference to the following figures. Throughout the text, the same reference numerals may be used to refer to similar features and components shown in the figures:

[0007] Figure 1 An example environment for implementing the techniques discussed in this article is shown.

[0008] Figure 2 A partial cross-sectional view of a client device according to one or more embodiments is depicted.

[0009] Figure 3 An example of an interference mitigation system is shown that implements the techniques discussed in this paper.

[0010] Figure 4 An example client device is shown.

[0011] Figure 5 An example of generating an enhanced output image is shown.

[0012] Figure 6 An example of a color channel in a region of interest is shown.

[0013] Figure 7 Another example of generating an enhanced output image is shown.

[0014] Figure 8 An example recalibration module is shown.

[0015] Figure 9 An example brightness adjustment module is shown.

[0016] Figure 10 and 11 An example client device is shown.

[0017] Figure 12 An example of a gating display is shown.

[0018] Figure 13 , 14 Figures 1 and 15 show example client devices.

[0019] Figure 16 , 17 Figures 18 and 19 illustrate example processes for implementing the techniques discussed herein, according to one or more embodiments.

[0020] Figure 20 Various components of an example device that can be used to implement the techniques described herein are shown. Detailed Implementation

[0021] This article discusses using calibrated images to mitigate display interference with captured images. Typically, the techniques discussed here are for devices where a digital camera is located below and captures images through the display. This positioning of the digital camera allows for increased display space compared to conventional devices that typically reduce the display size to accommodate the camera's position adjacent to the display. However, placing a digital camera below the display presents several implementation challenges. For example, when capturing images with a digital camera located below the display, light emitted from the display can enter the camera, interfering with or damaging the captured image. The techniques discussed here reduce or mitigate interference from light emitted from the device's display when capturing images with a camera located below the display.

[0022] One technique for mitigating light interference from a device display when capturing images using a camera located below the display is based on calibration images. Typically, a calibration image (referred to as a captured calibration image) is captured by a digital camera while the display shows a calibration image of a specific color. In one or more embodiments, different calibration images are displayed and captured for each pixel type included in the display (e.g., red, green, and blue). The sensor gain and integration time of each captured calibration image are recorded or known. When other images are subsequently captured (e.g., when a user takes a selfie or uses facial recognition for authentication), a capture parameter ratio is determined based on the sensor gain and integration time of the captured calibration images and the sensor gain and integration time of any additional captured images. For each pixel type, the amount of color emitted by the pixels on the display (e.g., within the field of view of the digital camera) for the currently displayed image is determined. In one or more embodiments, a subtractive image for each pixel type is generated by determining a multiplier and multiplying the corresponding calibration image by that multiplier, which is the product of the capture parameter ratio of the pixel type and the amount of color emitted by the pixel type of the displayed image on the display. The subtracted image is subtracted from the additional captured image to obtain an enhanced output image that compensates for the light emitted by the display when the additional captured image was captured.

[0023] Alternatively, for each pixel type, a capture parameter ratio is determined and a color channel for that pixel type is generated. A subtractive image can be generated for each pixel type by multiplying the corresponding calibration image by the capture parameter ratio and then by the color channel on a per-pixel basis. Subtracting the subtractive image from the additional capture image yields an enhanced output image that compensates for the light emitted by the display during the capture of the additional image.

[0024] Using such captured calibration images to generate enhanced output images reduces interference from light emitted by the display and improves the operation of the client device by providing a more accurate scene image outside the client device. For example, user authentication can be performed more accurately using facial or fingerprint images captured by a camera through the display.

[0025] Another technique for reducing light interference from the device display when capturing images using a camera located below the display is based on a recalibrated image. The recalibrated image is captured in the same manner as the captured calibration image, but at a later time. For example, the original captured calibration image may be captured in a factory or at the user's home, and the captured recalibrated image may be captured days, months, or years after the original captured calibration image. In one or more embodiments, different recalibrated images are displayed and captured for each pixel type included in the display (e.g., red, green, and blue). The captured recalibrated image is merged with the previously captured calibration image to generate a merged calibration image, which is then used as the captured calibration image. For example, after generating the merged calibration image, the merged calibration image becomes the new captured calibration image for the client device; therefore, a subtractive image is generated based on the merged calibration image rather than the previously captured calibration image.

[0026] Recalibrated images can be captured in response to a variety of triggering events that may cause or indicate changes in the alignment and geometry of display pixels with imager pixels, such as the client device being struck or dropped, the client device's temperature exceeding a threshold, user authentication failure when using an image captured by a digital camera, etc. Various additional criteria can also be used to determine the timing for capturing a recalibrated image in response to a triggering event, such as the client device being in a low-light environment, no object being located near the front of the display, the client device being approximately face-up, the client device being stationary, the client device being charged (e.g., plugged into a charging cable), etc.

[0027] The merged calibration image can be generated in any of a variety of different ways, such as replacing the previously captured calibration image with the captured recalibrated image, or blending the corresponding pixel values ​​from the previously captured calibration image and the captured recalibrated image (e.g., averaging or using another combination ratio).

[0028] The generation of this captured recalibrated image allows the client device to adapt to changes in the alignment and geometry of the display pixels relative to the imager pixels over time or in response to various events (e.g., the client device being dropped). This enables the continued generation of enhanced output images that mitigate interference from light emitted by the display, even as the alignment and geometry of the display pixels relative to the imager pixels change.

[0029] Another technique for reducing light interference from the device's display when capturing images using a camera located below the display is to control the display's brightness while capturing images. For example, the brightness of the display can be reduced only in a portion of the camera's field of view, or the brightness of only an extended area of ​​the display can be reduced, where the extended area includes the camera area of ​​the display, which is located within the camera's field of view and the area surrounding the camera area, but is smaller than the entire display. As another example, the brightness of the entire display can be reduced, or the brightness of only specific colors of the display can be reduced (e.g., the brightness of red display pixels can be reduced, but the brightness of green and blue pixels can remain unchanged). As another example, the camera area of ​​the display within the camera's field of view can be blocked, or an extended area including the camera area and the area surrounding the camera area but smaller than the entire display can be blocked. Optionally, the display can be gated to reduce its brightness when the camera's imager is active, or to block a portion of the display without reducing its brightness while the imager is capturing images, and not to block a portion of the display when the imager is not capturing images.

[0030] In one or more embodiments, the way the display brightness is controlled varies based on ambient light level measurements from a light sensor on a client device (e.g., a dedicated ambient light sensor or an imager from a camera). For example, in low light conditions (e.g., less than 50 lux), a portion of the display (including the camera area) is shaded; in medium light conditions (e.g., between 50 and 1000 lux), the brightness of the display is reduced (e.g., the camera area or the entire display); in medium to high light conditions (e.g., between 1000 and 50,000 lux), the display is gated; and in strong light conditions (e.g., above 50,000 lux), no measures are taken to mitigate interference from light emitted from the display 104.

[0031] Controlling the display's brightness reduces interference from light emitted from the display and improves the operation of the client device by providing a more accurate image of the external scene. For example, it can more accurately perform user authentication using facial or fingerprint data captured by the camera through the display.

[0032] While the features and concepts of using calibrated images to mitigate display interference to captured images can be implemented in many different devices, systems, environments and / or configurations, implementations of using calibrated images to mitigate display interference to captured images are described in the context of the following example devices, systems and methods.

[0033] Figure 1 An example environment 100 for implementing the techniques discussed herein is shown. Environment 100 includes a client device 102, which can be or include many different types of computing or electronic devices. For example, computing device 102 can be a smartphone or other wireless phone, a camera (e.g., a compact or single-lens reflex camera), or a tablet or tablet computer. As further examples, computing device 102 can be a laptop computer (e.g., a netbook or ultrabook), a laptop computer, a wearable device (e.g., a smartwatch, a fitness tracker), a personal media player, a personal navigation device (e.g., a GPS), an entertainment device (e.g., a game console, a portable gaming device, a streaming media player, a digital video recorder, a music or other audio playback device), a camera, an Internet of Things (IoT) device, an in-vehicle computer, etc. References are provided below. Figure 20 Let's take device 2000 as an example to discuss other example properties of client device 102.

[0034] Client device 102 includes functionality for visual output, including a display (also referred to as a display screen) 104 operable via display module 106. Typically, display 104 can be implemented using a variety of different display technologies, such as light-emitting diodes (LEDs), liquid crystal displays (LCDs), organic light-emitting diodes (OLEDs), etc. Display 104 can be activated and deactivated. When deactivated, display 104 is blank and not lit, emitting little or no light. When activated, display 104 is lit and emits light to display any of various graphics, text, etc. In one or more embodiments, display 104 can be selectively activated and deactivated, allowing individual pixels or groups of pixels to be activated and deactivated. Thus, at any given time, some pixels of display 104 can be activated, while other pixels of display 104 are deactivated.

[0035] Furthermore, display module 106 represents functions for managing the functions of display 104, such as outputs provided by display 104 and inputs received by display 104, activating and deactivating display 104, etc. Display module 106 includes, for example, hardware and logic for driving the functions of display 104.

[0036] Client device 102 also includes functions for enabling image capture, including a camera module 108 that implements digital camera 110. Generally, camera 110 refers to hardware for capturing images such as still images and videos. Camera 110 includes, for example, hardware such as an optical lens or a set of lenses, a light sensor, a flash, an aperture, an imager, etc. Furthermore, camera module 108 includes hardware and / or logic for managing images captured via camera 110. Camera 110 can digitally capture images using an imager implemented in any of a variety of different technologies, such as a charge-coupled device (CCD) sensor, a complementary metal-oxide-semiconductor (CMOS) sensor, a combination thereof, etc.

[0037] Camera 110 can be activated and deactivated. When deactivated, camera 110 is in a low-power mode, consuming little or no power. When activated, camera 110 is in a higher-power mode and is capable of capturing images. Although capable of capturing images, it should be noted that the imager in camera 110 does not need to capture images constantly. For example, the imager may capture images at a rate of 30 frames per second (images) and an integration time (exposure time) of 15 milliseconds (ms), resulting in the imager alternately spending 15 ms to capture an image and 18.33 ms before starting to capture the next image.

[0038] Typically, camera 110 is located at camera area 112 of display 104. Camera area 112 represents, for example, a sub-area of ​​display 104, and camera 110 is located below display 104 at camera area 112. For example, camera 110 is located below the surface of display 104 and below camera area 112, as shown in the examples below. Figure 2 The image is shown in the figure. Here, for illustrative purposes only, a dashed circle is used to indicate the camera area 112 (i.e., the dashed circle is not shown in operation), and it should be understood that the camera area 112 is generally not visually distinguishable from other areas of the display 104. In one or more embodiments, the camera area 112 corresponds to the field of view of the camera 110. The portion of the display 104 other than the camera area 112 is referred to as the peripheral area 114.

[0039] Client device 102 also includes a display interference mitigation system 116. When camera 110 captures an image, display interference mitigation system 116 implements one or more of various techniques to reduce light interference from display 104. One example of these techniques is capturing a calibration image, using the calibration image to generate subtractive images, and then subtracting these subtractive images from subsequently captured images. Another example of these techniques is capturing recalibrated images in certain situations (such as when client device 102 is in a low-light environment or after client device 102 has been dropped or shaken). These recalibrated images may be mixed with or replace previously captured calibration images. Another example of these techniques is altering display 104, such as reducing display brightness, blocking light from camera area 112, expanding camera area 112, etc.

[0040] The display interference mitigation system 116 can be implemented in a variety of different ways. For example, the display interference mitigation system 116 can be implemented as a plurality of instructions stored on a computer-readable storage medium and can be executed by a processing system. Additionally or alternatively, the display interference mitigation system 116 can be implemented at least partially in hardware (e.g., as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), etc.).

[0041] Despite Figure 1 While shown as a separate system, the display interference mitigation system 116 may additionally or alternatively be implemented as part of another system or module. For example, the display interference mitigation system 116 may be implemented as part of the display module 106.

[0042] Various images captured by camera 110 (including calibration and recalibration images, which will be discussed in more detail below) are stored in storage device 118. Storage device 118 refers to a digital storage medium, which can be implemented using any of a variety of storage technologies, such as flash memory or other solid-state storage, disks, optical discs, etc. Storage device 118 may also store various program instructions and data of camera module 108 or display interference mitigation system 116.

[0043] Figure 2A partial cross-sectional view 200 depicts a client device 102 according to one or more embodiments. Cross-sectional view 200 shows a display 104 including a camera region 112 and a peripheral region 114, and a camera 110 fixed to an internal structure 202 of the client device 102. For example, the internal structure 202 represents a printed circuit board (PCB) to which the various components of the client device 102, including the camera 110, are attached. As shown, the camera 110 is located directly below the camera region 112 of the display 104. Typically, light captured at the camera 110 is transmitted through the camera region 112 as part of an image captured by the camera 110, and typically includes light emitted by the display 104. Furthermore, the camera 110 may also transmit light through the camera region 112, for example, to provide flash illumination as part of image capture.

[0044] Various objects (not shown) may optionally be located in front of the display 104 at various times. Examples of such objects include a user's face, a user's fingerprints, various home or office objects (such as telephones, lights, televisions, fans), various objects in a wallet or briefcase, etc. The front of the display 104 refers to the portion of the display opposite to the camera 110, shown as the front of the display 204. Opposite to the front of the display 104 is the rear of the display 104, shown as the rear of the display 206.

[0045] Camera area 112 is usually circular, such as Figure 1 As shown, this corresponds to the field of view of camera 110 through the camera lens. It should be noted that although camera area 112 is circular, light passing through camera area 112 and camera lens will be focused onto the imager, which is generally rectangular.

[0046] Figure 3 An example of an interference mitigation system 300 implementing the techniques discussed herein is shown. The interference mitigation system 300 is... Figure 1 An example of a display interference mitigation system 116 is provided. The display interference mitigation system 300 includes a calibration image acquisition module 302, a region of interest (ROI) color determination module 304, and an enhanced image generation module 306. Typically, the calibration image acquisition module 302 acquires captured calibration images and stores them on a memory device 108. For subsequently captured images, the ROI color determination module 304 determines the colors in the region of interest (e.g., camera region 112) of the image to be displayed on the display 104. The enhanced image generation module 306 generates a subtractive image based on the captured calibration image and the colors in the ROI, and generates an enhanced output image 308 by subtracting the subtractive image from the captured calibration image.

[0047] More specifically, the calibration image collection module 302 collects calibration images by generating a calibration image 310 and transmitting the calibration image 310 to the display module 106. In one or more embodiments, a calibration image 310 is generated for each type of display pixel in the display 104. For example, for an RGB display, the calibration image collection module 302 generates a red calibration image, a green calibration image, and a blue calibration image and provides them to the display module 106. Additionally or alternatively, other types of calibration images may be generated and provided to the display module 106, such as generating a white calibration image if the display includes white pixels, generating a yellow calibration image if the display includes yellow pixels, etc.

[0048] In one or more embodiments, the associated color value of each calibration image 310 is set to high (e.g., set to the maximum value), while the values ​​of other colors are set to low (e.g., set to the minimum value). For example, using the RGB color space (R, G, B), a red calibration image may have all pixels at (255, 0, 0), a green calibration image may have all pixels at (0, 255, 0), and a blue calibration image may have all pixels at (0, 0, 255). In one or more embodiments, for the entire display 104, the associated color value of each calibration image 310 is set to high, while the values ​​of other colors are set to low. Additionally or alternatively, the associated color value of each calibration image 310 is set to high, while the values ​​of other colors are set to low only for a portion of the display 104 (such as camera area 112 or an extended area including camera area 112 and an extended area surrounding camera area 112 (e.g., the radius of the extended area starting from the center of camera area 112 may be 10% or 20% larger than the radius of camera area 112)).

[0049] Additionally or alternatively, in addition to generating calibration images for display by display module 106, calibration image collection module 302 may provide indication to display module 106 to activate only specific pixels of display 104 (e.g., only red pixels for the red calibration image). These specific pixels may be, for example, pixels in camera region 112, including pixels in the camera region and pixels in an extended region surrounding camera region 112. Activating only specific pixels of display 104 allows for energy savings when collecting calibration images, because areas outside camera region 112 (or areas outside camera region 112 that represent at least a 10% or 20% threshold amount) do not need to be activated.

[0050] Display module 106 displays calibration image 310. If the display is not already activated, display module 106 activates display 104 in response to receiving calibration image 310 (e.g., waking display 104 from low power mode). Calibration image 310 displayed by display module 106 is captured by camera 110, and camera module 108 provides the captured calibration image 312 to calibration image collection module 302. Calibration images are captured individually. Thus, if a red calibration image, a green calibration image, and a blue calibration image are displayed, the captured calibration image 312 includes the red captured calibration image, the green captured calibration image, and the blue captured calibration image. For example, when each calibration image 310 is displayed, the calibration image is captured by camera module 108 in response to a calibration image capture signal sent by calibration image collection module 302 to camera module 108.

[0051] The calibration image acquisition module 302 stores the captured calibration images 312 and capture information 314 in the storage device 118. The capture information 314 includes, for example, the sensor gain and integration time (e.g., exposure time) for each captured calibration image 312. Additionally or alternatively, the display interference mitigation system 300 may otherwise know the capture information 314, in which case the capture information 314 does not need to be stored in the storage device 118. For example, the calibration image acquisition module 302 may be configured to use known or static sensor gain and integration time, in which case the sensor gain and integration time of the captured calibration images 312 do not need to be stored in the storage device 118.

[0052] In one or more embodiments, when the client device 102 is in a low-light environment, the camera module 108 displays and captures a calibration image 310. A low-light environment is defined as an environment where the ambient light level is less than a threshold amount (e.g., 1 lux). The ambient light level can be obtained from various sensors such as an ambient light sensor, the imager of camera 110 (e.g., exposure information in camera 110). Typically, a smaller amount of light in the environment allows the captured calibration image 312 to more accurately include only the light generated by display 104, rather than additional light from other sources. As discussed in more detail below, this allows the amount of light introduced into the captured image by display 104 to be subtracted. This display and capture of the calibration image can be performed at different times, such as installing the client device 102 at the factory before delivery to the user, or as part of the user initialization process after purchasing the client device 102.

[0053] In one or more embodiments, the display interference mitigation system 300 for each client device 102 captures a calibration image. Additionally or alternatively, client devices 102 of different types or models or displays 104 of different types or models can operate similarly, and thus calibration images can be displayed and captured for a subset of the types or models of client devices 102 or displays 104 (e.g., a single client device or a single display). The captured calibration image 312 and capture information 314 can then be stored and provided to other client devices 102 of the same type or model (or with the same display 104 type or model), instead of having each individual client device 102 display and capture a calibration image.

[0054] After receiving the captured calibration image 312, the camera module 108 obtains the captured image 316 from the digital camera 110. The camera module 108 obtains the captured image 316 in response to any of a variety of events (such as user input requesting image capture, user request for authentication (e.g., face or iris recognition), detection of an object near the display 104, a request from another device or system, etc.).

[0055] Display module 106 provides the displayed image 318 to region of interest color determination module 104. This image is the one displayed on display 104 when captured image 316 is captured. Region of interest color determination module 304 determines the color in the region of interest of display 104. This region of interest is, for example, a portion of display 104 within the field of view of digital camera 110 (camera area 112). The color in the region of interest of display 104 refers to the color in displayed image 318 within the region of interest (also referred to as the region of interest of displayed image 318).

[0056] Figure 4 An example client device 400 is shown. Client device 400 includes a display 402 that displays an image (e.g., an autumn scene as shown). A camera area 404 is shown, which shows the dominant yellow / orange hue of the image displayed in the camera area 404. It should be noted that although the camera area 404 is shown in a circle, this circle is not required (and typically is not) displayed on the display 402.

[0057] Displayed while captured image 316 is being captured. Figure 4 The image shown is displayed. Therefore, while emitting light to display the image, the digital camera 110 is capturing the captured image 316. Therefore, when capturing the captured image 316, at least some of the light emitted by the display 402 within the camera area 404 is captured by the digital camera 110.

[0058] return Figure 3The region of interest (ROI) color determination module 304 determines the color in the region of interest (ROI) of the displayed image 318 and provides this indication to the enhanced image generation module 306 as the ROI color 320. In one or more embodiments, the ROI color determination module 304 determines the ROI color 320 by determining the amount of that color included in the ROI of the displayed image 318 for each color associated with the calibration image 310. For example, if the calibration image 310 includes a red calibration image, a green calibration image, and a blue calibration image, the ROI color determination module 304 determines the amount of red, the amount of green, and the amount of blue in the ROI of the displayed image 318.

[0059] In one or more embodiments, the region of interest (ROI) color determination module 304 determines the amount of color in the ROI of the displayed image 318 by combining pixel values ​​for that color within the ROI. Pixel values ​​can be combined in various ways, such as by averaging pixel values ​​within the ROI. For example, the amount of red in the ROI can be determined by averaging the red values ​​of the pixels within the ROI of the displayed image 318.

[0060] The enhanced image generation module 306 receives the region of interest color 320 and the captured image 316, and generates an enhanced output image 322 by generating subtractive images for the colors associated with each calibration image 310. These subtractive images are generated based on the calibration image 310, capture information 314, and the region of interest color 320, and are subtracted from the captured image 316, as discussed in more detail below. In one or more embodiments, the enhanced output image 322 is stored in the storage device 118. Additionally or alternatively, the enhanced output image 322 is provided to the display module 106 for display, provided to another module or system for communication with another device, etc.

[0061] Figure 5 An example 500 of generating enhanced output images is shown. Example 500 shows a captured calibration image 502 (red), a captured calibration image 504 (green), and a captured calibration image 506 (blue). For each captured calibration image 502, 504, and 506, the enhanced image generation module 306 generates subtraction images 508, 510, and 512 based on the corresponding multiplier. The multiplier for each color is generated based on the capture parameter ratio and the amount of the corresponding color in the region of interest of the displayed image 318. The capture parameter ratio takes into account the difference in gain and integration time between the captured calibration image 312 and the captured image 316. In one or more embodiments, the multiplier for color C... C) generated as:

[0062]

[0063] Where, Captured_Gain refers to the gain of the captured image 316, Captured_It refers to the integration time of the captured image 316, Calibration_GainC refers to the gain of the captured calibration image 312 corresponding to color C, Calibration_ITC refers to the integration time of the captured calibration image 312 corresponding to color C, and Color_Amount C This refers to the amount of color in the region of interest of the displayed image 318 corresponding to color C. (This is related to the generation of the Multiplier.) C At that time, the capture parameter ratio is:

[0064]

[0065] For each color, each corresponding captured calibration image 502, 504, and 506 is multiplied by a corresponding multiplier to generate corresponding subtractive images 508, 510, and 512. This multiplication refers to multiplying the pixel value of each pixel in the captured calibration images 502, 504, and 506 by the corresponding multiplier. Subtracting 514, 516, and 518 from the captured image 316 yields the subtractive images 508, 510, and 512, resulting in the enhanced output image 322.

[0066] In the examples shown, for subtraction image 508, subtraction means subtracting 514 from the corresponding pixel in captured image 316 to obtain the pixel value of each pixel in subtraction image 508. For subtraction image 510, the pixel value of each pixel in subtraction image 510 is obtained by subtracting 516 from the result of subtracting 514. For subtraction image 512, the pixel value of each pixel in subtraction image 512 is obtained by subtracting 518 from the result of subtracting 514, resulting in enhanced output image 322.

[0067] As shown in Example 500, the enhanced image generation module 306 subtracts from the calibration image the subtracted image for each color, corresponding to the color of light emitted by the display 104 when the display image 318 is displayed. Therefore, when the display image 318 is displayed, the enhanced output image 322 mitigates the interference introduced by the display 104.

[0068] return Figure 3The region of interest (ROI) color determination module 304 can determine the amount of color in the region of interest of the displayed image 318 in various ways other than averaging the pixel values ​​in the ROI. In one or more embodiments, the ROI color determination module 304 uses a trained machine learning system to determine the amount of color in the ROI. A machine learning system refers to a computer representation that can be tuned (e.g., trained) to approximate unknown functions based on input. In particular, a machine learning system may include a system that uses algorithms to learn and predict known data by analyzing known data, in order to learn to generate outputs that reflect patterns and attributes of the known data. For example, a machine learning system may include decision trees, support vector machines, linear regression, logistic regression, Bayesian networks, random forest learning, dimensionality reduction algorithms, augmentation algorithms, artificial neural networks, deep learning, etc.

[0069] A machine learning system can be trained by modifying its features or weights to accurately determine the amount of color in a region of interest (ROI). This training can be performed using either supervised or unsupervised learning. For example, training data can include images of ROIs, and each image can have an associated label indicating the amount of color in the ROI. The machine learning system is trained to generate a certain amount of color in the ROI to reduce or minimize the loss between the labels associated with the images and the amount of color generated by the machine learning system.

[0070] In one or more embodiments, instead of averaging or otherwise combining pixel values ​​in the region of interest to determine the amount of color in the region of interest, the region of interest color determination module 304 identifies the color channel in the region of interest and generates a subtractive image based on each pixel. The region of interest color determination module 304 provides the color channel to the enhanced image generation module 306 as the region of interest color 320.

[0071] Figure 6 An example 600 of the color channel in a region of interest is shown. Display 602 is displaying an image (e.g., an autumn scene as shown). Region of interest 604 is shown, which shows the dominant yellow / orange within region of interest 604. It should be noted that region of interest 604 is shown with a circle, which is not required (and typically not) displayed on display 602.

[0072] The region of interest 604 corresponds to the image 606, which is the image to be captured by the digital camera 110 if the region of interest 604 will be captured by the digital camera 110. Given the known dimensions of the imager and lens of the digital camera 110, the circular region of interest 604 can be easily mapped to the image 606.

[0073] The enhanced image generation module 306 generates color channels for each type of display pixel in the display 602. Each color channel is a grayscale image made from image 606 using only the colors of one type of display pixel. In the example 600 shown, the enhanced image generation module 306 generates a red channel 608 for image 606, a green channel 610 for image 606, and a blue channel 612 for image 606.

[0074] Figure 7 Example 700 of generating an enhanced output image is shown. Example 700 is similar to... Figure 5 In Example 500, except in Example 700, the enhanced image generation module 306 generates a subtractive image based on each pixel, rather than averaging or otherwise combining the pixel values ​​in the region of interest.

[0075] Example 700 shows the captured calibration image 502 (red), 504 (green), and 506 (blue) as described above. Each captured calibration image 502, 504, and 506 is multiplied by a capture parameter ratio that takes into account the difference in gain and integration time between the captured calibration image 312 and the captured image 316. In one or more embodiments, the capture parameter ratio (Ratio) of color C... C ) generated as:

[0076]

[0077] Where, Captured_Gain refers to the gain of the captured image 316, Captured_It refers to the integration time of the captured image 316, and Calibration_Gain C This refers to the gain of the calibrated image 312 corresponding to color C, Calibration_IT C This refers to the integration time of the captured calibration image 312 corresponding to color C.

[0078] For each color channel, the product of the capture parameter ratio and the captured calibration images 502, 504, and 506 is multiplied by the corresponding color channel 610, 612, and 614. This multiplication involves multiplying the pixel value of each pixel in the captured calibration images 502, 504, and 506 (after multiplying by the capture parameter ratio) by the pixel value of the corresponding color channel 610, 612, and 614, respectively. Subtracting the resulting subtracted image from the captured image 316 yields the enhanced output image 322.

[0079] In the illustrated example, for the red channel, subtraction refers to subtracting 702 from the corresponding pixel in the captured image 316 to obtain the pixel value of each pixel in the resulting subtracted image. For the green channel, subtracting 704 from the result of subtracting 702 yields the pixel value of each pixel in the resulting subtracted image. For the blue channel, subtracting 706 from the result of subtracting 704 yields the pixel value of each pixel in the resulting subtracted image, resulting in the enhanced output image 322.

[0080] As shown in Example 500, the enhanced image generation module 306 subtracts a subtracted image from the calibration image for each color channel. This subtracted image corresponds to the light emitted by the display 104 in that color channel when the display image 318 is displayed. Therefore, when the display image 318 is displayed, the enhanced output image 322 mitigates interference introduced by the display 104.

[0081] return Figure 3 In some cases, the capture information 314 is not stored in the storage device 118. In this case, the gain and integration time of the captured calibration image 312 are known to the enhancement image generation module 306, and therefore do not need to be received from the storage device 118.

[0082] Additionally or alternatively, in some cases, the capture parameter ratio is known to the enhancement image generation module 306 and therefore does not need to be determined by it. For example, the calibration image collection module 302 and the camera module 108 can capture images using static (e.g., identical) gain and integration times, so the capture parameter ratio will be readily known (e.g., if the captured calibration image 312 and captured image 316 have the same gain and integration times, then the capture parameter ratio is 1). As another example, the calibration image collection module 302 can store multiple sets of captured calibration images 312 for each possible gain and integration time used by the camera module 108, such as a set of captured calibration images 312. In this case, a set of captured calibration images 312 with the same gain and integration time as captured image 316 is received from the storage device 118, in which case the capture parameter ratio is 1.

[0083] refer to Figure 5 The process of combining or averaging pixel values ​​and using reference values. Figure 7 An example of the enhanced image generation module 306 is discussed, based on subtracting the image from each pixel. Additionally or alternatively, the enhanced image generation module 306 may subtract the image based on any grouping of pixels between pixels, as well as combinations or averages of pixel values. For example, pixel values ​​in the region of interest may be divided into quarters, and a combined (e.g., average) value may be generated for each quarter.

[0084] Different captured calibration images 312 have different repeating patterns, such as Figure 5 and Figure 7 As shown. These patterns are, for example, the result of aligning display pixels in display 104 with imager pixels in digital camera 110. In one or more embodiments, the region of interest color determination module 304 identifies repeating patterns in the captured calibration images and uses the pattern appearing once as a pixel group for each captured calibration image 312. Pixel values ​​in the pattern appearing once are combined (e.g., averaged or otherwise similar to the discussion above) to generate a color corresponding to the single appearance, also referred to herein as the pattern color. For each appearance of the pattern in the calibration image, when generating the Multiplier... C The color of this pattern will be used as Color_Amount. C .

[0085] In one or more embodiments, when the region of interest color determination module 304 identifies a repeating pattern in the calibration image, the display module 106 activates only the pixels of the display 104 in the camera region 112 that include the pattern appearing only once. When the pattern repeats, the display module 106 uses the single-appearance pattern instead of capturing the entire camera region 112. Additionally or alternatively, the display module 106 activates less of the entire camera region 112 compared to activating the pixels of the display in the camera region 112 that displays multiple appearances of the pattern.

[0086] This document includes a discussion of an enhanced image generation module 306, which generates an enhanced output image 322 by subtracting one or more subtraction images from a captured image 316. Additionally or alternatively, the enhanced image generation module 306 generates the enhanced output image 322 in other ways based on one or more of the captured image 316, the displayed image 318, and the captured calibration image 312. For example, a machine learning system can be trained to generate the enhanced output image 322 by modifying its features or weights. This training can be performed using supervised or unsupervised learning. For example, training data can include datasets, each including captured and displayed images. Each training dataset can have associated labels, such as a target enhanced output image, which is the image that would be captured by the camera module 108 without interference from the display 104. For example, a target enhanced output image can be captured when the display 104 is off (not emitting any light). The machine learning system is trained to generate enhanced output images to reduce or minimize the loss between the labels associated with the training datasets and the enhanced output images generated by the machine learning system. Various additional data may also be optionally included in the training data, such as captured calibration images.

[0087] In one or more embodiments, the calibration image acquisition module 302 stores multiple sets of captured calibration images 312, such as different sets of captured calibration images 312 for different temperature thresholds (e.g., for a sensed temperature outside the client device 102, a sensed temperature inside the client device 102, or a sensed temperature inside a specific component of the client device 102 (such as the display 104). For example, if the sensed temperature is below the temperature threshold, one set of captured calibration images 312 is used, while if the sensed temperature is equal to or above the temperature threshold, another set of captured calibration images 312 is used. The use of such different sets of captured calibration images 312 allows the display interference mitigation system 300 to consider changes to the display or capture of calibration images based on temperature.

[0088] Additionally or alternatively, when generating the subtractive image, the enhancement image generation module 306 may use interpolation to determine a temperature-based calibration image based on two different captured calibration images 312. For example, two sets of captured calibration images 312 may be stored, one set captured at a low temperature (e.g., the lowest typical temperature of client device 102) and the other set captured at a high temperature (e.g., the highest typical temperature of client device 102). When generating the subtractive image, the enhancement image generation module 306 interpolates between the low-temperature calibration image set and the high-temperature calibration image set based on the current temperature to determine the calibration image set based on the current temperature, and uses the determined set of calibration images to generate the subtractive image. The enhancement image generation module 306 may use various types of interpolation, such as linear interpolation or other interpolation shapes.

[0089] In one or more embodiments, an interference mitigation system is shown (e.g., Figure 1 Display interference reduction system 116 or Figure 3 The display interference mitigation system 300 includes a recalibration module. The alignment and geometry of the display pixels in the display 104 with the imager pixels in the digital camera 110 result in various patterns in calibrated and other captured images as described above. These patterns may change if the alignment or geometry of the display pixels with the imager pixels changes for any reason. Therefore, in various cases, the recalibration module causes newly captured calibrated images to be collected to reflect changes (or potential changes) in the alignment or geometry of the display pixels with the imager pixels.

[0090] Figure 8 An example recalibration module 800 is shown. In one or more embodiments, the recalibration module 800 is included in a display interference mitigation system (e.g., Figure 1 Display interference mitigation system 116 or Figure 3The display interference mitigation system 300. The recalibration module 800 includes a recalibration determination module 802, a calibration image verification module 804, and a calibration image merging module 806.

[0091] Typically, the recalibration determination module 802 determines when to collect a new calibration image (also known as a recalibration image) based on various trigger data 808. When a recalibration image is collected, the calibration image verification module 804 receives the recalibration image, and the calibration image merging module 806 merges or mixes the recalibration image with previously stored calibration images.

[0092] The recalibration determination module 802 receives trigger data 808 from various sources and determines when a trigger event has occurred based on the trigger data 808. A trigger event is anything that may cause or indicate a change in the alignment or geometry of the display pixels and imager pixels, and thus instructs the recalibration module 800 to collect a recalibrated image. In response to the occurrence of a trigger event, the recalibration determination module 802 determines the time to collect the recalibrated image based on the environment in which the client device 102 is located.

[0093] The recalibration determination module 802 receives trigger data 808 from any of a variety of different sources, such as various sensors in client device 102, the operating system of client device 102, digital camera 110, etc. In one or more embodiments, the recalibration determination module 802 receives trigger data 808 from one or more of various low-power sensors, such as proximity sensors, ambient light sensors, accelerometers, gyroscopes, etc. These low-power sensors typically consume less power than digital camera 110. By using data from low-power sensors to determine whether a trigger event has occurred or to determine the time for collecting a recalibration image, client device 102 can save power by not activating or waking up the high-power digital camera 110 until a trigger event occurs and the time for collecting a recalibration image is reached.

[0094] The recalibration determination module 802 determines when a triggering event occurs in any of a variety of different ways. In one or more embodiments, the recalibration determination module 802 determines when a triggering event occurs based on elapsed time, such as determining that the triggering event occurs at regular or irregular intervals (e.g., weekly). Trigger data 808 includes indications of timing in the client device 102, such as an indication of the current time, an indication that a timer has expired, etc. This timing indication can be received from various sources, such as an operating system running on the client device 102.

[0095] Additionally or alternatively, the recalibration determination module 802 determines when a trigger event occurs based on temperature. Trigger data 808 includes an indication of the internal or external temperature (e.g., the temperature of the environment in which the client device 102 is located) of the client device 102 (or a component of the client device, such as display 104). For example, the recalibration determination module 802 determines that a trigger event has occurred in response to the internal temperature exceeding a first threshold or falling below a second threshold, or in response to the external temperature exceeding a third threshold or falling below a fourth threshold.

[0096] Additionally or alternatively, the recalibration determination module 802 determines when a triggering event occurs based on the movement of the client device 102. For example, the recalibration determination module 802 determines that a triggering event has occurred in response to the client device 102 being dropped or impacted. Trigger data 808 includes indications of whether the client device 102 has been dropped or impacted, such as notifications from the operating system running on the client device 102 that the client device 102 has been dropped or impacted, indications of motion detected by an accelerometer or gyroscope (e.g., quantity, type, rate, etc.), etc. Where training data 808 includes indications of motion (e.g., quantity, type, rate, etc.), the recalibration determination module 802 uses any of a variety of public or proprietary techniques to determine whether the client device 102 has been dropped or impacted. For example, movement data indicating a vertical descent followed by a sudden stop may indicate that the client device 102 has dropped or fallen, movement data indicating a sudden change in near-horizontal movement may indicate that the client device 102 has been impacted, and so on.

[0097] Additionally or alternatively, the recalibration determination module 802 determines when a triggering event occurs based on the authentication failure of the client device 102. For example, trigger data 808 includes an indication of user authentication failure based on a captured image (e.g., facial or fingerprint features were not recognized in an image captured by the digital camera 110 after the user provided a password or personal identification number to authenticate). Trigger data 808 includes an indication of when user authentication failed based on the captured image, such as an indication from the operating system running on the client device 102 or an indication from the security or authentication system of the client device 102.

[0098] This document discusses various methods for determining when a triggering event occurs. It should be understood that the recalibration determination module 802 can use any one or a combination of these different methods to determine when a triggering event occurs. For example, the recalibration determination module 802 can use any of the different methods discussed herein to determine when a triggering event occurs. As another example, the recalibration determination module 802 can determine that a triggering event has occurred when at least two of the different methods discussed herein (e.g., the internal temperature of the display 104 exceeds a first threshold and user authentication based on a captured image fails) occur.

[0099] In response to a triggering event, the recalibration determination module 802 determines the time to collect a recalibration image based on the current environment in which the client device 102 is located. This determination is based on any of a variety of criteria. In one or more embodiments, the recalibration determination module 802 determines that the recalibration image is collected when the client device 102 is in a low-light environment. The light level in the current environment can be obtained from various sensors, such as the imager of camera 110 (e.g., exposure information in camera 110), an ambient light sensor, etc. A low-light environment is defined as an environment where the light level (e.g., ambient light level) is less than a threshold amount (e.g., 1 lux). Typically, less light in the environment allows recalibration images to be captured more accurately because they include only the light generated by display 104 and not additional light from other sources.

[0100] Additionally or alternatively, the recalibration determination module 802 determines the timing of collecting the recalibration image based on whether the object is adjacent to the display 104. Whether an object is adjacent to the display 104 can be determined based on data or indications from various sensors, such as a proximity sensor that determines the proximity of an object in front of the display 104 (e.g., an object that may be within the field of view of the digital camera 110). Such a proximity sensor can provide an indication that the object is within a threshold distance of the display 104, provide data indicating the distance of the nearest object to the display 104, etc. An object adjacent to the display 104 is defined as an object within a threshold distance of the display 104, such as 6 inches.

[0101] The digital camera 110 captures images via the display 104. Therefore, if an object is adjacent to the display 104, the captured recalibrated image is more likely to include light emitted from the display 104 and reflected by the object, resulting in a more likely captured recalibrated image containing reflected light and thus less accurately capturing light emitted solely by the display 104. Therefore, when no object is adjacent to the display 104, the recalibration determination module 802 determines to collect a recalibrated image.

[0102] Additionally or alternatively, the recalibration determination module 802 determines the timing of collecting the recalibration image based on the orientation of the client device 102. The orientation of the client device 102 can be determined based on data or indications from various sensors, such as accelerometers or gyroscopes. Such sensors can provide indications of the orientation of the client device 102 (e.g., whether the client device 102 is horizontally oriented, whether the display 104 is facing upwards (e.g., away from the Earth rather than towards it), data indicating the direction the display 104 faces in three-dimensional space, etc.

[0103] Digital camera 110 captures images via display 104. Therefore, if an object is close to display 104, the captured recalibrated image is more likely to include light emitted from display 104 and reflected by the object, resulting in a captured recalibrated image that is more likely to include reflected light and thus less accurately captures the light emitted by display 104. If client device 102 is horizontally oriented and display 104 is facing upwards, client device 102 is more likely to be placed on a table or other surface where no object will reflect light. Conversely, if client device 102 is vertically oriented and display is perpendicular to the ground, client device 102 is more likely to be in a wallet, pocket, backpack, or other carrying device that will reflect light or include other objects that will reflect light. Therefore, when client device 102 is horizontally oriented (or within a threshold amount in the horizontal direction, such as 5 degrees) and display 104 is facing upwards (or within a threshold amount in the upward orientation, such as 5 degrees), recalibration determination module 802 determines to collect a recalibrated image.

[0104] Additionally or alternatively, the recalibration determination module 802 determines the time for collecting the recalibration image based on the motion of the client device 102. Whether the client device 102 is moving or stationary can be determined based on data or indications from various sensors such as accelerometers or gyroscopes. Such sensors can provide indications that the client device 102 is moving or not moving (e.g., stationary or static), provide data indicating the speed at which the client device 102 is moving, etc.

[0105] Digital camera 110 captures images via display 104. Therefore, if an object is close to display 104, the captured recalibrated image is more likely to include light emitted from display 104 and reflected by the object, resulting in a captured recalibrated image that may contain reflected light and thus less accurately capture light emitted solely by display 104. If client device 102 is stationary, it is more likely that it is stationary at the location surface without an object that would reflect light in front of display 104. Conversely, if client device 102 is moving, it is more likely that it will encounter an object that would reflect light in front of display 104. Therefore, recalibration determination module 802 determines when to collect a recalibrated image when client device 102 is stationary (or within a threshold amount of stationary movement, such as 1 inch per second).

[0106] Additionally or alternatively, when the client device 102 is stationary relative to a vehicle in motion, the recalibration determination module 802 determines to collect the recalibration image. For example, if the client device 102 is moving at a speed greater than 45 miles per hour, the recalibration determination module 802 can determine that the client device 102 is in a car, train, airplane, or other vehicle. Therefore, the client device 102 is less likely to encounter an object reflecting light in front of the display 104.

[0107] Additionally or alternatively, the recalibration determination module 802 determines the time for collecting the recalibration image based on whether the client device 102 is charging (e.g., the battery of the client device 102 is currently charging) or physically connected to another device or outlet (e.g., via a Universal Serial Bus (USB) cable). Whether the client device 102 is charging or connected to another device or outlet can be determined based on data or indications from various sources, such as indications from the operating system on the client device 102 that the client device 102 is charging or connected to another device or outlet, or indications that the client device 102 is connected to another device or outlet via a USB cable, etc.

[0108] Digital camera 110 captures images via display 104. Therefore, if an object is close to display 104, the captured recalibrated image is more likely to include light emitted from display 104 and reflected by the object, resulting in a more likely captured recalibrated image containing reflected light and thus less accurately capturing light emitted solely by display 104. If client device 102 is charging or connected to another device or outlet, client device 102 is more likely to be located in a position where there is no reflected light in front of display 104. Conversely, if client device 102 is not charging and not connected to another device or outlet, client device 102 is more likely to be moving or elsewhere, thus encountering an object reflecting light in front of the display. Therefore, recalibration determination module 802 determines that a recalibrated image is collected when client device 102 is charging or connected to another device or outlet.

[0109] This document discusses various criteria used to determine the timing of collecting a recalibration image in response to a triggering event. It should be understood that the recalibration determination module 802 may use any one or a combination of these criteria. For example, the recalibration determination module 802 may determine to collect a recalibration image in response to the client device 102 being in a low-light environment. As another example, the recalibration determination module 802 may determine to collect a recalibration image in response to the client device 102 being in a low-light environment with no object adjacent to the display 104. As yet another example, the recalibration determination module 802 may determine to collect a recalibration image in response to the client device 102 being in a low-light environment and the display 104 being in an upward horizontal orientation.

[0110] In one or more embodiments, the recalibration determination module 802 activates the display 104 in response to determining the current time for collecting the recalibration image. The display 104 is activated by transmitting a screen activation request to the display module 106. In response to the screen activation request, if the display 104 is not already activated, the display module 106 activates the display 104 (e.g., wakes the display 104 from a low-power mode). This allows the display 104 to be activated and ready to display the calibration image 310, thereby allowing the captured recalibration image to be captured.

[0111] Similarly, in one or more embodiments, the recalibration determination module 802 activates the digital camera 110 in response to determining the current time for collecting the recalibration image. The digital camera 110 is activated by transmitting a camera activation request to the camera module 108. In response to the camera activation request, if the digital camera 110 is not already activated, the camera module 108 activates the digital camera 110 (e.g., waking it from a low-power mode). This allows the digital camera 110 to be activated and ready to capture the displayed calibration image 310.

[0112] In response to determining that a triggering event has occurred and that it is the current time for collecting the recalibrated image, the recalibration determination module 802 sends a recalibration request 810 to... Figure 3 The calibration image collection module 302, in response to a recalibration request 810, sends a calibration image 310 to the display module 106 (or instructs the display module 106 to activate only specific pixels of the display 104) and receives a captured calibration image 312 and capture information 314 from the camera module 108 as described above. In one or more embodiments, the calibration image 310 is the same calibration image used for previously or initially collected captured calibration images 312. Similar to the discussion above, except in response to performing the recalibration request 810, these calibration images 310 and captured calibration images 312 are displayed and captured. Therefore, the captured calibration images 312 may differ from previously captured calibration images 312 due to changes in the alignment or geometry of the display pixels relative to the imager pixels. These captured calibration images 312 are captured in response to the recalibration request 810 and are therefore also referred to as recalibrated images. These recalibrated images have capture information similar to the capture information 314 discussed above.

[0113] The calibration image verification module 804 receives the captured recalibrated image 812 and capture information 814 from the calibration image collection module 302, and provides the captured recalibrated image 812 and capture information 814 to the calibration image merging module 806. In one or more embodiments, the calibration image verification module 804 provides the captured recalibrated image 812 and capture information 814 to the calibration image merging module 806 without analyzing the captured recalibrated image 812. Additionally or alternatively, the calibration image verification module 804 analyzes the captured recalibrated image 812 and determines whether to compare the calibration image 812 with a previously stored calibration image (e.g., ...). Figure 3 The captured calibration images (312) are merged or mixed.

[0114] As described above, when capturing a recalibrated image, the digital camera 110 may capture light received from various objects. These reflections are, for example, reflections of light emitted by the display 104 during the capture of the recalibrated image. In one or more embodiments, the calibration image verification module 804 analyzes the captured recalibrated image 812 to determine whether aberrations exist in the captured recalibrated image 812. Such aberrations are, for example, light reflected from objects in front of the display 104 and captured by the digital camera 110, or the result of a user interrupting the client device 102 during the capture of the captured recalibrated image 812.

[0115] The calibration image verification module 804 can identify aberrations in any of a variety of different ways. As described above, although each captured recalibrated image may have various repeating patterns, each recalibrated image is monochrome (e.g., red, green, or blue). In one or more embodiments, the calibration image verification module 804 analyzes the captured recalibrated images 812 to identify one or more non-repeating patterns in each captured recalibrated image 812. Non-repeating patterns are likely reflections from objects and are therefore considered aberrations. Additionally or alternatively, the calibration image verification module 804 analyzes the captured recalibrated images 812 to identify blur (e.g., due to user movement of the client device 102) or other artifacts in the captured recalibrated images 812 that are considered aberrations.

[0116] The calibration image verification module 804 may also optionally determine the magnitude of the aberration. Light reflected from an object may result in a larger non-repeating pattern (e.g., in terms of pixel size, or relative to the intensity of other parts of the captured recalibrated image) compared to alignment or geometry variations between display pixels and imager pixels, or damage to the digital camera 110 (e.g., defective pixels in the imager of the digital camera 110). Therefore, if the magnitude of the aberration exceeds a threshold amount (e.g., having a number of pixels exceeding 25% of the number of pixels in a repeating pattern of the image, or having an intensity exceeding 25% of the average intensity of pixels in the image), the calibration image verification module 804 determines that the aberration is a result of light reflected from the object.

[0117] In one or more embodiments, the calibration image verification module 804 determines, based on additional captured recalibrated images, whether aberrations in the captured recalibrated images are a result of light reflected from an object or a result of user interruption. For example, suppose the captured recalibrated images 812 include red, green, and blue images. If aberrations occur in each of the three images at slightly different locations, the calibration image verification module 804 can determine that the aberrations are a result of light reflected from a moving object (e.g., a fan blade). This determination can be made even if, for example, the magnitude of the aberration does not exceed a threshold amount.

[0118] In one or more embodiments, for each captured recalibrated image 812 without any aberrations, the phase difference being determined to be a result of light reflected from an object, the calibration image verification module 804 provides the captured recalibrated image 812 to the calibration image merging module 806 for merging. If the captured recalibrated image 812 has aberrations, the phase difference being determined to be a result of light reflected from an object or a result of user interruption, the calibration image verification module 804 does not provide the captured recalibrated image 812 to the calibration image merging module for merging. This allows the recalibration module 800 to avoid merging captured recalibrated images with significant aberrations with previously captured calibration images or recalibrated images. Additionally or alternatively, if the captured recalibrated image 812 has aberrations, the phase difference being determined to be a result of light reflected from an object or a result of user interruption, the calibration image verification module 804 provides the captured recalibrated image 812 and an indication of the aberration location to the calibration image merging module 806. This allows the calibration image merging module 806 to merge a portion of the captured recalibrated image 812 without aberrations with a previously captured calibration or recalibrated image.

[0119] In one or more embodiments, the recalibration module 800 merges a captured recalibrated image with a previously captured calibration or recalibrated image based on each image. For example, if the captured recalibrated image 812 includes a red image, a green image, and a blue image, one of those images (e.g., the red image) can be merged with a previously captured calibration or recalibrated image, while another of those images (e.g., the blue image) is not merged with a previously captured calibration or recalibrated image because the aberrations in that image (e.g., the blue image) are determined to be a result of light reflected from an object or object, or a result of user interruption.

[0120] Additionally or alternatively, the recalibration module 800 merges the captured recalibrated images with previously captured calibration or recalibrated images based on groups. For example, if the captured recalibrated images 812 include red, green, and blue images, each of those images is merged with the corresponding previously captured calibration or recalibrated image only if none of the three captured recalibrated images 812 are determined to be an aberration resulting from light reflected from an object or from a user-interrupted result.

[0121] The calibration image merging module 806 merges the captured recalibrated image 812 with previously captured calibration images stored in the storage device 118. These previously captured calibration images may be the initially captured calibration image 312 as described above or the previously captured recalibrated image 812. In the discussion herein, these previously captured recalibrated images are also referred to as previously captured calibration images.

[0122] In one or more embodiments, the calibration image merging module 806 merges the captured recalibrated image 812 with the previously captured calibration image in the storage device 118 by replacing the previously captured calibration image with the captured recalibrated image 812, as shown as a merged calibration image 814. The calibration image merging module 806 also stores capture information 818 corresponding to the merged calibration image 814 in the storage device 118.

[0123] Additionally or alternatively, the calibration image merging module 806 merges the captured recalibrated image 812 with the previously captured calibration image 820 by retrieving the previously captured calibration image 820 from the storage device 118 and mixing the previously captured calibration image 820 with the captured recalibrated image 812. The calibration image merging module 806 replaces the previously captured calibration image in the storage device 118 with the merged calibration image, shown as a merged calibration image 816. The calibration image merging module 806 also stores the capture information 818 corresponding to the merged calibration image in the storage device 118.

[0124] The calibration image merging module 806 can blend a previously captured calibration image 820 with a captured recalibrated image 812 in any of a variety of different ways. In one or more embodiments, the two images are blended using a ratio (such as 80 / 20 or 50 / 50) between the previously captured calibration image 820 and the captured recalibrated image 812. For example, using an 80 / 20 ratio for each pixel value in the blended calibration image, the blended calibration image is generated by adding 80% of the value of the corresponding pixel value in the captured recalibrated image 812 to 20% of the value of the corresponding pixel value in the previously captured calibration image 820. Similarly, the capture information of the blended calibration image in this example is generated by adding 80% of the value of the capture information 814 of the captured recalibrated image 812 (e.g., for each gain and integration time) to 20% of the value of the capture information 822 of the previously captured calibration image 820 received from the storage device 118 (e.g., for each gain and integration time). Optionally, the two images are registered (lit up) before mixing to account for, for example, hardware variations between the time the two images are captured.

[0125] Alternatively or additionally, the recalibrated image 812 can be captured using the same capture information (e.g., gain and integration time) as the previously captured calibration image 312. This can be achieved in various ways, for example, the calibration image collection module 302 instructs the camera module 108 to capture information for capturing the recalibrated image. In this case, it is not necessary to receive capture information 814 from the calibration image collection module 302. Instead, capture information 822 received from the storage device 118 can be used as capture information 814.

[0126] Additionally or alternatively, the calibration image merging module 806 uses multiple previously captured calibration images 820 and capture information 822. The multiple previously captured calibration images 820 and capture information 822 can be stored in the storage device 118 and weighted differently to generate a blended image. For example, a 60 / 30 / 10 ratio can be used to obtain a blended image obtained by adding 60% of the pixel values ​​from the captured recalibrated image 812, 30% of the pixel values ​​from a first previously captured calibration image 820 (e.g., the most recently captured calibration image 820), and 10% of the pixel values ​​from a second previously captured calibration image 820 (e.g., the second most recently captured calibration image 820).

[0127] This document includes a discussion of the calibration image merging module 806 blending a previously captured calibration image 820 with a captured recalibrated image 812. Additionally or alternatively, the calibration image merging module 806 uses a machine learning system to blend the previously captured calibration image 820 with the captured recalibrated image 812. For example, the machine learning system can be trained by modifying its features or weights to generate a merged calibration image 816. This training can be performed using supervised or unsupervised learning. For example, the training data can include datasets, each containing previously captured calibration images and captured recalibrated images. Each training dataset can have associated labels, such as the merged calibration image, which will be a good merge of the previously captured calibration image and the captured recalibrated image. For example, a good merge of the two images can be generated manually. The machine learning system is trained to generate the merged calibration image to reduce or minimize the loss between the labels associated with the training datasets and the merged calibration image generated by the machine learning system.

[0128] As described above, in one or more embodiments, a portion of the captured recalibrated image 812 includes aberrations. In this case, the calibration image merging module 806 generates a blended calibration image using a portion of the captured recalibrated image 812 that does not include aberrations and at least a portion of the previously captured calibration image 820 that corresponds to the aberrations. For example, the calibration image verification module 804 provides the calibration image merging module 806 with an indication that a portion of the captured recalibrated image 812 includes aberrations. For the portion of the captured recalibrated image 812 that includes aberrations, the calibration image merging module 806 generates the blended image by using pixel values ​​from the previously captured calibration image 820 that correspond to the pixel values ​​in the aberration-including portion of the captured recalibrated image 812 as pixel values ​​in the blended image. For the portion of the captured recalibrated image 812 that does not include aberrations, the calibration image merging module 806 generates the blended image by using pixel values ​​from the previously captured calibration image 820 that correspond to the pixel values ​​in the portion of the captured recalibrated image 812 that does not include aberrations, or by combining the pixel values ​​from the previously captured calibration image 820 and the captured recalibrated image 812 in other ways (e.g., according to a certain ratio as described above).

[0129] In cases where a portion of the captured recalibrated image 812 includes aberrations, the calibration image merging module 806 can determine the capture information of the blended image in various different ways. In one or more embodiments, as described above, the capture information of the blended image is the capture information 822 of the previously captured calibration image 820. Additionally or alternatively, the capture information 822 of the previously captured calibration image 820 and the capture information 814 of the captured recalibrated image 812 can be combined in the same manner (e.g., at the same ratio) as the pixel values ​​of the aberration-free portion of the captured recalibrated image 812.

[0130] After the merged calibration image 816 has been stored in storage device 118, the calibration image merging module 806 sends a reset instruction 824 to the recalibration determination module 802. The reset instruction 824 notifies the recalibration determination module 802 that the recalibrated image has been captured, merged, and stored in storage device 118. In response to the reset instruction 824, the recalibration determination module 802 resumes analysis of trigger data 808 to determine when the trigger event occurred.

[0131] It should be noted that in some cases, the calibration image collection module 302 collects multiple sets of calibration images, such as different sets of calibration images for each of multiple different gain and integration time values. In this case, in response to the recalibration request 810, the same set of recalibration images is also collected.

[0132] return Figure 3 In one or more embodiments, an interference mitigation system is shown (e.g., Figure 1 Display interference reduction system 116 or Figure 3 The display interference mitigation system 300 includes a brightness adjustment module. The brightness adjustment module reduces interference from the display 104 during image capture by adjusting the brightness of the display. The brightness adjustment module can use various techniques to adjust the brightness of the display, as discussed in more detail below.

[0133] Figure 9 An example brightness adjustment module 900 is shown. In one or more embodiments, the brightness adjustment module 900 is included in a display interference mitigation system (e.g., Figure 1 Display interference mitigation system 116 or Figure 3 In a display interference mitigation system 300, the brightness adjustment module 900 can be used in conjunction with techniques that employ calibration and recalibration images, resulting in an enhanced output image being displayed while brightness is adjusted by the brightness adjustment module 900. Additionally or alternatively, the brightness adjustment module 900 may be... Figure 1 The display interference mitigation system 116, in this case, does not need to generate and capture calibration images, does not need to generate and capture recalibration images, and does not need to generate enhanced output images.

[0134] The brightness adjustment module 900 includes an adjustment determination module 902, a brightness control module 904, and a light-blocking control module 906. Typically, the adjustment determination module 902 determines, while the digital camera 110 is capturing an image, whether to adjust the brightness of the display 104 or to block out a portion of the display 104 to reduce or eliminate the impact of light emitted by the display 104 on the image captured by the digital camera 110. When the brightness of the display 104 needs to be adjusted, the brightness control module 904 determines when and how to reduce the brightness of the display 104. When a portion of the display 104 needs to be blocked, the light-blocking control module 906 determines the portion of the display 104 that needs to be blocked and when that portion needs to be blocked.

[0135] Although the brightness adjustment module 900 is shown as including a brightness control module 904 and a light-shielding control module 906, additionally or alternatively, the brightness adjustment module 900 may not include both modules 904 and 906. For example, if the brightness adjustment module 900 does not support adjusting the brightness of the display 104, then the brightness control module 904 need not be included in the brightness adjustment module 900. As another example, if the brightness adjustment module 900 does not support shading a portion of the display 104, then the light-shielding control module 906 need not be included in the brightness adjustment module 900.

[0136] The adjustment determination module 902 receives an ambient light level measurement 910. The ambient light level measurement 910 is an indication of the light level in the environment where the client device 102 is located. In one or more embodiments, as described above, the ambient light level measurement 910 is obtained from a low-power ambient light sensor. Additionally or alternatively, the ambient light level measurement 910 may be obtained from other sensors such as the digital camera 110 (e.g., the ambient light level measurement 910 may be a lux level measured by the imager of the digital camera 110).

[0137] Generally, the lower the ambient light level, the more light emitted from display 104 will interfere with the image captured by digital camera 110. In one or more embodiments, when the ambient light level is sufficiently high (e.g., at least 50,000), the light emitted from display 104 does not have a visual effect of interfering with the image captured by digital camera 110, so the display interference mitigation system 116 does not need to take any measures to mitigate the interference from the light emitted from display 104.

[0138] The adjustment determination module 902 determines, at least in part, a suitable technique for mitigating interference from light emitted from the display 104 based on ambient light level measurement 910. In one or more embodiments, the adjustment determination module 902 determines the appropriate technique to use by balancing the mitigation of interference in the captured image with the disruption of the user experience. For example, the adjustment determination module 902 selects an appropriate technique to mitigate interference in the captured image while minimizing (e.g., reducing or minimizing) the disruption to the user experience.

[0139] In one or more embodiments, the adjustment determination module 902 determines which technique to use (e.g., blocking a portion of the screen or reducing the screen brightness) based on ambient light level measurements and different threshold levels. As an example, for low light conditions (e.g., less than 50 lux), the adjustment determination module 902 determines to block a portion of the display 104; for medium light conditions (e.g., between 50 and 1000 lux), the adjustment determination module 902 determines to reduce the brightness of the display 104; for medium to high light conditions (e.g., between 1000 and 50,000 lux), the adjustment determination module 902 determines to gate the display 104; and for strong light conditions (e.g., greater than 50,000 lux), the adjustment determination module 902 determines not to take any measures to mitigate interference from the light emitted by the display 104.

[0140] It should be noted that the threshold levels used for employing these different techniques often vary based on the specific model or type of display 104 and the specific model or type of digital camera 110 implemented in client device 102. Different models or types of digital cameras 110 have different light sensitivities, and therefore can have different threshold levels for employing these different techniques. Similarly, due to the use of different techniques to enable the display 104 of the digital camera 110 to achieve different threshold levels, such as by reducing... Figure 1 Camera area 112 relative to Figure 1 The number of pixels in the peripheral region 114, the geometry of the pixels in the camera region 112 relative to the peripheral region, and the brightness of the pixels in the camera region 112 relative to the peripheral region 114 are all used to capture images. Therefore, different models or types of displays 104 have different characteristics. These different characteristics result in different models or types of displays 104 having different threshold levels when using these different technologies.

[0141] For example, a client device 102 may use a specific type of digital camera 110 and a specific type of display 104, resulting in a reduction in display brightness sufficient to mitigate interference from light emitted from display 104 when the ambient light level is measured between 40 and 50 lux. Therefore, in this client device 102, when the ambient light level is measured between 40 and 50 lux, the display brightness is reduced, rather than partially blocking light from display 104. However, another client device 102 may use a different type of digital camera 110 and a different type of display 104, resulting in a reduction in display brightness insufficient to mitigate interference from light emitted from display 104 when the ambient light level is measured between 40 and 50 lux. Therefore, in this client device 102, when the ambient light level is measured between 40 and 50 lux, a portion of display 104 is blocked, rather than the display brightness is reduced.

[0142] The appropriate threshold level for using these different techniques in different combinations of display and digital camera types can be determined in any of a variety of different ways. In one or more embodiments, the appropriate threshold level is determined empirically by the developer or designer by observing the differences between captured images obtained using different ambient light levels, different threshold levels, and different techniques and a reference image (e.g., an image captured when display 104 is off and not emitting light). Additionally or alternatively, the configuration module on client device 102 can determine how close the captured images obtained using different ambient light levels, different threshold levels, and different techniques are to the reference image, and select a value corresponding to the least interfering technique that is sufficiently close to the reference image (e.g., within a threshold amount, such as a pixel value variation of less than 5%) as a given threshold level.

[0143] In response to determining that a portion of the display 104 should be shaded, the adjustment determination module 902 transmits a shading adjustment instruction 912 to the shading control module 906, which determines the portion of the display 104 to be shaded and when that portion will be shaded. The shading adjustment instruction 912 includes an indication of an ambient light level measurement 910, allowing the shading control module 906 to determine the appropriate manner for shading a portion of the display 104. In one or more embodiments, in response to an ambient light level measurement 910 being less than a first threshold (e.g., 1 lux), the shading control module 906 shades the camera region 112 and the extended portion of the peripheral region 114; in response to an ambient light level measurement 910 being at least the first threshold (e.g., 1 lux) but less than a second threshold (e.g., 50 lux), the camera region 112 is shaded (but the extended region is not shaded); and in response to an ambient light level measurement 910 being between a third threshold (e.g., 10,000 lux) and a fourth threshold (e.g., 50,000 lux), when the camera region 112 is shaded, the display for capturing images is gated.

[0144] Alternatively or additionally, the light-shielding control module 904 may light-shield a portion of the display 104 (e.g., camera area 112 or camera area 112 with an extension) in the same manner, regardless of the ambient light level measurement 910. Alternatively or additionally, the manner in which a portion of the light-shielding display 104 may be a user-selectable configuration setting.

[0145] Blocking the camera area 112 (but not the extended portion of the peripheral area 114) means deactivating the pixels in the camera area 112 or otherwise displaying them as black. When blocked, no light is emitted from the camera area 112, therefore the camera area 112 will not emit light that would interfere with image capture. The light-blocking control module 906 is pre-programmed or otherwise aware of the position of the camera area 112 and provides a screen light-blocking instruction 914 to the display module 106 to deactivate the pixels in the camera area 112 or otherwise display them as black.

[0146] In one or more embodiments, the display module 106 is synchronized with the camera module 108 such that the shading of the camera area 112 occurs simultaneously with the image capture by the digital camera 110. Therefore, the camera area 112 is shaded while the digital camera 110 is capturing an image, but not at other times. Additionally or alternatively, the display module 106 may otherwise determine the timing for shading the camera area 112, such as when the imager of the digital camera 110 is active (e.g., powered on).

[0147] Figure 10An example client device 1000 is shown. Client device 1000 includes a display 1002 that displays an image (e.g., an autumn scene as shown). Camera region 1004 is illustrated as being blocked from light, so that light that might interfere with capturing the image is not emitted from camera region 1004. In one or more embodiments, camera region 1004 corresponds to the field of view of camera 110, and therefore camera region 1004 includes only the portion of display 1002 within the field of view of camera 110.

[0148] return Figure 9 Blocking the light from the extended portion of camera area 112 and peripheral area 114 means deactivating pixels in the extended area or otherwise displaying pixels as black. This extended area includes camera area 112 and some peripheral areas surrounding camera area 112. When blocked, no light will be emitted from the extended area, therefore, the extended area will not emit light that would interfere with image capture.

[0149] The light-blocking control module 906 is pre-programmed or otherwise knows the location of the camera area 112. The light-blocking control module 906 also determines the peripheral area surrounding the camera area 112 and the amount included in the extended area. This amount can be identified in various ways, such as a percentage of the size of the camera area 112 (e.g., determining the radius of the light-blocking area from the center of the camera area 112 to be 10% or 20% larger than the radius of the camera area 112), a specific number of pixels (e.g., determining the radius of the light-blocking area to be 20 or 50 pixels larger than the radius of the camera area 112), etc. The light-blocking control module 906 provides a screen light-blocking instruction 914 to the display module 106 to deactivate or otherwise display the pixels in the extended area as black.

[0150] In one or more embodiments, the display module 106 is synchronized with the camera module 108 such that shading of the extended area occurs simultaneously with image capture by the digital camera 110. Therefore, the extended area is shaded while the digital camera 110 is capturing an image, but is not shaded at other times. Additionally or alternatively, the display module 106 may determine the timing of shading the extended area in other ways, such as when the imager of the digital camera 110 is active (e.g., powered on).

[0151] Figure 11 Another example client device 1100 is shown. Client device 1100 includes a display 1102 that displays an image (e.g., an autumn scene as shown). The extended region 1104 shown is shielded so that light that might interfere with capturing the image will not be emitted from the extended region 1104. The extended region 1104 corresponds to the field of view of camera 110 and an additional portion of the peripheral region surrounding the field of view of camera 110. As shown, the extended region 1104 is larger than... Figure 10The extended area 1004. By using the extended area 1104 instead of the smaller camera area 1004, no light is emitted from the peripheral area surrounding the field of view, thus preventing any stray light that might be reflected or scattered within the display 1102 from interfering with the captured image.

[0152] return Figure 9 When camera area 112 is blocked from light, the display that selects to capture the image refers to deactivating or otherwise displaying the pixels in that area as black when the digital camera 110 captures an image but is not blocked from light at other times. This area can be, for example, camera area 112 or an extended area including camera area 112 and some peripheral areas surrounding camera area 112 as described above. No light is emitted from this area when it is blocked, therefore it does not emit light that interferes with the captured image. The light-blocking control module 906 is pre-programmed or otherwise knows the location of the area to be blocked. This area can be determined in the various ways described above regarding blocking camera area 112 or the extended area. The light-blocking control module 906 provides a screen light-blocking instruction 914 to the display module 106 to deactivate or otherwise display the pixels in the extended area as black.

[0153] The display module 106 is synchronized with the camera module 108 so that the light blocking of the area occurs simultaneously with the image capture by the digital camera 110, but the area is not blocked when the digital camera 110 is not capturing an image. For example, assuming that the imager of the digital camera 110 captures images at a rate of 30 frames per second (images) and the integration time is 15 milliseconds, the imager captures an image for 15 milliseconds (ms) every 33.33 milliseconds. Therefore, the area will be blocked for 15 ms, and then powered on or the image will be displayed for 18.33 ms.

[0154] Figure 12 An example 1200 of a gating display is shown. Client device displays of images (e.g., an autumn scene as shown) are shown at times 1202, 1204, 1206, and 1208. At time 1202, area 1210 is blocked from light while an image is captured by the imager of digital camera 110. As shown, the image is displayed in the peripheral area 114 of the display. At a later time 1204, the area is powered on and a portion of the image is displayed. At a later time 1206, area 1210 is blocked from light while an image is captured by the imager of digital camera 110. As shown, the image is displayed in the peripheral area 114 of the display. At a later time 1208, the area is powered on and a portion of the image is displayed. By gating the display, the area is blocked from light while the imager captures the image, and powered on and a portion of the image is displayed at other times. This reduces the amount of time the area is blocked from light, which reduces the visibility of the blocked area to the user, while mitigating interference from light emitted by the area during image capture.

[0155] In response to determining to reduce the brightness of display 104, adjustment determination module 902 transmits brightness adjustment instruction 916 to brightness control module 904, which determines when and how to reduce the brightness of display 104. In one or more embodiments, brightness adjustment instruction 916 includes an indication of ambient light level measurement 910, thereby allowing brightness control module 904 to determine the appropriate way to reduce the brightness of display 104. Additionally or alternatively, brightness control module 904 may reduce the brightness of display 104 in the same manner, regardless of ambient light level measurement 910.

[0156] In one or more embodiments, the brightness control module 904 reduces the brightness of the display 104 only in the camera area (or an extended area similar to the extended shading area described above). Additionally or alternatively, the brightness control module 904 reduces the brightness of the display 104 across the entire display. The brightness control module 904 can determine whether the brightness of the display 104 is reduced only in the camera area (or extended area) or across the entire display in various ways, such as based on ambient light level measurement 910 (e.g., if the ambient light level measurement 910 is below a threshold level, the brightness of the display 104 is reduced across the entire image; if the ambient light level measurement 910 at least reaches the threshold, the brightness of the display is reduced only in the camera or extended area). As another example, whether the brightness is reduced across the entire display or only in the camera area (or extended area) can be a user-selectable configuration setting, pre-configured in the brightness control module 940, etc.

[0157] Brightness control module 904 provides brightness setting instruction 918 to display module 106 to reduce the brightness of the entire display 104 or a specific portion of display 104. The amount of brightness reduction can be specified in various ways, such as a fixed amount (e.g., reducing brightness by 20% within a brightness setting range of 0 to 100) or a variable amount (e.g., reducing the current brightness setting by 20%). The amount of brightness reduction can also vary based on an ambient light level measurement; for example, the reduction amount is greater if the ambient light level measurement is below a threshold than if the ambient light level measurement is above a threshold. Brightness control module 904 is pre-programmed or otherwise knows the location of camera area 112 and can include the amount of brightness reduction and a specific portion or the entire display 104 (e.g., camera area 112 or extended area) in brightness setting instruction 918. Display module 106 reduces the brightness of that portion of the display by reducing the brightness of individual pixels within that portion.

[0158] In one or more embodiments, the display module 106 is synchronized with the camera module 108 such that the brightness reduction occurs simultaneously with the digital camera 110 capturing an image. Therefore, the brightness of a portion or all of the display 104 is reduced while the digital camera 110 is capturing an image, but not at other times. Additionally or alternatively, the display module 106 may determine the timing for when to otherwise reduce the brightness of a portion or all of the display 104, such as when the imager of the digital camera 110 is active (e.g., powered on).

[0159] Figure 13 Another example client device 1300 is shown. Client device 1300 includes a display 1302 that displays an image (e.g., an autumn scene shown). A region 1304 with reduced brightness is shown, thus reducing light from region 1304 that might interfere with image capture. In one or more embodiments, region 1304 is a camera region 112 corresponding to the field of view of camera 110. Additionally or alternatively, region 1304 is an extended region that blocks light from a portion of display 104 as discussed above, although in Figure 13 In this part 1304, the brightness is reduced rather than blocked.

[0160] Figure 14 Another example client device 1400 is shown. Client device 1400 includes a display 1402 that displays an image (e.g., an autumn scene as shown). Display 1402 is shown with reduced brightness (e.g., relative to the image). Figure 13 The peripheral area 114 of the display 1302 will reduce the light from the display 1402 that may interfere with the capture of the image.

[0161] return Figure 9 In some cases, a different type or model of display 104 absorbs more colors than other types, effectively reducing the amount of certain colors emitted by that particular type or model of display 104 and captured by the digital camera 110. In this situation, the brightness control module 904 either increases or does not change the brightness of the colors absorbed more by display 104, or decreases the brightness of the colors absorbed less by display 104. Therefore, the brightness control module 904 can increase or decrease the brightness of certain colors in the displayed image but not others.

[0162] Which color is absorbed more than other colors can be determined in any of a variety of different ways. In one or more embodiments, the amount of various colors absorbed by a particular type or model of display 104 is determined empirically by the developer or designer testing the particular type or model of display 104. This test may include, for example, illuminating the front of display 104 with light and measuring the amount of a particular color (e.g., primary colors or red, green, and blue) passing through the back of display 104. As another example, the test may include capturing calibration images (e.g., red, green, and blue calibration images) and measuring the brightness of the captured calibration images relative to each other to identify (based on known imager response information) which color(s) is absorbed more than one or more other colors. As yet another example, the transmittance profiles of the individual color channels of display 104 may be analyzed to identify which color channels absorb or transmit the most energy, and thus it may be easy to identify which colors are absorbed more by display 104.

[0163] Figure 15 Another example client device 1500 is shown. Client device 1500 includes a display 1502 that displays images (e.g., an autumn scene as shown). Figure 15 The example assumes that display 1502 absorbs more blue than red or green and less green than red or blue, thus showing display 1502 with reduced green brightness (e.g., relative to red or blue). Figure 4 The display 104 absorbs less light from captured images (402) and increases blue brightness. Therefore, the brightness of images in which the display 104 absorbs less light decreases, while the brightness of images in which the display 104 absorbs more light increases. This reduces interference from captured images of colors from which the display 104 absorbs less light, while helping to maintain the brightness of the display 104 without reducing the brightness of other colors.

[0164] return Figure 1 In one or more embodiments, the image displayed on display 104 while the digital camera is capturing an image is a known image, such as an image frequently displayed on display 104. For example, the displayed image could be an image from the lock screen of a client device displayed when the digital camera captures an image used to authenticate a user. In this case, display interference mitigation system 116 can perform additional processing or analysis to determine how to mitigate interference from display 104 when displaying a particular display image. This processing could include, for example, additional processing or analysis of the colors in the display image to determine an appropriate subtractive image, determine an appropriate reduction in brightness, and so on.

[0165] This article discusses various techniques for mitigating interference from the display on captured images. It should be understood that the display interference mitigation system 116 can use any one or a combination of these different techniques to reduce interference from the display on the captured image. For example, any technique for reducing the brightness of the display can be performed simultaneously with image capture, and a calibration image can be subtracted from the captured image to generate an enhanced output image. As another example, the brightness of a specific color on the display can be reduced while blocking light from the camera area or an extended area of ​​the display.

[0166] In one or more embodiments, as described above, the display interference mitigation system 116 shades or otherwise reduces the brightness of the camera area or extended area. In this case, while capturing a calibration image or recalibration image, the display interference mitigation system 116 optionally shades or otherwise reduces the brightness of the camera area or extended area. The display module 106 may reduce the brightness of the calibration image or recalibration image, or may generate the calibration image or recalibration image at a reduced brightness (e.g., displaying black in the camera area or extended area). This prevents the display interference mitigation system 116 from using the captured calibration image or captured recalibration image, which emits light in the area of ​​the display at a brightness that will not be used when subsequently capturing images.

[0167] It should be noted that this may result in multiple sets of captured calibration images or captured recalibrated images, each set corresponding to a specific shading or brightness reduction in the camera area or extended area, and the appropriate set of multiple sets is used to generate an enhanced output image based on the area and how to reduce the brightness of the area when capturing subsequent images.

[0168] Figure 16 Example processes 1600 for implementing the techniques discussed herein, according to one or more embodiments, are illustrated. Through methods such as… Figure 1 The display interference mitigation system 116 executes process 1600 and can be implemented in software, firmware, hardware, or a combination thereof. Process 1600 is shown as a set of actions and is not limited to the sequence of operations shown for performing the various actions.

[0169] In process 1600, an image is captured by a digital camera mounted below the illumination display (action 1602). The field of view of the digital camera extends through the illumination display, and the digital camera can capture an image while another image is displayed on the illumination display.

[0170] An enhanced output image is generated from the captured image by mitigating interference from the illumination display (Action 1604). As discussed herein, light emitted from the illumination display can be captured along with the captured image, but is mitigated to generate an enhanced output image.

[0171] The enhanced output image is stored, displayed, or transmitted (Action 1606). The enhanced output image can be stored locally, displayed on a lit monitor, communicated to or transmitted to another device or system, etc.

[0172] Figure 17 Another example process 1700 for implementing the techniques discussed herein, according to one or more embodiments, is shown. Process 1700 is performed by, for example... Figure 1 The display interference mitigation system 116 is executed by the display interference mitigation system and can be implemented in software, firmware, hardware, or a combination thereof. Process 1700 is shown as a set of actions and is not limited to the sequence of operations shown for performing the various actions.

[0173] In process 1700, a calibration image corresponding to each pixel type in the display is displayed (at 1702). For example, for a display that includes red, green, and blue pixels, a red calibration image is displayed, a green calibration image is displayed, and a blue calibration image is displayed. These calibration images will be displayed individually (e.g., the display shows full red, then full green, then full blue).

[0174] Each displayed calibration image is captured by a digital camera mounted below the monitor (Action 1704). The digital camera's field of view extends across the monitor, and the digital camera captures an image of the displayed calibration image.

[0175] A digital camera mounted below the monitor captures a first image while simultaneously displaying a second image on the monitor (Action 1706). After the calibration images have been displayed and captured in Actions 1702 and 1704, the second image is displayed and the first image is captured in turn.

[0176] For each captured calibration image, a subtraction image is generated based on the captured image and the second image (Action 1708). These subtraction images correspond to the interference from the display caused by displaying the second image while the first image is being captured.

[0177] An enhanced output image is generated from the first image and the subtracted image (Action 1710). By subtracting the subtracted image from the first image, interference from the lighting display showing the second image is mitigated when the first image is captured.

[0178] Figure 18 Another example process 1800 for implementing the techniques discussed herein is illustrated according to one or more embodiments. Process 1800 is performed by a display interference mitigation system including a recalibration module, such as including Figure 8 The recalibration module 800 Figure 1The display interference mitigation system 116 can be implemented in software, firmware, hardware, or a combination thereof. Process 1800 is shown as a set of actions, and is not limited to the sequence of operations shown for performing the various actions.

[0179] In process 1800, an ambient light level measurement is received from the client device (action 1802). The ambient light level measurement is received from any of various sensors (such as the ambient light sensor of the client device, the imager of the client device's digital camera, etc.).

[0180] The client device's display is activated in response to an ambient light level measurement (Action 1804). This display is an illuminated display, such as an OLED display. The display is activated in response to an ambient light level measurement indicating that the client device is in a low-light environment (e.g., below a threshold light level). If the display is already activated, it does not need to be reactivated in Action 1804.

[0181] When the display is activated, the client device's digital camera captures a recalibrated image (Action 1806). The digital camera is positioned behind the display, with its field of view extending across the display. The digital camera captures the recalibrated image being displayed on the display, thus obtaining the captured calibration image.

[0182] Receive a previously captured calibration image (Action 1808). The previously captured calibration image is, for example, a calibration image that was previously captured and stored in the digital storage medium of the client device.

[0183] A merged calibration image is generated based on the previously captured calibration image and recalibrated image (Action 1810). The merged calibration image can be generated in various ways, such as by replacing one or more portions of the calibration image with corresponding portions of the recalibrated image, mixing (e.g., averaging or otherwise combining) the corresponding pixel values ​​from one or more portions of the calibration image and one or more portions of the recalibrated image, etc.

[0184] The merged calibration image is stored in a digital storage medium (Action 1812). In one or more embodiments, the merged calibration image replaces the previously captured calibration image received in Action 1808.

[0185] Actions 1806 to 1812 can be repeated for each pixel of multiple different pixel types in the display. For example, if the display contains red, green, and blue pixels, actions 1806 to 1812 are repeated for each of the red recalibrated image, the green recalibrated image, and the blue recalibrated image to obtain a merged calibration image of red, a merged calibration image of green, and a merged calibration image of blue.

[0186] Figure 19 Another example process 1900 for implementing the techniques discussed herein is illustrated according to one or more embodiments. Process 1900 is performed by a display interference mitigation system including a brightness adjustment module, such as including Figure 9 Brightness adjustment module 900 Figure 1 The display interference mitigation system 116 can be implemented in software, firmware, hardware, or a combination thereof. Process 1900 is shown as a set of actions, and is not limited to the sequence of operations shown for performing the various actions.

[0187] In process 1900, an ambient light level measurement is received from the client device (action 1902). The ambient light level measurement is received from any of various sensors, such as the ambient light sensor of the client device, the imager of the client device's digital camera, etc.

[0188] The first image is displayed on the illuminated display of the client device (Action 1904). The first image can be generated by any of the various applications, programs, or other modules or components of the client device.

[0189] The second image was captured by a digital camera mounted below the illuminated display (Action 1906). The digital camera had a field of view that extended through the illuminated display.

[0190] While capturing the second image, the brightness of the illumination display is controlled (Action 1908). The brightness can be controlled based on an ambient light level measurement. Such brightness control may include blocking a portion of the illumination display, reducing the brightness of one or more colors of the illumination display, and so on, as discussed above, to mitigate interference associated with the illumination display.

[0191] The image captured in action 1906 is stored in a digital storage medium and then displayed or transmitted (action 1908). The captured image can be selectively transmitted to any of a variety of other devices, systems, or modules.

[0192] Although this document discusses specific functions with reference to specific modules, it should be noted that the functions of the various modules discussed herein can be separated into multiple modules, or at least some functions of multiple modules can be combined into a single module. Furthermore, the specific module discussed herein for performing an action includes either the specific module itself performing the action, or the specific module calling or otherwise accessing another component or module that performs the action (or performing the action in conjunction with the specific module). Therefore, a specific module for performing an action includes either the specific module performing the action itself or another module called or otherwise accessed by the specific module performing the action.

[0193] Figure 20Various components of the example device 2000 are shown, in which aspects of the techniques described herein can be implemented. The example device 2000 can be implemented with reference to previous... Figure 1-19 Any device described, such as any type of wireless device, mobile phone, client device, companion device, pairing device, display device, tablet computer, computing, communication, entertainment, gaming, media playback, and / or any other type of computing and / or electronic device. For example, the client device 102 described above can be implemented as example device 2000.

[0194] Device 2000 includes a communication transceiver 2002 that enables wired and / or wireless communication of device data 2004 with other devices. Device data 2004 may include any type of audio, video, and / or image data. Example communication transceivers 2002 include those compliant with various IEEE 1002.15 (Bluetooth) standards. TM Wireless Personal Area Network (WPAN) radio conforming to the IEEE 1002.11 standard, and various WiFi standards. TM Wireless Local Area Network (WLAN) radios conforming to any of the IEEE 1002.16 standards, Wireless Wide Area Network (WWAN) radios for cellular telephone communications, and various IEEE 1002.16 (WiMAX) standards. TM Standard wireless metropolitan area network (WMAN) radios and wired local area network (LAN) Ethernet transceivers for network data communication.

[0195] Device 2000 may also include one or more data input ports 2006 through which any type of data, media content, and / or input can be received, such as user-selectable input from the device, messages, music, television content, recorded content, and any other type of audio, video, and / or image data received from any content and / or data source. Data input ports may include USB ports, coaxial cable ports, and other serial or parallel connectors (including internal connectors) for flash memory, DVDs, CDs, etc. These data input ports can be used to couple the device to any type of component, peripheral device, or accessory, such as a microphone and / or camera.

[0196] Device 2000 includes a processor system 2008 of one or more processors (e.g., any one of a microprocessor, controller, etc.) and / or a processor and memory system implemented as a system-on-a-chip (SoC) to process computer-executable instructions. The processor system may be implemented at least partially in hardware, which may include components of an integrated circuit or system-on-a-chip, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), and other implementations in silicon and / or other hardware. Alternatively or additionally, the device may be implemented using any or a combination of software, hardware, firmware, or fixed logic circuitry that incorporates processing and control circuitry, typically identified at 2010. Device 2000 may further include any type of system bus or other data and command transfer system coupling the various components within the device. The system bus may include any or a combination of different bus structures and architectures, as well as control and data lines.

[0197] Apparatus 2000 also includes a computer-readable storage memory 2012 (e.g., a memory device) that enables data storage, such as a data storage device accessible by a computing device, and provides persistent storage for data and executable instructions (e.g., software applications, programs, functions, etc.). Examples of computer-readable storage memory 2012 include volatile and non-volatile memory, fixed and removable media devices, and any suitable storage device or electronic data storage that maintains data accessible to a computing device. Computer-readable storage memory can include various implementations of random access memory (RAM), read-only memory (ROM), flash memory, and other types of storage media in various storage device configurations. Apparatus 2000 may also include mass storage media devices.

[0198] Computer-readable storage memory 2012 provides a data storage mechanism to store device data 2004, other types of information and / or data, and various device applications 2014 (e.g., software applications). For example, an operating system 2016 may be maintained as software instructions with memory devices and executed by a processor system 2008. Device applications may also include a device manager 2018, such as any form of control application, software application, signal processing and control module, device-specific native code, device-specific hardware abstraction layer, etc.

[0199] In this example, device 2000 includes a display interference mitigation system 2020, which further implements the use of a calibration image to mitigate display interference on the captured image. Examples of the display interference mitigation system 2020 include references... Figure 1 The description of the display interference mitigation system 116 and the above reference Figure 3The described display interference mitigation system 300 can be implemented to perform various aspects of the techniques described herein.

[0200] In this example, device 2000 also includes a camera 2022 and device sensors 2024, such as temperature sensors for monitoring the operating temperature of device components, temperature sensors for monitoring the external temperature of device 2000, and device sensors such as components implemented as inertial measurement units (IMUs). Sensor 2024 may include various additional sensors, such as proximity sensors, ambient light sensors, accelerometers, gyroscopes, etc. Device 2000 may also include one or more power sources 2026, for example, when the device is implemented as a wireless or collaborative device. The power source may include a charging and / or power system and may be implemented as a flexible strip battery, a rechargeable battery, a rechargeable supercapacitor, and / or any other type of active or passive power source.

[0201] Device 2000 may also include an audio and / or video processing system 2028, which generates audio data for audio system 2030 and / or display data for display system 2032. The audio system and / or display system may include any device that processes, displays, and / or otherwise presents audio, video, display, and / or image data. Display data and audio signals may be transmitted to the audio and / or display components via RF (radio frequency) links, S-video links, HDMI (High-Definition Multimedia Interface), composite video links, component video links, DVI (Digital Video Interface), analog audio connections, or other similar communication links, such as media data port 2034. In this implementation, the audio system and / or display system are integrated components of the example device. Alternatively, the audio system and / or display system are external peripheral components of the example device.

[0202] Although embodiments of techniques for mitigating display interference to captured images using calibration images have been described in language specific to features or methods, the subject matter of the appended claims is not intended to be limited to the specific features or methods described. Rather, specific features and methods are disclosed as example implementations of techniques for mitigating display interference to captured images using calibration images. Furthermore, various different embodiments are described, and it should be understood that each described embodiment can be implemented independently or in combination with one or more other described embodiments. Other aspects of the techniques, features, and / or methods discussed herein relate to one or more of the following:

[0203] An apparatus includes: an illumination display; a digital camera positioned below the illumination display, wherein the field of view of the digital camera passes through the illumination display; a display interference mitigation system coupled to the digital camera to generate an enhanced output image from an input image captured by the digital camera to mitigate interference from the illumination display; and a storage device for storing the enhanced output image.

[0204] As an alternative or supplement to the aforementioned device, any one or a combination of the following may be included. The display interference mitigation system generates the enhanced output image by subtracting a predetermined calibration image from the input image. The predetermined calibration image includes a captured calibration image generated from a calibration image displayed on the illumination display. The predetermined calibration image is captured when the device is in an environment with a light level below a threshold. The digital camera captures the input image while the illumination display is displaying a display image. The illumination display will display a red calibration image, a green calibration image, and a blue calibration image. The digital camera captures a red captured calibration image when displaying the red calibration image, a green captured calibration image when displaying the green calibration image, and a blue captured calibration image when displaying the blue calibration image; the illumination display will display the display image while capturing the input image. The display interference mitigation system generates a first subtraction image based on the captured red calibration image and the display image, a second subtraction image based on the captured green calibration image and the display image, and a third subtraction image based on the captured blue calibration image and the display image. The enhanced output image is generated by subtracting the first, second, and third subtraction images from the first image. Specifically, the display interference mitigation system determines the amounts of red, green, and blue in the region of interest (ROI) of the second image and generates first, second, and third multipliers corresponding to the amounts of red, green, and blue in the ROI, respectively. The first subtraction image is generated by multiplying each pixel in the captured red calibration image by the first multiplier; the second subtraction image is generated by multiplying each pixel in the captured green calibration image by the second multiplier; and the third subtraction image is generated by multiplying each pixel in the captured blue calibration image by the third multiplier. The ROI includes a portion of the display image displayed on the illuminated display within the field of view of the digital camera. Specifically, determining the amount of red in the region of interest (ROI) involves averaging the red values ​​of the pixels in the ROI; determining the amount of green in the ROI involves averaging the green values ​​of the pixels in the ROI; and determining the amount of blue in the ROI involves averaging the blue values ​​of the pixels in the ROI. Furthermore, determining the amounts of red, green, and blue in the ROI involves inputting the pixel values ​​in the ROI into a machine learning system trained to determine the amounts of red, green, and blue in the ROI, and receiving indications of the amounts of red, green, and blue in the ROI from the machine learning system.The display interference mitigation system generates the enhanced output image by: generating a color channel with multiple color channels for each of the multiple pixel types in the illumination display; determining a capture parameter ratio that takes into account the gain and integration time differences between a previously captured calibration image and the first image; generating a subtractive image for each color channel by multiplying the corresponding previously captured calibration image by the capture parameter ratio and by multiplying it by the color channel on a per-pixel basis; and subtracting the subtractive image from the first image.

[0205] A method includes: capturing a first image using a digital camera mounted below a lighting display in a client device, wherein the field of view of the digital camera passes through the lighting display; generating an enhanced output image from the first image by mitigating interference from the lighting display; and storing the enhanced output image.

[0206] As an alternative or supplement to the above method, any one or a combination of the following may be included. The method further includes displaying a second image on the illumination display, and the capturing includes capturing the image while the second image is displayed on the illumination display. The method further includes: displaying a red calibration image, a green calibration image, and a blue calibration image; capturing a red captured calibration image while displaying the red calibration image; capturing a green captured calibration image while displaying the green calibration image; capturing a blue captured calibration image while displaying the blue calibration image; generating a first subtraction image based on the red captured calibration image and the second image displayed on the illumination display while the first image is captured; generating a second subtraction image based on the green captured calibration image and the second image; generating a third subtraction image based on the blue captured calibration image and the second image; and generating an image by subtracting the first subtraction image, the second subtraction image, and the third subtraction image from the first image. The method further includes: determining the amounts of red, green, and blue in a region of interest (ROI) of the second image; generating first, second, and third multipliers corresponding to the amounts of red, green, and blue in the ROI, respectively; generating the first subtraction image by multiplying each pixel in the captured calibration image of red by the first multiplier; generating the second subtraction image by multiplying each pixel in the captured calibration image of green by the second multiplier; and generating the third subtraction image by multiplying each pixel in the captured calibration image of blue by the third multiplier. The ROI comprises a portion of the display image displayed on the illumination display within the field of view of the digital camera. Determining the amount of red in the ROI comprises averaging the red values ​​of the pixels in the ROI, determining the amount of green in the ROI comprises averaging the green values ​​of the pixels in the ROI, and determining the amount of blue in the ROI comprises averaging the blue values ​​of the pixels in the ROI. Generating the enhanced output image includes: generating color channels with multiple color channels for each of the multiple pixel types in the illumination display; determining a capture parameter ratio that takes into account the gain and integration time differences between a previously captured calibration image and the first image; generating a subtractive image for each color channel by multiplying the corresponding previously captured calibration image by the capture parameter ratio and by the color channel on a per-pixel basis; and subtracting the subtractive image from the first image. The method also includes performing the capture in an environment where the light level is less than a threshold amount.

[0207] A client device includes: a processor implemented in hardware; and a lighting display; a digital camera disposed below the lighting display, wherein the field of view of the digital camera passes through the lighting display; and a computer-readable storage medium storing a plurality of instructions responsive to execution of the processor, causing the processor to perform the following actions: capturing a first image through the digital camera; generating an enhanced output image from the first image by mitigating interference from the lighting display; and storing the enhanced output image.

Claims

1. A device for mitigating display interference, comprising: Lighting displays; A digital camera positioned below the illumination display, wherein the field of view of the digital camera passes through the illumination display; and Display interference mitigation system, the display interference mitigation system being coupled to the digital camera, to: Using the digital camera, a captured calibration image is captured while a calibration image of a specific color is being displayed on the illumination display, wherein the illumination display will display a red calibration image, a green calibration image, and a blue calibration image, and the digital camera will capture a red captured calibration image when the red calibration image is displayed, a green captured calibration image when the green calibration image is displayed, and a blue captured calibration image when the blue calibration image is displayed; Using the digital camera, an input image is captured while an image is being displayed on the illuminated display; The amounts of red, green, and blue in the region of interest of the displayed image are determined, and a first multiplier, a second multiplier, and a third multiplier corresponding to the amounts of red, green, and blue in the region of interest are generated, respectively. Each pixel in the captured calibration image of red is multiplied by the first multiplier to generate a first subtraction image; each pixel in the captured calibration image of green is multiplied by the second multiplier to generate a second subtraction image; and each pixel in the captured calibration image of blue is multiplied by the third multiplier to generate a third subtraction image. An enhanced output image is generated by subtracting the first subtraction image, the second subtraction image, and the third subtraction image from the input image to mitigate interference from the lighting display; and A storage device for storing the enhanced output image.

2. The device according to claim 1, wherein the captured calibration image is captured when the device is in an environment where the light level is less than a threshold amount.

3. The device of claim 1, wherein the region of interest includes a portion of the display image displayed on the illumination display within the field of view of the digital camera.

4. The device according to claim 3, wherein, Determining the amount of red in the region of interest involves averaging the red values ​​of the pixels in the region of interest; determining the amount of green in the region of interest involves averaging the green values ​​of the pixels in the region of interest; and determining the amount of blue in the region of interest involves averaging the blue values ​​of the pixels in the region of interest.

5. The device according to claim 3, wherein, Determining the amount of red, green, and blue in the region of interest involves inputting the pixel values ​​in the region of interest into a machine learning system trained to determine the amount of red, green, and blue in the region of interest, and receiving indications of the amount of red, green, and blue in the region of interest from the machine learning system.

6. The device according to claim 1, wherein, The specific color of the calibration image is one of red, green, and blue.

7. A method for mitigating display interference, comprising: A first calibration image is captured by a digital camera installed below a lighting display in a client device while a second calibration image of a specific color is being displayed on the lighting display, wherein the field of view of the digital camera extends through the lighting display; The digital camera captures an input image while an image is being displayed on the illumination display, wherein the illumination display will display a red calibration image, a green calibration image, and a blue calibration image, and the digital camera captures a red captured calibration image when the red calibration image is displayed, captures a green captured calibration image when the green calibration image is displayed, and captures a blue captured calibration image when the blue calibration image is displayed. The amounts of red, green, and blue in the region of interest of the displayed image are determined, and a first multiplier, a second multiplier, and a third multiplier corresponding to the amounts of red, green, and blue in the region of interest are generated, respectively. Each pixel in the captured calibration image of red is multiplied by the first multiplier to generate a first subtraction image; each pixel in the captured calibration image of green is multiplied by the second multiplier to generate a second subtraction image; and each pixel in the captured calibration image of blue is multiplied by the third multiplier to generate a third subtraction image. An enhanced output image is generated by subtracting the first subtraction image, the second subtraction image, and the third subtraction image from the input image; and Store the enhanced output image.

8. The method of claim 7, wherein the region of interest includes a portion of the display image displayed on the illumination display within the field of view of the digital camera.

9. The method of claim 8, wherein determining the amount of red in the region of interest comprises averaging the red values ​​of the pixels in the region of interest, determining the amount of green in the region of interest comprises averaging the green values ​​of the pixels in the region of interest, and determining the amount of blue in the region of interest comprises averaging the blue values ​​of the pixels in the region of interest.

10. The method according to claim 7, wherein, The first calibration image is captured in an environment where the light level is less than a threshold amount.

11. A client device for mitigating display interference, comprising: A processor implemented in hardware; Lighting displays; A digital camera positioned below the illumination display, wherein the field of view of the digital camera passes through the illumination display; and A computer-readable storage medium having stored thereon a plurality of instructions, the instructions being responsive to execution by the processor to cause the processor to perform the following actions, including: The first image is captured by the digital camera while the second image is displayed on the illumination display; For each of the multiple pixel types of the illumination display, a color channel is generated in a plurality of color channels, wherein the color channel is a grayscale image of the region of interest in the second image and includes a type of color of the display pixel; For each color channel, a capture parameter ratio is determined to capture the difference between a previously captured calibration image and the first image when the calibration image for a specific color is displayed; The subtractive image is generated by multiplying the previously captured calibration image by the capture parameter ratio of the corresponding color channel and multiplying it by the corresponding color channel on a per-pixel basis. An enhanced output image is generated by subtracting the subtracted image from the first image; and Store the enhanced output image.

12. The client device according to claim 11, wherein, After multiplying by the capture parameter ratio, the pixel value of each pixel in the previously captured calibration image is multiplied by the pixel value of the corresponding color channel.

13. The client device of claim 11, wherein the difference includes gain and integration time differences between the previously captured calibration image and the first image.

14. The client device of claim 11, wherein the region of interest is a portion of the second image displayed on the illumination display within the field of view of the digital camera.

Images