Systems and methods for imaging with multi-domain image sensor

Abstract

Imaging systems and techniques are described. An imaging system may receive image data from an image sensor having different sets of photodetectors sensitive to different electromagnetic (EM) frequency domains. The imaging system determines a first focus setting for focusing on a scene using the image sensor in a first domain based on image data of the scene received from the image sensor. The imaging system determines a second focus setting for focusing on the scene using the image sensor in the second domain based on the first focus setting and a difference between respective characteristics of the first domain and the second domain. The imaging system causes the image sensor to capture a first image of the scene in the first domain according to the first focus setting and capture a second image of the scene in the second domain according to the second focus setting.

Description

Technical Field

[0001] The present application relates to image capture and processing. More specifically, the present application relates to systems and methods using image sensors configured to capture different electromagnetic frequency domains (such as visible light and infrared), including reducing cross-domain contamination and generating domain-specific focus settings. Background Art

[0002] Many devices include one or more cameras. For example, a smartphone or tablet includes a front-facing camera for capturing selfie images and a rear-facing camera for capturing scene images (such as landscapes or other scenes of interest to the device user). The camera can capture images using an image sensor of the camera, which can include an array of photodetectors. The photodetectors can be sensitive to light from a given electromagnetic (EM) frequency domain (such as the visible light EM frequency domain), thereby allowing a corresponding image sensor to capture images in the given EM frequency domain. Summary of the invention

[0003] In some examples, systems and techniques for image processing are described. An imaging system may receive image data from an image sensor having different sets of photodetectors sensitive to different electromagnetic (EM) frequency domains. The imaging system determines a first focus setting for focusing on a scene using the image sensor in a first EM frequency domain based on image data of the scene received from the image sensor. The imaging system determines a second focus setting for focusing on the scene using the image sensor in a second EM frequency domain based on the first focus setting and a difference between respective characteristics of the first EM frequency domain and a second EM frequency domain. The imaging system causes the image sensor to capture a first image of the scene in the first EM frequency domain according to the first focus setting, and causes the image sensor to capture a second image of the scene in the second EM frequency domain according to the second focus setting.

[0004] According to at least one example, a method for determining one or more image settings is provided. The method includes: determining a first focus setting for focusing on the scene using the image sensor in a first electromagnetic (EM) frequency domain based on image data of a scene received from an image sensor; determining a second focus setting for focusing on the scene using the image sensor in a second EM frequency domain based on the first focus setting and a difference between respective characteristics of the first EM frequency domain and a second EM frequency domain; causing the image sensor to capture a first image of the scene in the first EM frequency domain according to the first focus setting; and causing the image sensor to capture a second image of the scene in the second EM frequency domain according to the second focus setting.

[0005] In another example, an apparatus for determining one or more image settings is provided, the apparatus comprising at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: determine a first focus setting for focusing on the scene using the image sensor in a first electromagnetic (EM) frequency domain based on image data of the scene received from the image sensor; determine a second focus setting for focusing on the scene using the image sensor in a second EM frequency domain based on the first focus setting and a difference between respective characteristics of the first EM frequency domain and a second EM frequency domain; cause the image sensor to capture a first image of the scene in the first EM frequency domain according to the first focus setting; and cause the image sensor to capture a second image of the scene in the second EM frequency domain according to the second focus setting.

[0006] In another example, a non-transitory computer-readable medium is provided, having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: determine a first focus setting for focusing on a scene in a first electromagnetic (EM) frequency domain using the image sensor based on image data of the scene received from the image sensor; determine a second focus setting for focusing on the scene in a second EM frequency domain using the image sensor based on the first focus setting and a difference between respective characteristics of the first EM frequency domain and a second EM frequency domain; cause the image sensor to capture a first image of the scene in the first EM frequency domain according to the first focus setting; and cause the image sensor to capture a second image of the scene in the second EM frequency domain according to the second focus setting.

[0007] In another example, an apparatus for determining one or more image settings is provided. The apparatus includes: means for determining a first focus setting for focusing on a scene in a first electromagnetic (EM) frequency domain using an image sensor based on image data of the scene received from the image sensor; means for determining a second focus setting for focusing on the scene in a second EM frequency domain using the image sensor based on the first focus setting and a difference between respective characteristics of the first and second EM frequency domains; means for causing the image sensor to capture a first image of the scene in the first EM frequency domain according to the first focus setting; and means for causing the image sensor to capture a second image of the scene in the second EM frequency domain according to the second focus setting.

[0008] In some aspects, the apparatus is part of and / or includes a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head mounted display (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smart phone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or a component of a vehicle, another device, or a combination thereof. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0010] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Exemplary aspects of the present application are described in detail below with reference to the following drawings:

[0012] Figure 1 is a block diagram illustrating an example architecture of an image capture and processing system according to some examples;

[0013] Figure 2 is a block diagram illustrating an example architecture of an imaging system using a multi-domain image sensor to perform an imaging process according to some examples;

[0014] Figure 3A is a perspective view illustrating a head mounted display (HMD) used as part of an imaging system according to some examples;

[0015] Figure 3B is an example based on some examples Figure 3A A perspective view of a head mounted display (HMD) being worn by a user;

[0016] Figure 4Ais a perspective view illustrating a front surface of a mobile phone including a front-facing camera and usable as part of an imaging system according to some examples;

[0017] Figure 4B is a perspective view illustrating a rear surface of a mobile phone including a rear camera and usable as part of an imaging system according to some examples;

[0018] Figure 5 is a conceptual diagram illustrating a photodetector array of a multi-domain image sensor according to some examples;

[0019] Figure 6 is a block diagram illustrating an example architecture of an imaging system that uses a multi-domain image sensor sensitive to a first electromagnetic (EM) frequency domain and a second EM frequency domain to perform an imaging process according to some examples;

[0020] Figure 7 is a flow chart illustrating an imaging process performed by an imaging system having a multi-domain image sensor sensitive to a first domain and a second domain according to some examples;

[0021] Figure 8 is a flow chart illustrating an imaging process performed by an imaging system having a multi-domain image sensor sensitive to a first domain and a second domain according to some examples;

[0022] Fig. 9 is a flow chart illustrating an imaging process according to some examples; and

[0023] Fig.10 is a diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0024] Certain aspects of the present disclosure are provided below. Some of these aspects may be applied independently, and some of them may be applied in combination, which will be apparent to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it will be apparent that various aspects may be practiced without these specific details. Each of the drawings and descriptions is not intended to be restrictive.

[0025] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide a description that can be used to implement the example aspects to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0026] A camera is a device that uses an image sensor to receive light and capture an image frame (such as a still image or a video frame). The terms "image", "image frame" and "frame" are used interchangeably herein. A camera may be configured with various image capture and image processing settings. Different settings produce images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, aperture number, shutter speed, focus and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters may be applied to an image sensor used to capture one or more image frames. Other camera settings may configure post-processing of one or more image frames, such as changes in contrast, brightness, saturation, sharpness, level, curve or color. For example, settings or parameters may be applied to a processor (e.g., an image signal processor or ISP) used to process one or more image frames captured by an image sensor.

[0027] The camera can capture images using an image sensor of the camera, which can include an array of photodetectors. The photodetectors can be sensitive to light from a given electromagnetic (EM) frequency domain, such as the visible light EM frequency domain, thereby allowing the corresponding image sensor to capture images in the given EM frequency domain.

[0028] In some examples, systems and techniques for image processing are described. An imaging system may receive image data from an image sensor having different sets of photodetectors sensitive to different electromagnetic (EM) frequency domains. The imaging system determines a first focus setting for focusing on a scene using the image sensor in a first EM frequency domain based on image data of the scene received from the image sensor. The imaging system determines a second focus setting for focusing on the scene using the image sensor in a second EM frequency domain based on the first focus setting and a difference between respective characteristics of the first EM frequency domain and a second EM frequency domain. The imaging system causes the image sensor to capture a first image of the scene in the first EM frequency domain according to the first focus setting, and causes the image sensor to capture a second image of the scene in the second EM frequency domain according to the second focus setting.

[0029] The imaging systems and techniques described herein provide many technical improvements over existing imaging systems. For example, the imaging systems and techniques described herein improve image quality for systems having multi-domain image sensors because both cross-domain contamination reduction and per-domain focus settings can improve image quality. For example, the imaging system performing cross-domain contamination reduction can reduce or eliminate visual artifacts in images captured in each EM frequency domain that are caused by image data from other EM frequency domains to which the image sensor is sensitive. The imaging system generating per-domain focus settings allows the imaging system to account for differences in the characteristics of different EM frequency domains and generate separate focus settings for each EM frequency domain, thereby improving focus for the corresponding images captured in each EM frequency domain.

[0030] Various aspects of the application will be described with respect to the accompanying drawings. Figure 1 1 is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing images of one or more scenes (e.g., images of scene 110). The image capture and processing system 100 may capture independent images (or photographs) and / or may capture a video including multiple images (or video frames) in a particular sequence. A lens 115 of the system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the light toward the image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130. In some examples, the scene 110 is a scene in an environment. In some examples, the scene 110 is a scene of at least a portion of a user. For example, the scene 110 may be a scene of one or both of the user's eyes and / or at least a portion of the user's face.

[0031] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include multiple mechanisms and components; for example, the control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include additional control mechanisms other than those illustrated, such as control mechanisms to control analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0032] A focus control mechanism 125B of the control mechanism 120 may obtain a focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B may adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B may move the lens 115 closer to the image sensor 130 or further away from the image sensor 130 by actuating a motor or a servo, thereby adjusting the focus. In some cases, additional lenses may be included in the system 100, such as one or more micro lenses above each photodiode of the image sensor 130, each of which bends the light received from the lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting.

[0033] The exposure control mechanism 125A of the control mechanism 120 may obtain an exposure setting. In some cases, the exposure control mechanism 125A stores the exposure setting in a memory register. Based on the exposure setting, the exposure control mechanism 125A may control the size of the aperture (e.g., the aperture size or the number of apertures), the duration of the aperture opening (e.g., the exposure time or the shutter speed), the sensitivity of the image sensor 130 (e.g., the ISO speed or the film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0034] A zoom control mechanism 125C of the control mechanism 120 may obtain a zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C may control the focal length of an assembly of lens elements (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C may control the focal length of the lens assembly by actuating one or more motors or servos to move one or more lenses relative to each other. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (which may be the lens 115 in some cases) that first receives light from the scene 110, wherein the light then passes through an afocal zoom system between the focusing lens (e.g., the lens 115) and the image sensor 130 before the light reaches the image sensor 130. The afocal zoom system may, in some cases, include two positive (e.g., converging, convex) lenses having equal or similar focal lengths (e.g., within a threshold difference), with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses.

[0035] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image generated by the image sensor 130. In some cases, different photodiodes may be covered by different color filters and thus may measure light that matches the color of the color filter covering the photodiode. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, where each pixel of an image is generated based on red light data from at least one photodiode covered in a red filter, blue light data from at least one photodiode covered in a blue filter, and green light data from at least one photodiode covered in a green filter. Other types of color filters may use yellow, magenta, and / or cyan (also known as "emerald") filters instead of or in addition to red, blue, and / or green filters. Some image sensors may have no color filters at all and may instead use different photodiodes (in some cases vertically stacked) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, thereby responding to different wavelengths of light.Monochrome image sensors may also lack color filters, and therefore lack color depth.

[0036] In some cases, image sensor 130 may alternatively or additionally include an opaque mask and / or a reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which may be used for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier for amplifying an analog signal output by a photodiode and / or an analog-to-digital converter (ADC) for converting an analog signal output by a photodiode (and / or an analog signal amplified by an analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more of control mechanisms 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0037] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more processors of any other type of processor 1010 discussed with respect to the computing system 1000. The host processor 152 may be a digital signal processor (DSP) and / or other type of processor. In some implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth TM, global positioning system (GPS), etc.), any combination thereof, and / or other components. The I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0038] The image processor 150 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 and / or 1020, read-only memory (ROM) 145 and / or 1025, cache, memory unit, another storage device, or some combination thereof.

[0039] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touch pad, a touch-sensitive surface, a printer, any other output device 1035, any other input device 1045, or some combination thereof. In some cases, subtitles may be input into the image processing device 105B through a physical keyboard or keypad of the I / O device 160, or through a virtual keyboard or keypad of a touch screen of the I / O device 160. The I / O 160 may include one or more ports, jacks, or other connectors that enable a wired connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. The I / O 160 may include one or more wireless transceivers that enable a wireless connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 160 discussed previously, and may themselves be considered I / O devices 160 once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0040] In some cases, the image capture and processing system 100 may be a single device. In some cases, the image capture and processing system 100 may be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0041] like Figure 1 As shown, the vertical dotted line will Figure 1 1 is divided into two parts, representing image capture device 105A and image processing device 105B. Image capture device 105A includes lens 115, control mechanism 120 and image sensor 130. Image processing device 105B includes image processor 150 (including ISP 154 and host processor 152), RAM 140, ROM 145 and I / O 160. In some cases, some components illustrated in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0042] The image capture and processing system 100 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communications (such as cellular network communications, 1002.11 wi-fi communications, wireless local area network (WLAN) communications, or some combination thereof). In some specific implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile phone, a desktop computer, or other computing devices.

[0043] Although the image capture and processing system 100 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 100 may include more than Figure 1 . The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 100 may include electronic circuits or other electronic hardware, and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 100.

[0044] Figure 22 is a block diagram illustrating an example architecture of an imaging system 200 that performs an imaging process using a multi-domain image sensor 205. The imaging system 200 may include at least one of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the HMD 310, the mobile phone 410, the multi-domain image sensor 500, the imaging system 600, the imaging system that performs the imaging process 700, the imaging system that performs the imaging process 800, the imaging system that performs the imaging process 900, the computing system 1000, or a combination thereof. In some examples, the imaging system 200 may include, for example, one or more laptop computers, phones, tablet computers, mobile phones, video game consoles, vehicle computers, desktop computers, wearable devices, televisions, media centers, extended reality (XR) systems, virtual reality (VR) systems, augmented reality (AR) systems, mixed reality (MR) systems, head mounted display (HMD) devices, other types of computing devices discussed herein, or a combination thereof.

[0045] The imaging system 200 includes a multi-domain image sensor 205 that captures multi-domain image data 210. Examples of the multi-domain image sensor 205 include the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image sensor 130, an image sensor of any of the cameras 330A-330D, an image sensor of any of the cameras 430A-430D, the multi-domain image sensor 500, an image sensor that captures dual-domain raw image data 605, an image sensor of the multi-domain camera 740 of the imaging process 700, an image sensor that captures an image of the imaging process 800, an image sensor of the imaging process 900, an image sensor of the input device 1045, or a combination thereof. In some examples, the multi-domain image data 210 includes raw image data, image data, pixel data, image frames, raw video data, video data, video frames, or a combination thereof.

[0046] A first portion of the multi-domain image sensor 205 is sensitive to a first electromagnetic (EM) frequency domain 255. A second portion of the multi-domain image sensor 205 is sensitive to a second EM frequency domain 265. In some examples, a third portion of the multi-domain image sensor 205 is sensitive to a third EM frequency domain, and so on, where different portions of the image sensor are sensitive to any number of different EM frequency domains. The different EM frequency domains may include, for example, a radio EM frequency domain, a microwave EM frequency domain, an infrared (IR) EM frequency domain, a visible light (VL) EM frequency domain, an ultraviolet (UV) EM frequency domain, an X-ray EM frequency domain, a gamma ray EM frequency domain, a subset of any of these, or a combination thereof. A given portion of the multi-domain image sensor 205 may be sensitive to any of these listed EM frequency domains, a subset thereof, and / or a combination thereof. A subset of EM frequency domains may include, for example, different colors of visible light (e.g., red, blue, green), a frequency band within a given EM frequency domain, a frequency band spanning at least a portion of two or more EM frequency domains, or a combination thereof. For example, in some examples, the VL EM frequency domain may include red (R), green (G), blue (B), or a combination thereof. In some examples, the IR EM frequency domain may include near infrared (NIR), mid infrared (MIR), far infrared (FIR), or a combination thereof.

[0047] The multi-domain image sensor 205 includes at least one photodetector array. In some examples, the photodetectors in the array may be covered by at least one filter that may control which EM frequency domain(s) (and / or subsets and / or combinations thereof) reach the photodetector, effectively controlling which EM frequency domain(s) (and / or subsets and / or combinations thereof) the photodetector is sensitive to. In some examples, different photodetectors in the array may be covered by different filters. In some examples, different photodetectors in the array may be sensitive to different EM frequency domains (and / or subsets and / or combinations thereof), for example, based on the use of filters, based on the characteristics of the photodetectors themselves, or a combination thereof. For example, in some examples, the multi-domain image sensor 205 includes a first set of photodetectors that is sensitive to a first EM frequency domain 255 and a second set of photodetectors that is sensitive to a second EM frequency domain 265. In some examples, additionally, the multi-domain image sensor 205 includes a third set of photodetectors that is sensitive to a third EM frequency domain, and so on, where different sets of photodetectors of the image sensor are sensitive to any number of different EM frequency domains.

[0048] In the illustrative example, the multi-domain image sensor 205 includes a first portion (e.g., a first set of photodetectors) that is sensitive to the visible EM frequency domain and a second portion (e.g., a second set of photodetectors) that is sensitive to the IR EM frequency domain. The first portion (e.g., the first set of photodetectors) may be further divided into sub-portions (e.g., a subset of the first set of photodetectors) that are sensitive to different colors (e.g., red, green, and / or blue) in the visible EM frequency domain. A diagram representing the multi-domain image sensor 205 is shown in FIG. Figure 2 , and illustrates a camera capturing a photograph of a scene in which there are two people in a room near a laptop computer on a table. An exemplary portion of the photodetector array of the camera's multi-domain image sensor 205 is illustrated as a grid, wherein each cell in the grid represents a photodetector that is sensitive to a particular EM frequency domain (or subset thereof) marked on the cell. Cells labeled "R" are sensitive to red light from the visible light EM frequency domain. Cells labeled "G" are sensitive to green light from the visible light EM frequency domain. Cells labeled "B" are sensitive to blue light from the visible light EM frequency domain. Cells labeled "I" are sensitive to infrared signals from the infrared EM frequency domain. The graphic also includes an icon of an eye representing the visible light EM frequency domain and an icon of a thermometer representing the infrared light EM frequency domain. In Figure 2 2, a graphic representing raw sensor data 210 illustrates a representation of an image of a scene depicted in a graphic representing multi-domain image sensor 205 and a grid representing an array of photodetectors of multi-domain image sensor 205 to indicate that multi-domain image data 210 may include raw image data that has not been demosaiced. It should be understood that this illustrative example is exemplary and not limiting. For example, multi-domain image sensor 205 may include a different arrangement of photodetectors. Multi-domain image sensor 205 may be sensitive to different sets of EM frequency domains.

[0049] In some examples, the multi-domain image sensor 205 may be oriented toward a user (e.g., may face the user), and thus may capture (e.g., depict or otherwise represent) sensor data (e.g., image data) of at least a portion of the user. In some examples, the multi-domain image sensor 205 may be oriented away from the user (e.g., may face away from the user) and / or toward an environment in which the user is located, and thus may capture (e.g., depict or otherwise represent) sensor data (e.g., image data) of at least a portion of the environment. In some examples, the multi-domain image data 210 captured by the multi-domain image sensor 205 is oriented away from the user and / or toward the user. In some examples, the multi-domain image data 210 captured by the multi-domain image sensor 205 may have a field of view (FoV) that includes, is included in, overlaps with, and / or otherwise corresponds to the FoV of the user's eyes.

[0050] In some examples, in addition to the multi-domain image sensor 205, the imaging system 200 may include one or more other sensors, such as one or more other cameras, other image sensors, microphones, heart rate monitors, oximeters, biometric sensors, positioning receivers, global navigation satellite system (GNSS) receivers, inertial measurement units (IMUs), accelerometers, gyroscopes, gyrometers, barometers, thermometers, altimeters, depth sensors, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, time of flight (ToF) sensors, structured light sensors, other sensors discussed herein, or combinations thereof. In some examples, the one or more sensors 205 include at least one input device 1145 of the computing system 1100. In some implementations, one or more of these additional sensors may supplement or refine the sensor readings from the multi-domain image sensor 205. For example, an inertial measurement unit (IMU), accelerometer, gyroscope, or other sensor may be used to identify the posture (e.g., position and / or orientation) and / or movement and / or acceleration of the imaging system 200 and / or the user in the environment, which the imaging system 200 may use to reduce motion blur, rotational blur, or a combination thereof.

[0051] The imaging system 200 passes the multi-domain image data 210 from the multi-domain image sensor 205 to an image processor 215. The image processor 215 may include the image processing device 105B, the image processor 150, the host processor 152, the ISP 154, the single-domain re-mosaic engine 220, the focus setting generator 235, the computing system 1000, the processor 1010, or a combination thereof. The image processor 215 may demosaic the multi-domain image data 210 and / or perform pixel interpolation. The image processor 215 may perform other image processing operations, such as adjusting the brightness, saturation, noise reduction, clarity, contrast, sharpness, white balance, black balance, and / or other properties of the multi-domain image data 210. In some examples, the image processor 215 may demosaic the multi-domain image data 210 and / or perform pixel interpolation to generate an image based on only a subset of the multi-domain image data 210 corresponding to a photodetector sensitive to a specified EM frequency domain.

[0052] In an illustrative example, the image processor 215 may demosaic the multi-domain image data 210 using only image data from photodetectors sensitive to the visible EM frequency domain, ignoring or skipping image data from photodetectors sensitive to the IR EM frequency domain. The image processor 215 may use pixel interpolation to fill gaps in the image that are caused by ignoring or skipping image data from photodetectors sensitive to the IR EM frequency domain. In another illustrative example, the image processor 215 may demosaic the multi-domain image data 210 using only image data from photodetectors sensitive to the IR EM frequency domain, ignoring or skipping image data from photodetectors sensitive to the visible EM frequency domain. In some examples, the image processor 215 may use pixel interpolation to fill gaps in the image that are caused by ignoring or skipping image data from photodetectors sensitive to the visible EM frequency domain.

[0053] In some cases, as discussed above, when the image processor 215 is used to process the multi-domain image data 210 to use only image data from the first EM frequency domain 255 to which the multi-domain image sensor 205 is sensitive, the resulting image generated by the image processor 215 may include image artifacts. These image artifacts may be caused by contamination from the second EM frequency domain 265 to which the multi-domain image sensor 205 is also sensitive. These visual artifacts may include false colors and / or chromatic aberrations, and may appear particularly at or around sharp edges in the image. Such contamination and / or these image artifacts may be referred to as cross-domain contamination. The image processor 215 may perform cross-domain contamination reduction on the multi-domain image data 210 to reduce the cross-domain contamination. The cross-domain contamination reduction may reduce the contamination of image data from one EM frequency domain captured by the multi-domain image sensor 205 with image data from another EM frequency domain captured by the multi-domain image sensor 205. In some examples, cross-domain contamination reduction may entail multiplying pixel data values ​​in the contaminating EM frequency domain at a location or region of the image by a constant and subtracting the product from pixel data values ​​in the target EM frequency domain at that location or region of the image.

[0054] In an illustrative example, the image processor 215 may generate a visible light image from the multi-domain image data 210 by skipping the infrared photodetector data from the multi-domain image data 210 while performing a demosaicing process as discussed above. The image processor 215 may use pixel interpolation to fill in the gaps as discussed above. The visible light image generated by the image processor 215 from the multi-domain image data 210 may still have cross-domain contamination from the IR image data. To reduce the cross-domain contamination, the image processor 215 uses the multi-domain image data 210 to perform cross-domain contamination reduction on the visible light image.

[0055] Each IR-sensitive photodetector in the multi-domain image sensor 205 may be adjacent to at least one VL-sensitive photodetector. Pixel data values ​​from a given IR-sensitive photodetector may be multiplied by a constant, and the product may be subtracted from pixel data values ​​from VL-sensitive photodetectors adjacent to the IR-sensitive photodetector. Different constants may be used for different colors or subsets of the EM frequency domain. For example, pixel data values ​​from a given IR-sensitive photodetector may be multiplied by a first constant a, and the product may be subtracted from pixel data values ​​from red-sensitive photodetectors adjacent to the IR-sensitive photodetector. Pixel data values ​​from a given IR-sensitive photodetector may be multiplied by a second constant b, and the product may be subtracted from pixel data values ​​from green-sensitive photodetectors adjacent to the IR-sensitive photodetector. Pixel data values ​​from a given IR-sensitive photodetector may be multiplied by a third constant c, and the product may be subtracted from pixel data values ​​from blue-sensitive photodetectors adjacent to the IR-sensitive photodetector.

[0056] In some examples, the average of the pixel data from multiple IR-sensitive photodetectors in the area with the VL-sensitive photodetector multiplied by an appropriate constant may be subtracted from the pixel data from the VL-sensitive photodetector. The average may be a weighted average based on the distance from the VL-sensitive photodetector to the corresponding IR-sensitive photodetector. For example, in the illustrative example, the average of the pixel data from the four IR-sensitive photodetectors closest to a given VL-sensitive photodetector may be multiplied by an appropriate constant and subtracted from the pixel data from the VL-sensitive photodetector.

[0057] In some examples, the average of the pixel data from one or more IR-sensitive photodetectors from multiple image frames in an area with a VL-sensitive photodetector at a given frame multiplied by an appropriate constant may be subtracted from the pixel data from the VL-sensitive photodetector. The average may be a weighted average based on how far apart in time the frames with the IR-sensitive photodetector are from the frames with the VL-sensitive photodetector and / or the distance as discussed above. For example, in an illustrative example, the average of the pixel data from the IR-sensitive photodetector that is closest to a given VL-sensitive photodetector from a given frame and a previous frame may be multiplied by an appropriate constant and subtracted from the pixel data from the VL-sensitive photodetector of the given frame.

[0058] Figure 6 Additional examples of cross-domain contamination reduction are further illustrated and discussed in The image processor 215 may output a single-domain image 225 , which may be processed using demosaicing, pixel interpolation, and / or cross-domain contamination reduction as discussed above.

[0059] A diagram showing the image processor 215 is shown in FIG. Figure 2 2 and illustrates that the image processor 215 converts the multi-domain image data 210 (represented by the photodetector array) into a single-domain image by demosaicing and / or pixel interpolation as discussed above. The single-domain image is illustrated as an image of a scene captured by the multi-domain image sensor 205 in a single EM frequency domain as represented by the eye icon. The figure also illustrates that the image processor 215 performs cross-domain contamination reduction on the single-domain image to generate a processed image, which is illustrated as looking similar to the single-domain image, but with a flash pattern representing cross-domain contamination reduction. The processed image can be the single-domain image 225.

[0060] A graphic representation of a single domain image 225 is shown in Figure 2 2 and 3. The multi-domain image sensor 205 is illustrated in FIG. 2 and illustrates an image of a scene depicted as being captured by the multi-domain image sensor 205 in a single EM frequency domain as represented by an eye icon. The eye icon represents the visible light EM frequency domain. The single domain may be the visible light EM frequency domain, and the contaminated EM frequency domain may be the IR EM frequency domain, as discussed in some of the illustrative examples discussed above. The single domain may be the IR EM frequency domain, and the contaminated EM frequency domain may be the visible EM frequency domain, as discussed in some of the other illustrative examples discussed above.

[0061] In some examples, the imaging system 200 includes a single-domain re-mosaicing engine 220. The single-domain re-mosaicing engine 220 may include the image processing device 105B, the image processor 150, the host processor 152, the ISP 154, the image processor 215, the focus setting generator 235, the computing system 1000, the processor 1010, or a combination thereof. The single-domain re-mosaicing engine 220 may re-mosaic (e.g., such as processing using demosaicing, pixel interpolation, and / or cross-domain contamination reduction) an image generated using the image processor 215 (e.g., a single-domain image 225) according to a photodetector array arrangement associated with a single EM frequency domain. The output of the single-domain re-mosaicing engine 220 may simulate raw image data captured by a single-domain image sensor. The photodetector array arrangement associated with a single EM frequency domain may be a photodetector array arrangement that may be used in a single-domain image sensor that is sensitive to a single EM frequency domain. For example, if the single EM frequency domain is a visible light EM frequency domain, the single domain re-mosaicing engine 220 may re-mosaic the image generated using the image processor 215 according to an arrangement of photodetectors that are respectively sensitive to red, green, and blue (e.g., a Bayer filter arrangement). In some examples, the output of the single domain re-mosaicing engine 220 is a single domain image 225.

[0062] A graphic representation of a single domain re-mosaic engine 220 is shown in Figure 2, and illustrates a conversion from a multi-domain photodetector array arrangement (e.g., having VL-sensitive photodetectors and IR-sensitive photodetectors) to a single-domain photodetector array arrangement (e.g., having only VL-sensitive photodetectors and no IR-sensitive photodetectors).

[0063] The imaging system 200 also includes a focus setting generator 235. The focus setting generator 235 may include the image processing device 105B, the image processor 150, the host processor 152, the ISP 154, the image processor 215, the single domain re-mosaic engine 220, the computing system 1000, the processor 1010, or a combination thereof. The focus setting generator 235 may determine focus settings for different EM frequency domains.

[0064] The focus setting generator 235 solves a problem for the multi-domain image sensor 205 in that the multi-domain image sensor 205 is not always able to focus optimally on an object in two different EM frequency domains because different EM radiation having different EM frequency domains refracts slightly differently. Such differences in refraction are sufficient to produce significant focus differences between different EM frequency domains, such as the visible EM frequency domain and the infrared EM frequency domain. For example, for an object closer than the hyperfocal distance, adjusting the lens to bring the visible EM frequency domain channel into focus may cause the infrared EM frequency domain channel to be defocused and blurred, and vice versa. Demultiplexing of image data from different EM frequency domains may cause the defocus blur to shift from one EM frequency domain to another EM frequency domain.

[0065] The focus setting generator 235 may use an autofocus process to determine a focus setting for a first EM frequency domain 255 of the plurality of EM frequency domains to which the multi-domain image sensor 205 is sensitive. The autofocus process used by the focus setting generator 235 may include contrast detection autofocus (CDAF), phase detection autofocus (PDAF), a hybrid autofocus (HAF) system using a combination of PDAF and CDAF, or a combination thereof. The focus setting generator 235 may perform autofocus based on input data, which may include the multi-domain image data 210 captured by the multi-domain image sensor 205, a single-domain image (e.g., single-domain image 225) generated using the image processor 215, a single-domain image (e.g., single-domain image 225) generated using the single-domain re-mosaic engine 220, information about a focus setting used during capture of the multi-domain image data 210 from the focus mechanism 240, or a combination thereof. To generate focus settings specific to the first EM frequency domain 255, the focus settings generator 235 may use image data specific to the first EM frequency domain 255 from the multi-domain image data 210, and may exclude image data specific to other EM frequency domains from the multi-domain image data 210 while performing an autofocus analysis of the image data as discussed below.

[0066] To perform CDAF, the focus setting generator 235 actuates the focus mechanism 240 to move the lens (e.g., lens 115) of the imaging system 200 through a range of lens positions. The focus mechanism 240 may include a linear actuator, such as a voice coil motor (VCM). The focus mechanism 240 may be an example of one or more focus control mechanisms 125B. In some examples, the focus setting generator 235 may use a pre-specified distance interval between each tested lens position. Between and / or during these lens movements, the focus setting generator 235 causes the multi-domain image sensor 205 to capture image frames, which the focus setting generator 235 analyzes to determine the contrast between pixels in the first EM frequency domain 255 of the image frames (e.g., the contrast between pixels representing an object and pixels representing a background). Ultimately, the focus setting generator 235 identifies one of the image frames at which the contrast between pixels of the object and pixels of the background is maximized. The lens position at which the contrast is maximized in the first EM frequency domain 255 is determined to be the lens position at which the multi-domain image sensor 205 is best focused on the object in the first EM frequency domain 255.

[0067] To perform PDAF, the focus setting generator 235 obtains phase detection (PD) data from a dedicated phase detection (PD) photodetector in the photodetector that is sensitive to the first EM frequency domain 255. The PD photodetector (e.g., via an opaque pixel aperture mask and / or a portion of a larger lens covering multiple photodetectors) is configured to receive light only from certain angles. The data from the PD photodetector is analyzed by the focus setting generator 235 to identify whether the light of the first EM frequency domain 255 received by the multi-domain image sensor 205 from different angles converges to produce a focused image that is "in phase" or fails to converge and thus produces a blurred image that is "out of phase". If the light received from different angles is out of phase, the focus setting generator 235 identifies the direction in which the light is out of phase to determine whether the lens needs to be moved forward or backward, and identifies the phase difference indicating how out of phase the light is to determine how far the lens (e.g., lens 115) is to be moved. The focus setting generator 235 then transmits an actuation signal to the focus mechanism 240 through a connector to move the lens (e.g., lens 115) to a lens position corresponding to optimal focus. In some cases, a hybrid autofocus (HAF) system is used that uses both PDAF and CDAF. For example, lens movement guided by PDAF may still sometimes produce imperfect focus, and CDAF may be used after PDAF within a small range of lens positions to further optimize focus and compensate for any errors or inaccuracies in PDAF.

[0068] As discussed above, the focus setting generator 235 may use PDAF, CDAF, HAF, or a combination thereof to generate the first focus setting 250 for the first EM frequency domain 255. In some examples, the focus setting generator 235 may generate the second focus setting 260 for the second EM frequency domain 265 based on the first focus setting 250 for the first EM frequency domain 255 and based on a difference in respective characteristics of the first EM frequency domain 255 and the second EM frequency domain 265. The characteristics may include respective frequencies, wavelengths, and / or focal lengths associated with the first EM frequency domain 255 and the second EM frequency domain 265.

[0069] The focal length varies based on the EM frequency domain. For example, according to the formula The refractive index of a medium (μ) depends on the speed of light in a vacuum (c) and the speed of light in the medium (v). According to the formula v = F·λ, the speed of light in the medium (v) also depends on the frequency of light (F) and the wavelength of light (λ). According to the formula The focal length (f) depends on the refractive index of the lens (μ2), the refractive index of the surrounding medium (μ1), the radius of curvature of the lens (R2), and the radius of curvature of the surrounding medium (R1). Therefore, the focal length (f) depends on the frequency (F) of the light and the wavelength (λ) of the light (e.g., in the EM frequency domain).

[0070] Thus, as noted above, the focus setting generator 235 may generate the second focus setting 260 for the second EM frequency domain 265 based on the first focus setting 250 for the first EM frequency domain 255 and based on the difference in respective characteristics of the first EM frequency domain 255 and the second EM frequency domain 265. According to the formula The focal length (f) depends on the distance (u) from the lens to the object and the distance (w) from the lens to the image sensor. The distance (u) from the lens to the object is the same regardless of the EM frequency domain. Therefore, the distance (u) from the lens to the object is the same for both the first EM frequency domain 255 and the second EM frequency domain 265. The first focus setting 250 for the first EM frequency domain 255 indicates the distance (w) from the lens to the image sensor for the first EM frequency domain 255. d1 The focal length (f ) used for the first EM frequency domain 255 d1 ) and the focal length (f d2 ) may be calculated by the focus setting generator 235, for example, based on some of the formulas in the previous paragraph and / or based on measurements. Thus, the focus setting generator 235 may calculate the focus setting according to the formula Determine the distance (w) from the lens to the image sensor for the second EM frequency domain 265 d2 ), generating a second focus setting 260 for a second EM frequency domain 265. This formula can be derived as follows:

[0071]

[0072]

[0073] In some examples, focus setting generator 235 may generate second focus setting 260 for second EM frequency domain 265 using PDAF, CDAF, HAF, or a combination thereof, as discussed above with respect to focus setting generator 235 generating first focus setting 250 for first EM frequency domain 255 .

[0074] The focus setting generator 235 may transmit the first focus setting 250 for the first EM frequency domain 255 and / or the second focus setting 260 for the second EM frequency domain 265 to the focus mechanism 240. For example, the focus setting generator 235 may transmit the first focus setting 250 for the first EM frequency domain 255 to the focus mechanism 240 to actuate a linear actuator of the focus mechanism 240 to move the lens (e.g., lens 115) to a position determined by the first focus setting 250 (e.g., based on the distance w d1 ) indicates the lens position P d1 The focus setting generator 235 may transmit a second focus setting 260 for a second EM frequency domain 265 to the focus mechanism 240 to actuate a linear actuator of the focus mechanism 240 to move the lens (e.g., lens 115) to a position determined by the second focus setting 260 (e.g., based on the distance w d2 ) indicates the lens position P d2 .

[0075] A graphic representation of the focus setting generator 235 is shown in FIG. Figure 2 2 and 3. In the embodiment of the present invention, two focus value curves are illustrated, and two focus value curves for a first EM frequency domain 255 and a second EM frequency domain 265 are illustrated, respectively. Peaks in the respective focus value curves represent a first focus setting 250 for the first EM frequency domain 255 and a second focus setting 260 for the second EM frequency domain 265, respectively. The peaks in the respective focus value curves may represent contrast peaks of CDAF, in-phase PD pixel data, or a combination thereof. One of the focus value curves is marked with an eye icon, indicating that one of the EM frequency domains may be a visible light EM frequency domain. Another of the focus value curves is marked with a thermometer icon, indicating that one of the EM frequency domains may be an IR EM frequency domain.

[0076] The graphic representation of the focusing mechanism 240 is shown in FIG. Figure 2, and illustrates a lens (e.g., lens 115), a linear actuator for moving the lens, an image sensor (e.g., multi-domain image sensor 205), and an object (e.g., an apple). Arrows are illustrated indicating the directions in which the linear actuator can move the lens based on various focus settings (such as first focus setting 250 or second focus setting 260) received by focus mechanism 240.

[0077] In some examples, the imaging system 200 may configure the focus mechanism 240 to quickly switch back and forth between the first focus setting 250 for the first EM frequency domain 255 and the second focus setting 260 for the second EM frequency domain 265. When the focus mechanism 240 is set to the first focus setting 250 for the first EM frequency domain 255, the imaging system 200 may generate the single-domain image 225 in the first EM frequency domain 255. When the focus mechanism 240 is set to the second focus setting 260 for the second EM frequency domain 265, the imaging system 200 may generate the single-domain image 225 in the second EM frequency domain 265. Because the difference in lens position between the first focus setting 250 and the second focus setting 260 may be small, in some examples, the imaging system 200 may switch back and forth between the first focus setting 250 and the second focus setting 260 for each frame captured by the multi-domain image sensor 205. In this way, the imaging system 200 may capture the single-domain image 225 in the first EM frequency domain 255 using the first focus setting 250 for the first EM frequency domain 255, and may capture the single-domain image 225 in the second EM frequency domain 265 using the second focus setting 260 for the second EM frequency domain 265, and so on. If the multi-domain image sensor 205 can capture the multi-domain image data 210 at a rate of 60 frames per second (fps), the imaging system 200 may capture the single-domain image 225 in the first EM frequency domain 255 using the first focus setting 250 for the first EM frequency domain 255 at a rate of 30 fps, and the imaging system 200 may capture the single-domain image 225 in the second EM frequency domain 265 using the second focus setting 260 for the second EM frequency domain 265 at a rate of 30 fps.

[0078] The imaging system 200 includes an output device 230. The output device 230 may include one or more visual output devices, such as a display or a connector therefor. The output device 230 may include one or more audio output devices, such as a speaker, headphones, and / or connectors therefor. The output device 230 may include one or more of the output device 1035 and / or the communication interface 1040 of the computing system 1000. In some examples, the imaging system 200 causes the display of the output device 230 to display the single-domain image 225. In some examples, the imaging system 200 causes the display of the output device 230 to display the single-domain image 225 in the first EM frequency domain 255 using the first focus setting 250 for the first EM frequency domain 255 at a first time, and to display the single-domain image 225 in the second EM frequency domain 265 using the second focus setting 260 for the second EM frequency domain 265 at a second time, the second time being different from the first time.

[0079] In some examples, the output device 230 includes one or more transceivers. The transceiver may include a wired transmitter, a receiver, a transceiver, or a combination thereof. The transceiver may include a wireless transmitter, a receiver, a transceiver, or a combination thereof. The transceiver may include one or more of the output device 1035 and / or the communication interface 1040 of the computing system 1000. In some examples, the imaging system 200 causes the transceiver to transmit the single-domain image 225 to the recipient device. In some examples, the recipient device may include another imaging system 200, an HMD 310, a mobile phone 410, a computing system 1000, or a combination thereof. In some examples, the recipient device may include a display, and the data transmitted from the transceiver of the output device 230 to the recipient device may cause the display of the recipient device to display the single-domain image 225.

[0080] In some examples, the display of the output device 230 of the imaging system 200 functions as an optical "see-through" display that allows light from the real-world environment (scene) surrounding the imaging system 200 to traverse (e.g., pass through) the display of the output device 230 to one or both eyes of the user. For example, the display of the output device 230 may be at least partially transparent, translucent, light-transmissive, light-transmissive, or a combination thereof. In an illustrative example, the display of the output device 230 includes a transparent, translucent, and / or light-transmissive lens and a projector. The display of the output device 230 may include a projector that projects virtual content (e.g., a single-domain image 225) onto the lens. The lens may be, for example, a lens of a pair of glasses, a lens of goggles, a contact lens, a lens of a head-mounted display (HMD) device, or a combination thereof. Light from the real-world environment passes through the lens and reaches one or both eyes of the user. The projector may project the virtual content (e.g., a single-domain image 225) onto the lens so that the virtual content appears to be overlaid on the user's view of the environment from the perspective of one or both of the user's eyes. In some examples, the projector may project virtual content onto one or both retinas of one or both eyes of the user rather than onto a lens, which may be referred to as a virtual retinal display (VRD), retinal scanning display (RSD), or retinal projector (RP) display.

[0081] In some examples, the display of the output device 230 of the imaging system 200 is a digital "transparent" display that allows a user of the imaging system 200 and / or a recipient device to view a view of the environment by displaying the view of the environment on the display of the output device 230. The view of the environment displayed on the digital transparent display may be a view of the real-world environment surrounding the imaging system 200, for example, based on sensor data (e.g., images, videos, depth images, point clouds, other depth data, or a combination thereof) captured by the multi-domain image sensor 205 and / or other sensors described herein (e.g., multi-domain image data 210 and / or single-domain image 225). The view of the environment displayed on the digital transparent display may be a virtual environment (e.g., as in VR), which in some cases may include elements based on the real-world environment (e.g., the boundaries of a room). The view of the environment displayed on the digital transparent display may be an augmented environment based on the real-world environment (e.g., as in AR). The view of the environment displayed on the digital transparent display may be a hybrid environment based on the real-world environment (e.g., as in MR). The view of the environment displayed on the digital pass-through display may include virtual content (eg, single-domain image 225 ) overlaid on or otherwise incorporated into the view of the environment.

[0082] exist Figure 2Within, a graphic representing an output device 230 illustrates a display, a speaker, and a wireless transceiver that outputs the first image illustrated in the graphic representing the multi-domain image sensor 205, the multi-domain image data 210, the image processor 215, and the single-domain image 225, wherein the eye icon indicates that the output device 230 is configured to output the single-domain image 225 associated with an EM frequency domain (e.g., the first EM frequency domain 255 and / or the second EM frequency domain 265).

[0083] It should be understood that references herein to the multi-domain image sensor 205 and other sensors described herein (as image sensors) should be understood to also include other types of sensors that can produce output in the form of images, such as depth sensors that produce depth images and / or point clouds that can be represented in the form of images and / or rendered images of 3D models (e.g., RADAR, LIDAR, SONAR, SODAR, ToF, structured light). It should be understood that references herein to image data and / or images produced by such sensors may include any sensor data that can be output in the form of images, such as depth images, point clouds that can be represented in the form of images, and / or rendered images of 3D models.

[0084] In some examples, certain elements of the imaging system 200 (e.g., the multi-domain image sensor 205, the image processor 215, the single-domain re-mosaic engine 220, the output device 230, the focus setting generator 235, or a combination thereof) include software elements, such as a set of instructions corresponding to a program, which is executed on a processor (such as the processor 1010 of the computing system 1000, the image processor 150, the host processor 152, the ISP 154, or a combination thereof). In some examples, one or more of these elements of the imaging system 200 may include one or more hardware elements, such as a dedicated processor (e.g., the processor 1010 of the computing system 1000, the image processor 150, the host processor 152, the ISP 154, or a combination thereof). In some examples, one or more of these elements of the imaging system 200 may include a combination of one or more software elements and one or more hardware elements.

[0085] Figure 3A300 is a perspective view illustrating a head mounted display (HMD) 310 used as part of the imaging system 200. The HMD 310 may be, for example, an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or some combination thereof. The HMD 310 may be an example of the imaging system 200. The HMD 310 includes a first camera 330A and a second camera 330B along the front of the HMD 310. The first camera 330A and the second camera 330B may be examples of the multi-domain image sensor 205 of the imaging system 200. The HMD 310 includes a third camera 330C and a fourth camera 330D that face the user's eyes when the user's eyes face the display 340. The third camera 330C and the fourth camera 330D may be examples of the multi-domain image sensor 205 of the imaging system 200. In some examples, the HMD 310 may have only a single camera with a single image sensor. In some examples, the HMD 310 may include one or more additional cameras in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D. In some examples, the HMD 310 may include one or more additional sensors in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D, which may also include other types of multi-domain image sensors 205 of the imaging system 200. In some examples, the first camera 330A, the second camera 330B, the third camera 330C, and / or the fourth camera 330D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0086] The HMD 310 may include one or more displays 340 visible to a user 320 (wearing the HMD 310 on the head of the user 320). The one or more displays 340 of the HMD 310 may be an example of one or more displays of the output device 230 of the imaging system 200. In some examples, the HMD 310 may include one display 340 and two viewfinders. The two viewfinders may include a left viewfinder for the left eye of the user 320 and a right viewfinder for the right eye of the user 320. The left viewfinder may be oriented so that the left eye of the user 320 sees the left side of the display. The right viewfinder may be oriented so that the right eye of the user 320 sees the right side of the display. In some examples, the HMD 310 may include two displays 340, including a left display that displays content to the left eye of the user 320 and a right display that displays content to the right eye of the user 320. The one or more displays 340 of the HMD 310 may be a digital "pass-through" display or an optical "see-through" display.

[0087] The HMD 310 may include one or more earpieces 335, which may be used as speakers and / or headphones to output audio to one or both ears of a user of the HMD 310 and may be an example of an output device 230. Figure 3A and Figure 3B 310, one earpiece 335 is illustrated, but it should be understood that the HMD 310 may include two earpieces, one for each ear of the user (left ear and right ear). In some examples, the HMD 310 may also include one or more microphones (not shown). In some examples, the audio output by the HMD 310 to the user through the one or more earpieces 335 may include or be based on audio recorded using one or more microphones.

[0088] Figure 3B is an example Figure 3A 310 is a perspective view 350 of a head mounted display (HMD) being worn by a user 320. The user 320 wears the HMD 310 on the head of the user 320 above the eyes of the user 320. The HMD 310 may capture images through a first camera 330A and a second camera 330B. In some examples, the HMD 310 displays one or more output images toward the eyes of the user 320 using a display 340. In some examples, the output image may include a single domain image 225. The output image may be based on the images captured by the first camera 330A and the second camera 330B (e.g., multi-domain image data 210 and / or single domain image 225), for example, overlaid with virtual content (e.g., single domain image 225). The output image may provide a stereoscopic view of the environment, in some cases superimposed with virtual content and / or with other modifications. For example, the HMD 310 may display a first display image to the right eye of the user 320, the first display image being based on the image captured by the first camera 330A. The HMD 310 may display a second display image to the left eye of the user 320, the second display image being based on the image captured by the second camera 330B. For example, the HMD 310 may provide in the display image an overlay virtual content overlaid on the image captured by the first camera 330A and the second camera 330B. The third camera 330C and the fourth camera 330D may capture images of the eyes before, during, and / or after the user views the display image displayed by the display 340. In this way, the sensor data from the third camera 330C and / or the fourth camera 330D may capture the reaction of the user's eyes (and / or other parts of the user) to the virtual content. The earpiece 335 of the HMD 310 is illustrated as being in the ear of the user 320. The HMD 310 may output audio to the user 320 through the earpiece 335 and / or through another earpiece (not shown) of the HMD 310 in the other ear (not shown) of the user 320.

[0089] Figure 4A 4 is a perspective view 400 illustrating a front surface of a mobile phone 410 that includes a front-facing camera and can be used as part of the imaging system 200. The mobile phone 410 can be an example of the imaging system 200. The mobile phone 410 can be, for example, a cellular phone, a satellite phone, a portable game console, a music player, a fitness tracking device, a wearable device, a wireless communication device, a laptop computer, a mobile device, any other type of computing device or computing system discussed herein, or a combination thereof.

[0090] The front surface 420 of the mobile phone 410 includes a display 440. The front surface 420 of the mobile phone 410 includes a first camera 430A and a second camera 430B. The first camera 430A and the second camera 430B can be examples of the multi-domain image sensor 205 of the imaging system 200. The first camera 430A and the second camera 430B can face the user, including the user's eyes, while content (e.g., multi-domain image data 210 and / or single-domain image 225) is displayed on the display 440. The display 440 can be an example of a display of the output device 230 of the imaging system 200.

[0091] The first camera 430A and the second camera 430B are illustrated in a bezel around the display 440 on the front surface 420 of the mobile phone 410. In some examples, the first camera 430A and the second camera 430B may be positioned in a notch or cutout cut out of the display 440 on the front surface 420 of the mobile phone 410. In some examples, the first camera 430A and the second camera 430B may be under-display cameras positioned between the display 440 and the rest of the mobile phone 410 so that light passes through a portion of the display 440 before reaching the first camera 430A and the second camera 430B. The first camera 430A and the second camera 430B of the perspective view 400 are front-facing cameras. The first camera 430A and the second camera 430B face a direction perpendicular to the planar surface of the front surface 420 of the mobile phone 410. The first camera 430A and the second camera 430B may be two cameras of one or more cameras of the mobile phone 410. In some examples, the front surface 420 of the mobile phone 410 may have only a single camera.

[0092] In some examples, the display 440 of the mobile phone 410 displays one or more output images to a user using the mobile phone 410. In some examples, the output images may include the single-domain image 225. The output images may be based on images captured by the first camera 430A, the second camera 430B, the third camera 430C, and / or the fourth camera 430D (e.g., multi-domain image data 210 and / or the single-domain image 225), for example, overlaid with virtual content (e.g., the single-domain image 225).

[0093] In some examples, the front surface 420 of the mobile phone 410 may include one or more additional cameras in addition to the first camera 430A and the second camera 430B. The one or more additional cameras may also be examples of the multi-domain image sensor 205 of the imaging system 200. In some examples, the front surface 420 of the mobile phone 410 may include one or more additional sensors in addition to the first camera 430A and the second camera 430B. The one or more additional sensors may also be examples of the multi-domain image sensor 205 of the imaging system 200. In some cases, the front surface 420 of the mobile phone 410 includes more than one display 440. The one or more displays 440 of the front surface 420 of the mobile phone 410 may be examples of displays of the output device 230 of the imaging system 200. For example, the one or more displays 440 may include one or more touch screen displays.

[0094] The mobile handset 410 may include one or more speakers 435A and / or other audio output devices (eg, earphones or headphones or connectors thereof) that may output audio to one or more ears of a user of the mobile handset 410 . Figure 4A 4 shows one speaker 435A, but it should be understood that the mobile phone 410 may include more than one speaker and / or other audio devices. In some examples, the mobile phone 410 may also include one or more microphones (not shown). In some examples, the mobile phone 410 may include one or more microphones along and / or adjacent to the front surface 420 of the mobile phone 410, where these microphones are examples of the multi-domain image sensor 205 of the imaging system 200. In some examples, the audio output to the user by the mobile phone 410 through one or more speakers 435A and / or other audio output devices may include or be based on audio recorded using one or more microphones.

[0095] Figure 4B 4 is a perspective view 450 illustrating a rear surface 460 of a mobile phone that includes a rear camera and can be used as part of an imaging system 200. The mobile phone 410 includes a third camera 430C and a fourth camera 430D on the rear surface 460 of the mobile phone 410. The third camera 430C and the fourth camera 430D of the perspective view 450 are rear-facing. The third camera 430C and the fourth camera 430D can be examples of a multi-domain image sensor 205 of the imaging system 200. The third camera 430C and the fourth camera 430D face a direction perpendicular to the planar surface of the rear surface 460 of the mobile phone 410.

[0096] The third camera 430C and the fourth camera 430D may be two cameras of the one or more cameras of the mobile phone 410. In some examples, the rear surface 460 of the mobile phone 410 may have only a single camera. In some examples, the rear surface 460 of the mobile phone 410 may include one or more additional cameras in addition to the third camera 430C and the fourth camera 430D. The one or more additional cameras may also be examples of the multi-domain image sensor 205 of the imaging system 200. In some examples, the rear surface 460 of the mobile phone 410 may include one or more additional sensors in addition to the third camera 430C and the fourth camera 430D. The one or more additional sensors may also be examples of the multi-domain image sensor 205 of the imaging system 200. In some examples, the first camera 430A, the second camera 430B, the third camera 430C, and / or the fourth camera 430D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0097] The mobile handset 410 may include one or more speakers 435B and / or other audio output devices (eg, earphones or headphones or connectors thereof) that may output audio to one or more ears of a user of the mobile handset 410 . Figure 4B 435B, it should be understood that the mobile phone 410 may include more than one speaker and / or other audio devices. In some examples, the mobile phone 410 may also include one or more microphones (not shown). In some examples, the mobile phone 410 may include one or more microphones along and / or adjacent to the rear surface 460 of the mobile phone 410, where these microphones are examples of the multi-domain image sensor 205 of the imaging system 200. In some examples, the audio output to the user by the mobile phone 410 through one or more speakers 435B and / or other audio output devices may include or be based on audio recorded using one or more microphones.

[0098] The mobile phone 410 may use the display 440 on the front surface 420 as a transparent display. For example, the display 440 may display an output image, such as the single-domain image 225. The output image may be based on an image captured by the third camera 430C and / or the fourth camera 430D (e.g., the multi-domain image data 210 and / or the single-domain image 225), for example, overlaid with virtual content (e.g., the single-domain image 225). The first camera 430A and / or the second camera 430B may capture images of the user's eyes (and / or other parts of the user) before, during, and / or after the output image with virtual content is displayed on the display 440. In this way, the sensor data from the first camera 430A and / or the second camera 430B may capture the reaction of the user's eyes (and / or other parts of the user) to the virtual content.

[0099] Figure 5 5 is a conceptual diagram illustrating a photodetector array of a multi-domain image sensor 500. The multi-domain image sensor 500 is an example of the multi-domain image sensor 205 of the imaging system 200. The photodetector array of the multi-domain image sensor 500 is illustrated as a grid that is divided into 1976 rows 510 and 2624 columns 515. Each cell in the grid represents a photodetector that is sensitive to a particular EM frequency domain (or a subset thereof) marked on the cell. The cell marked as "R" is sensitive to red light from the visible light EM frequency domain. The cell marked as "G" is sensitive to green light from the visible light EM frequency domain. The cell marked as "B" is sensitive to blue light from the visible light EM frequency domain. The cell marked as "I" is sensitive to infrared signals from the infrared EM frequency domain. The photodetectors sensitive to red, green, and blue are sensitive to the visible light EM frequency domain, which can be the first EM frequency domain of the multiple EM frequency domains to which the multi-domain image sensor 500 is sensitive. The IR sensitive photodetector is sensitive to an IR EM frequency domain, which may be a second EM frequency domain among the plurality of EM frequency domains to which the multi-domain image sensor 500 is sensitive.

[0100] The multi-domain image sensor 500 may include a plurality of active lines (e.g., 1944 or other number of active lines) for reading photodetector data from the multi-domain image sensor 500, for example, to capture the multi-domain image data 210 using the multi-domain image sensor 500. In some cases, the multi-domain image sensor 500 may include a plurality of “active dummy” lines (e.g., 32 or other number of active dummy lines) for reading “dummy” photodetector data. For example, the “dummy” photodetector data may be used to detect and / or correct leakage current, for image stabilization, for PDAF, for CDAF, for calibration, and / or for other purposes. In some cases, the “dummy” photodetector data is included in the multi-domain image data 210. In some examples, the “dummy” photodetector data is excluded from the multi-domain image data 210.

[0101] Figure 66 is a block diagram illustrating an example architecture of an imaging system 600 that performs an imaging process using a multi-domain image sensor that is sensitive to a first electromagnetic (EM) frequency domain 620 and a second EM frequency domain 625. The imaging system uses a dual-domain image sensor 602 to capture dual-domain raw image data 605. The dual-domain image sensor 602 is an example of a multi-domain image sensor 205 that is sensitive to the first EM frequency domain 620 and the second EM frequency domain 625. The dual-domain raw image data 605 is an example of the multi-domain image data 210 that includes photodetector data from a first set of photodetectors that is sensitive to the first EM frequency domain 620 and photodetector data from a second set of photodetectors that is sensitive to the second EM frequency domain 625.

[0102] The imaging system 600 (e.g., the image processor 215 of the imaging system 600) performs demosaicing and / or pixel interpolation 610 on the dual-domain raw image data 605 to divide the dual-domain raw image data 605 into single-domain images associated with the first EM frequency domain 620 and the second EM frequency domain 625, respectively. Figure 6 In the illustrated example, the first EM frequency domain 620 is a visible light EM frequency domain and includes red 630 (R), green 635 (G), and blue 640 (B) sub-domains. Figure 6 In the illustrated example, the second EM frequency domain 625 is the IR EM frequency domain.

[0103] At the stage immediately following demosaicing and / or pixel interpolation 610, the single-domain image data may include potential visual artifacts 615. Since the lens corresponding to the dual-domain image sensor 602 can only be in one position at a time, the potential visual artifacts 615 may include focus blur. Thus, if the focus mechanism 240 sets the position of the lens to a first lens position according to a first focus setting for the first EM frequency domain 620 (e.g., as determined using the focus setting generator 235), the single-domain image data for the first EM frequency domain 620 is clear, sharp, and in focus, but the single-domain image data for the second EM frequency domain 625 may be blurry and out of focus. On the other hand, if the focus mechanism 240 sets the position of the lens to a second lens position according to a second focus setting for the second EM frequency domain 625 (e.g., as determined using the focus setting generator 235), the single-domain image data for the second EM frequency domain 625 is clear, sharp, and in focus, but the single-domain image data for the first EM frequency domain 620 may be blurry and out of focus.

[0104] Potential visual artifacts 615 corresponding to focus blur can be addressed by alternating focus settings on a frame-by-frame basis. For example, the imaging system 600 can apply a second focus setting for the second EM frequency domain 625 before capturing the Nth frame using the dual-domain image sensor 602, and can use the Nth frame captured by the dual-domain image sensor 602 to generate a single-domain image for the second EM frequency domain 625 (e.g., one of the focused images 655). The imaging system 600 can apply a first focus setting for the first EM frequency domain 620 before capturing the (N+1)th frame using the dual-domain image sensor 602, and can use the (N+1)th frame captured by the dual-domain image sensor 602 to generate a single-domain image for the first EM frequency domain 620 (e.g., one of the focused images 650). In this way, two single-domain images are captured according to respective focus settings appropriate for the corresponding EM frequency domains. The imaging system 600 can continue in this manner, thereby generating a stream of focused images 650 in the first EM frequency domain 620 and a stream of focused images 655 in the second EM frequency domain 625.

[0105] If the dual-domain image sensor 602 can capture dual-domain raw image data 605 at a rate of 60 fps, the stream of focused images 650 in the first EM frequency domain 620 and the stream of focused images 655 in the second EM frequency domain 625 can both be output at a rate of 30 fps. If the dual-domain image sensor 602 can capture dual-domain raw image data 605 at a rate of 90 fps, the focused images 650 and the focused images 655 can both be output at a rate of 45 fps. If the dual-domain image sensor 602 can capture dual-domain raw image data 605 at a rate of 120 fps, the focused images 650 and the focused images 655 can both be output at a rate of 60 fps. If the dual-domain image sensor 602 can capture dual-domain raw image data 605 at a rate of 240 fps, the focused images 650 and the focused images 655 can both be output at a rate of 120 fps. If the dual-domain image sensor 602 can capture dual-domain raw image data 605 at a rate of 480 fps, both the focused image 650 and the focused image 655 can be output at a rate of 240 fps. If the dual-domain image sensor 602 can capture dual-domain raw image data 605 at a rate of 30 fps, both the focused image 650 and the focused image 655 can be output at a rate of 15 fps. If the dual-domain image sensor 602 can capture dual-domain raw image data 605 at a rate of 24 fps, both the focused image 650 and the focused image 655 can be output at a rate of 24 / 2 fps.

[0106] Potential visual artifacts 615 may also include cross-domain contamination, which may produce visual artifacts such as false color and / or chromatic aberration, and may particularly appear at or around sharp edges in the image. Potential visual artifacts 615 corresponding to focus blur may be addressed using cross-domain contamination reduction 640.

[0107] The imaging system 600 (e.g., the image processor 215 of the imaging system 600) performs cross-domain contamination reduction 640 on single-domain image data from the first EM frequency domain 620 and / or image data from the second EM frequency domain 625, e.g., as previously discussed with respect to the image processor 215. For example, to remove cross-domain contamination from the second EM frequency domain 625 (IR) from a single-domain image from the first EM frequency domain 620 (VL), pixel data values ​​from a given IR-sensitive photodetector may be multiplied by a constant, and the product may be subtracted from pixel data values ​​from VL-sensitive photodetectors adjacent to the IR-sensitive photodetector. Different constants may be used for different colors or subsets of the EM frequency domains. For example, pixel data values ​​from a given IR-sensitive photodetector may be multiplied by a first constant a, and the product may be subtracted from pixel data values ​​from red-sensitive photodetectors adjacent to the IR-sensitive photodetector. The pixel data values ​​from a given IR-sensitive photodetector may be multiplied by a second constant, b, and the product may be subtracted from the pixel data values ​​from the green-sensitive photodetector adjacent to the IR-sensitive photodetector. The pixel data values ​​from a given IR-sensitive photodetector may be multiplied by a third constant, c, and the product may be subtracted from the pixel data values ​​from the blue-sensitive photodetector adjacent to the IR-sensitive photodetector. These calculations are represented using the formulas red = R–a×IR, green = G–b×IR, and blue = B–c×IR.

[0108] In some examples, the values ​​of constants a, b, and / or c may be in the range of 0 to 1, including, for example, 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, or a value between any of the listed values. In some examples, the values ​​of constants a, b, and / or c may exceed 1. In some examples, the values ​​of constants a, b, and / or c may be different from each other. In some examples, the values ​​of two or more of the constants a, b, and / or c may be equal to each other.

[0109] In some examples, the formula IR may be used avg =(IR N +IR N+1 ) / 2, the value IR avg Determine the value IR of a given IR photodiode on frame N N and the value IR of a given IR photodiode on the (N+1)th frame N+1The cross-domain pollution reduction 640 can be calculated using red = R – a × IR avg , Green = G–b×IR avg and blue = B–c×IR avg .

[0110] In some examples, the imaging system 600 (e.g., the single-domain remosaicing engine 220 of the imaging system 600) may perform remosaicing 645 on the image data processed to the first EM frequency domain 620 using cross-domain contamination reduction 640, as discussed with respect to the single-domain remosaicing engine 220. Thus, the focused image 650 may be remosaiced using the remosaicing 645.

[0111] Figure 7 7 is a flow chart illustrating an imaging process 700 performed by an imaging system having a multi-domain image sensor sensitive to a first domain and a second domain. The imaging process 900 may be performed by an imaging system. In some examples, the imaging system may include, for example, an image capture and processing system 100, an image capture device 105A, an image processing device 105B, an image processor 150, an ISP 154, a host processor 152, an imaging system 200, a multi-domain image sensor 205, an image processor 215, a single-domain re-mosaic engine 220, an output device 230, a focus setting generator 235, an HMD 310, a mobile phone 410, a multi-domain image sensor 500, an imaging system 600, an imaging system performing the imaging process 800, an imaging system performing the imaging process 900, a computing system 1000, a processor 1010, or a combination thereof.

[0112] At operation 705, the imaging system may activate a multi-domain camera 740. The multi-domain camera 740 is configured to capture image data in multiple EM frequency domains, including a first EM frequency domain 745 (e.g., visible light) and a second EM frequency domain 750 (e.g., IR). In some examples, the multi-domain camera 740 is configured to capture image data at a rate of 60 fps. The multi-domain camera 740 can be an example of the image capture and processing system 100, the image capture device 105A, the image sensor 130, the multi-domain image sensor 205, the first camera 330A, the second camera 330B, the third camera 330C, the fourth camera 330D, the first camera 430A, the second camera 430B, the third camera 430C, the fourth camera 430D, the multi-domain image sensor 500, the dual-domain image sensor 602, a camera that captures images of the imaging process 800, a camera that includes an image sensor of the imaging process 900, a camera of the input device 1045, another camera and / or image sensor described herein, another sensor described herein, or a combination thereof.

[0113] At operation 710, the imaging system may skip odd frames of a first EM frequency domain 745 (eg, visible light) and skip even frames of a second EM frequency domain 750 (eg, IR). Figure 6 In the context of , the Nth frame may be an example of an odd frame to be skipped for the first EM frequency domain 745 (eg, visible light), and the (N+1)th frame may be an example of an even frame to be skipped for the second EM frequency domain 750 (eg, IR).

[0114] At operation 715, the imaging system may run autofocus using frames in the first EM frequency domain 745 (eg, visible light) and find the lens position P for best focus for the first EM frequency domain 745. d1 755. Lens position P for the first EM frequency domain 745 d1 755 may be an example of a first focus setting 250 for the first EM frequency domain 255. Running autofocus in operation 715 may be performed using the focus setting generator 235 and may include CDAF, PDAF, HAF, or a combination thereof.

[0115] At operation 720, the imaging system may d1 755 and the difference between the corresponding characteristics of the first EM frequency domain 745 and the second EM frequency domain 750 to calculate the corresponding lens position P for the frame in the second EM frequency domain 750 d2 760. Lens position P for the second EM frequency domain 750 d2 760 may be an example of a second focus setting 260 for a second EM frequency domain 265. d1 755 and the difference between the corresponding characteristics of the first EM frequency domain 745 and the second EM frequency domain 750 to calculate the lens position P for the frame in the second EM frequency domain 750 d2 760 may be performed using focus setting generator 235 and may be an example of calculating second focus setting 260 for second EM frequency domain 265 based on first focus setting 250 for first EM frequency domain 255 and a difference in respective characteristics of first EM frequency domain 255 and second EM frequency domain 265. Examples of respective characteristics include respective frequencies, wavelengths, and / or focal lengths associated with first EM frequency domain 745 and second EM frequency domain 750.

[0116] At operation 725, the imaging system may determine whether there is a scene change event. The scene change event may include an event in which a distance between the lens and the object (u) and / or a distance between the image sensor and the object changes. If there is a scene change event, the imaging system may perform operation 715 after operation 725. If there is no scene change event, the imaging system may perform operation 730 after operation 725.

[0117] At operation 730, the imaging system may set the lens position to P for odd frame numbers. d2 755, and for even frame numbers the lens position is set to P d1 760. In some examples, the imaging system may capture and / or generate image data in the second EM frequency domain 750 at odd frame numbers while the lens position is set to P d2 755. In some examples, the imaging system may capture and / or generate image data in the first EM frequency domain 745 at even frame numbers while the lens position is set to P d1 750.

[0118] At operation 730, the imaging system may perform operation 725 after operation 730. In some examples, the imaging system may perform operation 710 after operation 730.

[0119] Figure 8 8 is a flow chart illustrating an imaging process 800 performed by an imaging system having a multi-domain image sensor sensitive to a first domain and a second domain. The imaging process 900 may be performed by an imaging system. In some examples, the imaging system may include, for example, the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image processor 150, the ISP 154, the host processor 152, the imaging system 200, the multi-domain image sensor 205, the image processor 215, the single-domain re-mosaic engine 220, the output device 230, the focus setting generator 235, the HMD 310, the mobile phone 410, the multi-domain image sensor 500, the imaging system 600, the imaging system performing the imaging process 700, the imaging system performing the imaging process 900, the computing system 1000, the processor 1010, or a combination thereof.

[0120] At operation 805, the imaging system may use K frames to run autofocus for the first EM frequency domain and stabilize the autofocus at a lens position P for the first EM frequency domain. d1 840. The imaging system can also be based on the lens position P d1 840 and the difference between the corresponding characteristics of the first EM frequency domain and the second EM frequency domain to calculate the lens position P for the first EM frequency domain d2 845. Lens position P for the first EM frequency domain d1 840 may be an example of a first focus setting 250 for a first EM frequency domain 255. Lens position P for a second EM frequency domain d2 845 may be an example of a second focus setting 260 for a second EM frequency domain 265 .

[0121] At operation 810, the imaging system may be configured to:d2 At operation 815, the imaging system may capture and / or generate the (K+1)th frame when the lens is at lens position P d1 At operation 820, the imaging system may capture and / or generate the (K+2)th frame when the lens is at lens position P d2 At operation 825, the imaging system may capture and / or generate the (K+3)th frame when the lens is at lens position P d1 At 840, the (K+4)th frame is captured and / or generated.

[0122] At operation 830, the imaging system may detect a scene change, as in operation 725. The imaging system may perform operation 805 after detecting a scene change in operation 830.

[0123] Fig. 9 900 is a flow chart illustrating an imaging process 900. The imaging process 900 may be performed by an imaging system. In some examples, the imaging system may include, for example, the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image processor 150, the ISP 154, the host processor 152, the imaging system 200, the multi-domain image sensor 205, the image processor 215, the single-domain re-mosaic engine 220, the output device 230, the focus setting generator 235, the HMD 310, the mobile phone 410, the multi-domain image sensor 500, the imaging system 600, the imaging system performing the imaging process 700, the imaging system performing the imaging process 800, the computing system 1000, the processor 1010, or a combination thereof.

[0124] At operation 905, the imaging system may determine a first focus setting for focusing on a scene using the image sensor in a first electromagnetic (EM) frequency domain based on image data of the scene received from the image sensor. In one illustrative example, the imaging system may determine the first focus setting based on a focal length associated with the image sensor and the first EM frequency domain. Additionally or alternatively, in another illustrative example, the imaging system may determine the first focus setting based on a distance between a lens and at least a portion of the scene, wherein the lens is associated with the image sensor.

[0125] At operation 910, the imaging system may determine a second focus setting for focusing on a scene using an image sensor in a second EM frequency domain based on the first focus setting and a difference between respective characteristics of the first EM frequency domain and the second EM frequency domain. For example, similar to the examples described herein, the first EM frequency domain may include at least a portion of a visible light EM frequency domain, and the second EM frequency domain may include at least a portion of an infrared (IR) EM frequency domain. Examples of the first EM frequency domain and / or the second EM frequency domain include the first EM frequency domain 255, the second EM frequency domain 265, the first EM frequency domain 620, the second EM frequency domain 625, the first EM frequency domain 745, the second EM frequency domain 750, a radio EM frequency domain, a microwave EM frequency domain, an infrared (IR) EM frequency domain, a visible light (VL) EM frequency domain, an ultraviolet (UV) EM frequency domain, an X-ray EM frequency domain, a gamma ray EM frequency domain, a red EM frequency domain, a green EM frequency domain, a blue EM frequency domain, a NIR EM frequency domain, another EM frequency domain described herein, a subset of any of these, or a combination thereof.

[0126] Examples of the first focus setting include a first focus setting 250 for the first EM frequency domain 255, a distance (w d1 ), lens position P d1 755. Lens Position P d1 840 or a combination thereof. Examples of the second focus setting include the second focus setting 255 for the second EM frequency domain 265, the distance (w from the lens to the image sensor) for the second EM frequency domain 265 d2 ), lens position P d2 760, lens position P d2 845 or a combination thereof.

[0127] Examples of respective characteristics of the first EM frequency domain and the second EM frequency domain include respective frequencies, wavelengths, and / or focal lengths associated with the first EM frequency domain 745 and the second EM frequency domain 750 .

[0128] Examples of image sensors include image sensor 130, multi-domain image sensor 205, first camera 330A, second camera 330B, third camera 330C, fourth camera 330D, first camera 430A, second camera 430B, third camera 430C, fourth camera 430D, multi-domain image sensor 500, dual-domain image sensor 602, an image sensor of multi-domain camera 740 of imaging process 700, an image sensor that captures an image of imaging process 800, an image sensor of imaging process 900, an image sensor of input device 1045, another image sensor described herein, another sensor described herein, or a combination thereof.

[0129] In some cases, the image sensor includes a first set of photodetectors configured to detect a first EM frequency domain and a second set of photodetectors configured to detect a second EM frequency domain. In some examples, the first set of photodetectors includes multiple subsets of the first set of photodetectors. For example, the multiple subsets of the first set of photodetectors can be associated with the multiple subsets of the first EM frequency domain.

[0130] In some examples, the imaging system may receive image data from the image sensor prior to operation 905. In some examples, the imaging system includes an image sensor connector that couples and / or connects the image sensor to the rest of the imaging system (e.g., including a processor and / or memory of the imaging system). In some cases, the imaging system receives image data from the image sensor by receiving the image data from, on, and / or using the image sensor connector.

[0131] Examples of image data include image data captured using the image capture and processing system 100, multi-domain image data 210, image data captured using the first camera 330A, image data captured using the second camera 330B, image data captured using the third camera 330C, image data captured using the fourth camera 330D, image data captured using the first camera 430A, image data captured using the second camera 430B, image data captured using the third camera 430C, image data captured using the fourth camera 430D, image data captured using the multi-domain image sensor 500, dual-domain raw image data 605, image frames captured by the multi-domain camera 740 of the imaging process 700, various image frames captured during the imaging process 800, or a combination thereof.

[0132] At operation 915, the imaging system may cause the image sensor to capture a first image of the scene in the first EM frequency domain according to the first focus setting. At operation 920, the imaging system may cause the image sensor to capture a second image of the scene in the second EM frequency domain according to the second focus setting. In some cases, the imaging system may use the image sensor to capture the first image of the scene in the first EM frequency domain according to the first focus setting; and use the image sensor to capture the second image of the scene in the second EM frequency domain according to the second focus setting. In some aspects, the imaging system may move a lens to a first position corresponding to the first focus setting before capturing the first image, wherein the lens is associated with the image sensor. In some cases, the imaging system may further move the lens to a second position corresponding to the second focus setting after capturing the first image and before capturing the second image.

[0133] In some examples, the imaging system can reduce the data associated with the second EM frequency domain from the data associated with the first EM frequency domain (eg, using Figure 6 The imaging system may also reduce cross-domain contamination 640 described above to reduce contamination from the second EM frequency domain in the first image. Additionally or alternatively, the imaging system may reduce data associated with the first EM frequency domain from data associated with the second EM frequency domain (e.g., using a method related to Figure 6 The cross-domain contamination reduction 640 described above is performed to reduce contamination from the first EM frequency domain in the second image.

[0134] In some cases, the imaging system may output the first image and the second image. For example, the imaging system may output the first image at a first time and output the second image at a second time, the second time being different from the first time. In some examples, the imaging system may display the first image and the second image (e.g., via a display). In some cases, the imaging system may send the first image and the second image to a recipient device.

[0135] In some examples, the processes described herein (e.g., Figure 1 , Figure 2 , Figure 5 , Figure 6 The corresponding processes of the imaging process 700, the imaging process 800, the imaging process 900 and / or other processes described herein) can be performed by a computing device or apparatus. In some examples, the processes described herein can be performed by the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image processor 150, the ISP 154, the host processor 152, the imaging system 200, the multi-domain image sensor 205, the image processor 215, the single-domain re-mosaic engine 220, the output device 230, the focus setting generator 235, the HMD 310, the mobile phone 410, the multi-domain image sensor 500, the imaging system 600, the imaging system performing the imaging process 700, the imaging system performing the imaging process 800, the imaging system performing the imaging process 900, the computing system 1000, the processor 1010, or a combination thereof.

[0136] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR head-mounted device, an AR head-mounted device, an AR glasses, a network-connected watch or smart watch or other wearable device), a server computer, an automatic vehicle or a computing device of an automatic vehicle, a robotic device, a television, and / or any other computing device with resource capabilities to perform the processes described herein. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0137] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0138] The processes described herein are illustrated as logic flow diagrams, block diagrams, or conceptual diagrams, the operations of which represent sequences of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operations. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0139] Additionally, the processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or a combination of the above. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0140] Fig.10 is a diagram illustrating an example of a system for implementing certain aspects of the disclosed technology. Specifically, Fig.10 An example of a computing system 1000 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 1005. The connection 1005 may be a physical connection using a bus, or a direct connection into the processor 1010, such as in a chipset architecture. The connection 1005 may also be a virtual connection, a networked connection, or a logical connection.

[0141] In some aspects, computing system 1000 is a distributed system in which the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components that each perform some or all of the functionality for which the component is described. In some aspects, each component may be a physical or virtual device.

[0142] The example system 1000 includes at least one processing unit (CPU or processor) 1010 and connections 1005 that couple various system components including system memory 1015, such as read-only memory (ROM) 1020 and random access memory (RAM) 1025, to the processor 1010. The computing system 1000 may include a cache 1012 of high-speed memory directly connected to the processor 1010, in close proximity to the processor, or integrated as part of the processor.

[0143] Processor 1010 may include any general purpose processor and hardware or software services, such as services 1032, 1034, and 1036 stored in storage device 1030, that are configured to control processor 1010 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1010 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0144] To enable user interaction, the computing system 1000 includes an input device 1045 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1000 may also include an output device 1035 that can be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1000. The computing system 1000 may include a communication interface 1040, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transmission, Low energy (BLE) wireless signal transmission, Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 1002.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1040 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1000 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0145] The storage device 1030 may be a non-volatile and / or non-transitory and / or computer-readable memory device, and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, A smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0146] Storage device 1030 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1010, causes the system to perform functions. In some aspects, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to the necessary hardware components (such as processor 1010, connection 1005, output device 1035, etc.).

[0147] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store thereon code and / or machine-executable instructions that may represent any combination of a process, function, subroutine, program, routine, subroutine, module, software package, category, or instruction, data structure, or program statement. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0148] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media expressly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0149] Specific details are provided in the above description to provide a detailed understanding of the various aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can also be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in the method embodied in software or a combination of hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid confusing various aspects.

[0150] Various aspects may be described above as a process or method, which is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe an operation as a sequential process, many operations in the operation may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operation of the process is completed, but the process may have additional steps not included in the accompanying drawings. A process corresponds to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0151] The processes and methods according to the above examples can be implemented using stored computer executable instructions or computer executable instructions otherwise obtained from a computer readable medium. Such instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a function or a group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0152] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0153] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0154] In the above description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary aspects of the present application have been described in detail herein, it is to be understood that each inventive concept can be implemented and adopted in various other ways, and the appended claims are not intended to be interpreted as including such variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, various aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader essence and scope of this specification. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that, in alternative aspects, each method can be performed in a different order than described.

[0155] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0156] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0157] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0158] Claim language or other language that states "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that states "at least one of A and B" means A, B, or A and B. In another example, claim language that states "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language that states "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0159] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the various aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0160] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions that execute one or more of the above methods when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0161] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in a dedicated software module or hardware module configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0162] Exemplary aspects of the present disclosure include:

[0163] Aspect 1. A device for determining one or more image settings, the device comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory, the at least one processor being configured to: determine a first focus setting for focusing on the scene using the image sensor in a first electromagnetic (EM) frequency domain based on image data of the scene received from an image sensor; determine a second focus setting for focusing on the scene using the image sensor in a second EM frequency domain based on the first focus setting and a difference between corresponding characteristics of the first EM frequency domain and a second EM frequency domain; cause the image sensor to capture a first image of the scene in the first EM frequency domain according to the first focus setting; and cause the image sensor to capture a second image of the scene in the second EM frequency domain according to the second focus setting.

[0164] Aspect 2. The apparatus according to aspect 1, wherein the first EM frequency domain comprises at least a portion of a visible light EM frequency domain, and wherein the second EM frequency domain comprises at least a portion of an infrared (IR) EM frequency domain.

[0165] Aspect 3. An apparatus according to any one of Aspects 1 to 2, wherein the image sensor comprises a first set of photodetectors and a second set of photodetectors, the first set of photodetectors being configured to detect the first EM frequency domain, and the second set of photodetectors being configured to detect a second EM frequency domain.

[0166] Aspect 4. The apparatus of aspect 3, wherein the first set of photodetectors comprises a plurality of subsets of the first set of photodetectors, the plurality of subsets of the first set of photodetectors being associated with a plurality of subsets of the first EM frequency domain.

[0167] Aspect 5. An apparatus according to any one of Aspects 1 to 4, wherein, to determine the first focus setting, the at least one processor is configured to: determine the first focus setting based on a focal length associated with the image sensor and the first EM frequency domain.

[0168] Aspect 6. An apparatus according to any one of Aspects 1 to 5, wherein, in order to determine the first focus setting, the at least one processor is configured to: determine the first focus setting based on a distance between a lens and at least a portion of the scene, wherein the lens is associated with the image sensor.

[0169] Aspect 7. An apparatus according to any one of Aspects 1 to 6, wherein the at least one processor is configured to: before capturing the first image, move the lens to a first position corresponding to the first focus setting, wherein the lens is associated with the image sensor; and after capturing the first image and before capturing the second image, move the lens to a second position corresponding to the second focus setting.

[0170] Aspect 8. An apparatus according to any one of Aspects 1 to 7, wherein the at least one processor is configured to: reduce data associated with the second EM frequency domain from data associated with the first EM frequency domain to reduce contamination from the second EM frequency domain in the first image.

[0171] Aspect 9. An apparatus according to any one of Aspects 1 to 8, wherein the at least one processor is configured to: reduce data associated with the first EM frequency domain from data associated with the second EM frequency domain to reduce contamination from the first EM frequency domain in the second image.

[0172] Aspect 10. An apparatus according to any one of aspects 1 to 9, wherein the at least one processor is configured to: output the first image and the second image.

[0173] Aspect 11. An apparatus according to any one of Aspects 1 to 10, wherein the at least one processor is configured to: output the first image at a first time; and output the second image at a second time, the second time being different from the first time.

[0174] Aspect 12. The apparatus according to any one of aspects 1 to 11, further comprising: a display configured to display the first image and the second image.

[0175] Aspect 13. The apparatus according to any one of aspects 1 to 12, further comprising: a communication transceiver configured to send the first image and the second image to a recipient device.

[0176] Aspect 14. An apparatus according to any one of aspects 1 to 13, wherein the apparatus comprises at least one of a head mounted display (HMD), a mobile phone, or a wireless communication device.

[0177] Aspect 15. A method for determining one or more image settings, the method comprising: determining a first focus setting for using the image sensor to focus on the scene in a first electromagnetic (EM) frequency domain based on image data of a scene received from an image sensor; determining a second focus setting for using the image sensor to focus on the scene in a second EM frequency domain based on the first focus setting and a difference between corresponding characteristics of the first EM frequency domain and a second EM frequency domain; causing the image sensor to capture a first image of the scene in the first EM frequency domain according to the first focus setting; and causing the image sensor to capture a second image of the scene in the second EM frequency domain according to the second focus setting.

[0178] Aspect 16. The method according to aspect 15, wherein the first EM frequency domain includes at least a portion of a visible light EM frequency domain, and wherein the second EM frequency domain includes at least a portion of an infrared (IR) EM frequency domain.

[0179] Aspect 17. A method according to any one of Aspects 15 to 16, wherein the image sensor comprises a first set of photodetectors and a second set of photodetectors, the first set of photodetectors being configured to detect the first EM frequency domain, and the second set of photodetectors being configured to detect a second EM frequency domain.

[0180] Aspect 18. The method of aspect 17, wherein the first set of photodetectors comprises a plurality of subsets of the first set of photodetectors, the plurality of subsets of the first set of photodetectors being associated with a plurality of subsets of the first EM frequency domain.

[0181] Clause 19. The method according to any one of clauses 15 to 18, wherein determining the first focus setting comprises determining the first focus setting based on a focal length associated with the image sensor and the first EM frequency domain.

[0182] Clause 20. The method according to any one of Clauses 15 to 19, wherein determining the first focus setting comprises determining the first focus setting based on a distance between a lens and at least a portion of the scene, wherein the lens is associated with the image sensor.

[0183] Aspect 21. A method according to any one of Aspects 15 to 20, the method further comprising: before capturing the first image, moving the lens to a first position corresponding to the first focus setting, wherein the lens is associated with the image sensor; and after capturing the first image and before capturing the second image, moving the lens to a second position corresponding to the second focus setting.

[0184] Aspect 22. The method according to any one of aspects 15 to 21, further comprising: reducing data associated with the second EM frequency domain from data associated with the first EM frequency domain to reduce contamination from the second EM frequency domain in the first image.

[0185] Aspect 23. The method according to any one of aspects 15 to 22, the method further comprising: reducing the data associated with the first EM frequency domain from the data associated with the second EM frequency domain to reduce contamination from the first EM frequency domain in the second image.

[0186] Aspect 24. The method according to any one of aspects 15 to 23, further comprising: outputting the first image and the second image.

[0187] Aspect 25. The method according to any one of aspects 15 to 24, further comprising: outputting the first image at a first time; and outputting the second image at a second time, the second time being different from the first time.

[0188] Aspect 26. The method according to any one of aspects 15 to 25, further comprising: displaying the first image and the second image.

[0189] Aspect 27. The method according to any one of aspects 15 to 26, further comprising: sending the first image and the second image to a recipient device.

[0190] Aspect 28. The method according to any one of Aspects 15 to 27, further comprising: using the image sensor to capture the first image of the scene in the first EM frequency domain according to the first focusing setting; and using the image sensor to capture the second image of the scene in the second EM frequency domain according to the second focusing setting.

[0191] Aspect 29. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform the operations according to any one of aspects 15 to 28.

[0192] Aspect 30. An apparatus for image processing, the apparatus comprising one or more components for performing the operations according to any one of aspects 15 to 28.

Claims

1. An apparatus for determining one or more image settings, the apparatus comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory, the at least one processor configured to: determining, based on image data of a scene received from an image sensor, a first focus setting for focusing on the scene using the image sensor in a first electromagnetic (EM) frequency domain; determining a second focus setting for focusing on the scene using the image sensor in the second EM frequency domain based on the first focus setting and a difference between respective characteristics of the first EM frequency domain and a second EM frequency domain; causing the image sensor to capture a first image of the scene in the first EM frequency domain according to the first focus setting; as well as The image sensor is caused to capture a second image of the scene in the second EM frequency domain according to the second focus setting.

2. The apparatus of claim 1, wherein the first EM frequency domain comprises at least a portion of a visible light EM frequency domain, and wherein the second EM frequency domain comprises at least a portion of an infrared (IR) EM frequency domain.

3. The apparatus of claim 1, wherein the image sensor comprises a first set of photodetectors configured to detect the first EM frequency domain and a second set of photodetectors configured to detect a second EM frequency domain.

4. The apparatus of claim 3, wherein the first set of photodetectors comprises a plurality of subsets of the first set of photodetectors, the plurality of subsets of the first set of photodetectors being associated with a plurality of subsets of the first EM frequency domain.

5. The device according to claim 1, wherein: To determine the first focus setting, the at least one processor is configured to determine the first focus setting based on a focal length associated with the image sensor and the first EM frequency domain.

6. The device according to claim 1, wherein: To determine the first focus setting, the at least one processor is configured to determine the first focus setting based on a distance between a lens and at least a portion of the scene, wherein the lens is associated with the image sensor.

7. The apparatus of claim 1 , wherein the at least one processor is configured to: prior to capturing the first image, moving a lens to a first position corresponding to the first focus setting, wherein the lens is associated with the image sensor; and After capturing the first image and before capturing the second image, the lens is moved to a second position corresponding to the second focus setting.

8. The apparatus of claim 1, wherein the at least one processor is configured to: Data associated with the second EM frequency domain is reduced from data associated with the first EM frequency domain to reduce contamination from the second EM frequency domain in the first image.

9. The apparatus of claim 1 , wherein the at least one processor is configured to: Data associated with the first EM frequency domain is reduced from data associated with the second EM frequency domain to reduce contamination from the first EM frequency domain in the second image.

10. The apparatus of claim 1, wherein the at least one processor is configured to: The first image and the second image are output.

11. The apparatus of claim 1 , wherein the at least one processor is configured to: outputting the first image at a first time; and The second image is output at a second time, the second time being different from the first time.

12. The device according to claim 1, further comprising: A display configured to display the first image and the second image.

13. The apparatus according to claim 1, further comprising: A communication transceiver configured to transmit the first image and the second image to a recipient device.

14. The apparatus of claim 1, wherein the apparatus comprises at least one of a head mounted display (HMD), a mobile handset, or a wireless communication device.

15. A method of determining one or more image settings, the method comprising: determining, based on image data of a scene received from an image sensor, a first focus setting for focusing on the scene using the image sensor in a first electromagnetic (EM) frequency domain; determining a second focus setting for focusing on the scene using the image sensor in the second EM frequency domain based on the first focus setting and a difference between respective characteristics of the first EM frequency domain and a second EM frequency domain; causing the image sensor to capture a first image of the scene in the first EM frequency domain according to the first focus setting; as well as The image sensor is caused to capture a second image of the scene in the second EM frequency domain according to the second focus setting.

16. The method of claim 15, wherein the first EM frequency domain comprises at least a portion of a visible light EM frequency domain, and wherein the second EM frequency domain comprises at least a portion of an infrared (IR) EM frequency domain.

17. The method of claim 15, wherein the image sensor comprises a first set of photodetectors configured to detect the first EM frequency domain and a second set of photodetectors configured to detect a second EM frequency domain.

18. The method of claim 17, wherein the first set of photodetectors comprises a plurality of subsets of the first set of photodetectors, the plurality of subsets of the first set of photodetectors being associated with a plurality of subsets of the first EM frequency domain.

19. The method of claim 15, wherein determining the first focus setting comprises: The first focus setting is determined based on a focal length associated with the image sensor and the first EM frequency domain.

20. The method of claim 15, wherein determining the first focus setting comprises: The first focus setting is determined based on a distance between a lens and at least a portion of the scene, wherein the lens is associated with the image sensor.

21. The method according to claim 15, further comprising: prior to capturing the first image, moving a lens to a first position corresponding to the first focus setting, wherein the lens is associated with the image sensor; as well as After capturing the first image and before capturing the second image, the lens is moved to a second position corresponding to the second focus setting.

22. The method according to claim 15, further comprising: Data associated with the second EM frequency domain is reduced from data associated with the first EM frequency domain to reduce contamination from the second EM frequency domain in the first image.

23. The method according to claim 15, further comprising: Data associated with the first EM frequency domain is reduced from data associated with the second EM frequency domain to reduce contamination from the first EM frequency domain in the second image.

24. The method according to claim 15, further comprising: The first image and the second image are output.

25. The method according to claim 15, further comprising: outputting the first image at a first time; as well as The second image is output at a second time, the second time being different from the first time.

26. The method according to claim 15, further comprising: The first image and the second image are displayed.

27. The method according to claim 15, further comprising: The first image and the second image are sent to a recipient device.

28. The method of claim 15, further comprising: capturing, using the image sensor, the first image of the scene in the first EM frequency domain according to the first focus setting; as well as The second image of the scene is captured in the second EM frequency domain using the image sensor according to the second focus setting.

29. A non-transitory computer readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: determining, based on image data of a scene received from an image sensor, a first focus setting for focusing on the scene using the image sensor in a first electromagnetic (EM) frequency domain; determining a second focus setting for focusing on the scene using the image sensor in the second EM frequency domain based on the first focus setting and a difference between respective characteristics of the first EM frequency domain and a second EM frequency domain; causing the image sensor to capture a first image of the scene in the first EM frequency domain according to the first focus setting; as well as The image sensor is caused to capture a second image of the scene in the second EM frequency domain according to the second focus setting.

30. The non-transitory computer readable medium of claim 29, wherein the first EM frequency domain comprises at least a portion of a visible light EM frequency domain, and wherein the second EM frequency domain comprises at least a portion of an infrared (IR) EM frequency domain.

Images