Infrared-based processing of an image

Abstract

Multi-point measurements of a scene captured within an image frame can be used to process the image frame, for example, by applying white balance correction to the image frame. In some examples, the image frame can be segmented into parts illuminated by different light sources. The different parts of the image frame can be white-balanced differently based on the color temperature of the light source for the corresponding part. Infrared measurements at multiple points in the scene can be used to determine the characteristics of the light sources for different parts of the scene. For example, an image including indoor and outdoor portions can be illuminated by at least two light sources that produce different infrared measurement values. White balance can be applied differently to these two portions to correct the color temperature of the different sources.

Description

[Technical Field]

[0001] This patent application claims the benefit of U.S. Patent Application No. 17 / 334182, filed May 28, 2021, entitled “Infrared-Based Processing of an Image,” the entire contents of which are expressly incorporated herein by reference.

[0002] This case involves various systems related to image processing. Some features of this case can be implemented and provide improvements in image processing. [Previous Technology]

[0003] An image capturing device is a device capable of capturing one or more digital images (whether still images, photographs, or video image sequences) and can be incorporated into various devices. For example, an image capturing device may include a standalone digital camera or digital video camera, a wireless communication device equipped with a camera (e.g., a mobile phone, cellular, or satellite radio phone), a personal digital assistant (PDA), a panel or tablet computer, a gaming device, a computer device (e.g., a webcam), a video surveillance camera, or other devices with digital imaging or video capabilities.

[0004] An image capturing device captures a scene in the presence of light, and the light illuminating the scene is called a light source. A light source is usually not pure white, but rather has a bias toward a specific color. Color bias is usually measured according to color temperature. The human eye can compensate for non-pure white lighting, so colors appear relatively consistent across a wide range of lighting conditions. For example, the human eye adapts to different lighting conditions, making gray objects appear gray.

[0005] When the lighting source changes, electronic sensors, such as those used in image capturing devices, can perceive the same scene differently. Electronic sensors capture images, but may not be able to compensate for different lighting conditions with different color temperatures. Image sensors used in image capturing devices can capture images that exhibit color shifts attributable to lighting from non-pure white sources. The color shifts presented in the captured images may appear unnatural to the human eye, resulting in the perception that the sensor or capturing device is of low quality and cannot accurately capture images of the real world. [Summary of the Invention]

[0006] Image frames captured from an image sensor can be processed to compensate for the illumination conditions and color temperature of the illumination source. The white balance of the image depends on the color temperature of the illumination source. A white balance configured for an illumination source at a first color temperature may not correct for the second color temperature of a second illumination source, and may further reduce the image quality perceived by humans by introducing additional color shifts into the image.

[0007] When multiple illumination sources are present for a scene being captured, the white balance applied to the image frame captured by the image sensor can lead to further degraded image quality. Infrared measurements of the scene corresponding to the captured image frame can be used to process the image frame in order to correct the white balance in the image by taking into account the different illumination sources. Processing using infrared measurements can enhance the image frame by processing it with infrared measurements to correct the white balance based on the different light sources within the scene. Alternatively or additionally, other data corresponding to the image frame (e.g., feature maps and / or other relay data generated by computer vision processing) can be used to process the image frame, for example, to correct the white balance.

[0008] In the absence of infrared or other data for processing image frames, each illumination source in a scene may have a different color temperature, and white balance correction operates on image frame statistics without taking into account the different illumination source colors. White balance in a scene with mixed illumination sources may cause the entire image to be adjusted in a way that skews the color temperature of the illumination sources, which may result in an even more unnatural appearance to the human eye, and these drawbacks can be addressed by using the data processing image described according to embodiments of this invention.

[0009] Multi-point measurements of the scene corresponding to the captured image frame can be used to determine the portions of the image frame illuminated by different light sources. In some embodiments, different portions of the image frame can be white-balanced differently based on the color temperature of the light source for the corresponding portion. Infrared measurements at multiple points in the scene can be used to determine the characteristics of the light sources for different portions of the scene. For example, an image including indoor and outdoor portions can be illuminated by at least two light sources. The outdoor portion of the scene is illuminated by natural light sources (e.g., the sun) during the day, and the indoor portion of the scene is illuminated by artificial light sources (e.g., incandescent bulbs, fluorescent bulbs, LED bulbs, etc.). The portion of the scene illuminated by the sun can have a higher infrared measurement than other portions of the scene illuminated by artificial light sources. Therefore, multi-point infrared measurements can be used to distinguish between a first portion of the scene illuminated by the sun and a second portion of the scene illuminated by indoor light. White balance can be applied differently to these two portions to correct the color temperature of the different sources, resulting in an image with improved color accuracy and a more natural appearance to the human eye. If different parts are not processed differently during white balance, the image frame may include unnatural color casts that darken colors, such as making blues look too yellow.

[0010] Multi-point infrared measurements of a scene can have a lower resolution than image frames. For example, an image frame can have a resolution of 1920 values ​​by 1080 values ​​(where values ​​can correspond to pixels), while multi-point infrared measurements can have a resolution of 8 values ​​by 8 values. The lower resolution of multi-point infrared measurements can reduce the cost and complexity of the equipment used to acquire infrared measurements without significantly reducing the ability to determine white balance correction. The scene generally includes only one, two, or a few different lighting sources, making it unnecessary to determine the high resolution of the infrared measurements for the different lighting sources present in the scene.

[0011] Computer vision can be used to determine different portions of an image frame illuminated by different light sources. Computer vision analysis can identify features in a scene. Computer vision analysis can identify features found outdoors, such as window frames, door frames, trees, or animals, within a portion of an image frame, and determine the first portion of the image frame based on these features. Multi-point infrared measurements can be used in conjunction with computer vision analysis to refine the determination of different portions of the image frame. For example, several multi-point infrared measurements can be used to roughly define the outdoor portion of the image frame, and computer vision analysis can be used to determine the features defining the boundaries around the outdoor portion. In a scene including an indoor room with windows leading to the outside, multi-point infrared measurements can roughly identify a portion of the image within the window frame that is illuminated by sunlight, and computer vision analysis is used to identify the window frame that borders the outdoor portion of the scene. A first white balance can be applied to the outdoor portion of the scene inside the window frame to correct for the color temperature of sunlight, and a second white balance can be applied to the indoor portion of the scene around the window frame to correct for the color temperature of artificial lighting.

[0012] The determination of different portions can also, or alternatively, be based on relay data about the image frame. For example, the time of day and / or date of the image frame capture can indicate whether an outdoor area of ​​the scene is illuminated by direct overhead sunlight, evening sunlight, or moonlight. As another example, the location where the image frame is captured can indicate whether the image scene is an outdoor scene, for example, when the location corresponds to the presence of buildings, or can indicate a scene with mixed lighting sources, for example, when the location corresponds to the presence of buildings. Further examples of relay data include data received from a smart home assistant and / or wireless connections to other devices. When an image frame is generated, smart devices (such as light bulbs or televisions) near the image capturing device can indicate whether nearby artificial lighting sources are on or off, and the relay data of the image frame can be used to determine the white balance of one or more portions determined in the image frame and / or to determine the number of different portions present in the image frame. In some embodiments, each light bulb that is on can be used to indicate the presence of another portion of the image illuminated by an artificial light source.

[0013] The following outlines some aspects of this application to provide a basic understanding of the technology under discussion. This outline is not a comprehensive overview of all anticipated features of the application, and is neither intended to identify key or essential elements of all aspects of the application, nor to illustrate the scope of any or all aspects of the application. Its sole purpose is to present some concepts of one or more aspects of the application in a general form as a prelude to the more detailed description to be presented later.

[0014] Generally, the present invention describes image processing technology for digital cameras, which have image sensors and circuitry for processing data received from the image sensors, such as image signal processors (ISPs), general-purpose single-core or multi-core processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable or fixed logic devices. The image signal processor can be configured to control the capture of image frames from one or more image sensors and process the image frames from one or more image sensors to generate a view of the scene within the corrected image frames, wherein white balance can be applied to adjust the colors within the image frames captured from one or more image sensors as part of the process of generating the corrected image frames. In an example, the image signal processor can receive instructions to capture a sequence of image frames in response to the loading of software (such as a camera application) on a CPU. The image signal processor can be configured to generate a stream of single output frames based on the corresponding corrected images from the image sensors. A single output frame stream can include calibrated image frames containing image data from an image sensor that has been calibrated (e.g., via white balance).

[0015] Corrected image frames can be generated by combining the various white balance patterns of this invention with other computational photography techniques, such as high dynamic range (HDR) photography or multiple frame noise reduction (MFNR) techniques. For example, different white balances applied to different parts of an image frame can be applied to individual frames before combining the frames to generate HDR or MFNR photography. As another example, white balances applied to different parts of an image frame can be applied to a combined HDR photographic image frame or a combined MFNR photographic image frame.

[0016] Similarly, infrared-based segmentation can be used to modify and enhance other image processing techniques. For example, automatic exposure and autofocus operations can receive infrared measurements and use the infrared data to determine the exposure level and focal plane, respectively. For instance, autofocus operations can ignore portions of the image with high infrared values, as these values ​​correspond to distant portions of the scene outside the viewport. As another example, automatic exposure operations can determine the exposure level or shutter length based on portions of the image with low infrared light, as these portions may include the subject, which is the intended object of the photograph. Other operations can benefit from infrared measurements, such as computer vision tasks. For example, scene classification, pixel classification, object detection, feature extraction, and / or plane detection can receive and use infrared measurements when determining the morphology of an image frame.

[0017] After the image signal processor determines the output frames representing the scene, the view of the scene can be displayed on a device display, saved as a picture or a sequence of pictures as video to a storage device, transmitted via a network, and / or printed to an output medium. For example, the image signal processor can be configured to obtain input frames (e.g., pixel values) of image data from different image sensors, and then generate corresponding output frames of image data (e.g., preview display frames, still image capture, video frames, etc.). In other examples, the image signal processor can output frames of image data to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization (e.g., autofocus (AF), auto white balance (AWB), and auto exposure control (AEC)), generate video files via output frames, configure frames for display, generate frames for storage, etc. That is, the image signal processor can obtain input frames from one or more image sensors each coupled to one or more camera lenses, and in turn can generate a stream of output frames and output them to various output destinations. In such examples, an image signal processor can be configured to produce a stream of output frames that can have reduced ghosting or other artifacts caused by time filtering.

[0018] In one embodiment of this case, a method for image processing includes receiving a first image frame and a set of corresponding infrared measurements. The method may further determine a first portion and a second portion of the first image frame based on the set of infrared measurements. The method may further determine a second image frame based on the first image frame by performing the following operations: applying a first white balance to the first portion of the first image frame based on image content in the first portion; and applying a second white balance to the second portion of the first image frame based on image content in the second portion.

[0019] In an example application, a method can be applied to determine separate indoor and outdoor regions as a first portion and a second portion of a first image frame. The indoor and outdoor regions can be distinguished by thresholding infrared measurements, such that the indoor region has infrared measurements below a threshold and the outdoor region has infrared measurements above a threshold. White balance can be applied to the indoor region by applying a first set of weighted values ​​of the color intensity of each pixel in the indoor region to a lower correlated color temperature (CCT). White balance can be applied to the outdoor region by applying a second set of weighted values ​​of the color intensity of each pixel in the outdoor region to a higher correlated color temperature (CCT).

[0020] Although examples of scenes with mixed indoor and outdoor lighting are described in some example applications, different white balances can be applied to different parts of an image frame illuminated by different lighting sources that can be distinguished by infrared measurements. As another example, different indoor lighting sources, such as incandescent and LED lighting, can be distinguished by infrared measurements because the tungsten filament of incandescent lighting produces more infrared illumination than the semiconductor device of LED lighting. Furthermore, although infrared measurements for distinguishing different light sources within a scene are described, other measurements can be used to distinguish different parts of the scene. For example, ultraviolet (UV) measurements can also be used to distinguish between natural and artificial lighting sources. As another example, depth measurements can be used to distinguish between nearby indoor areas illuminated by artificial lighting sources and distant outdoor areas illuminated by natural lighting sources.

[0021] In some embodiments, the method can be performed for HDR photography, wherein the input image frames used for white balance operation are themselves a combination of first and second image frames captured using different exposure times, different apertures, different lenses, or other different characteristics that can produce a fused image when the two image frames are combined. In some embodiments, the method can be performed for MFNR photography, wherein the first and second image frames are captured using the same or different exposure times. White balance according to the embodiments of the present invention can be applied, for example, after computational photography for performing HDR or MFNR photography. In some embodiments, white balance according to the embodiments of the present invention can be applied to the first and second image frames before computational photography for producing HDR or MFNR photography.

[0022] In some embodiments, the method can be performed on multiple image sensors with different lens configurations (e.g., W, UW, T). For example, infrared measurements (or other data about the scene captured in an image frame) can be associated with two or more image frames captured from two or more cameras with different lens configurations. Different image frames can be processed to determine a first, second, or additional portion of each image frame that is related to each other. White balance or other processing can be applied to each image frame captured from each camera, and multiple processed image frames are combined to produce a fused image frame for output, or one of multiple processed image frames to be output is selected based on one or more criteria. In some embodiments, the processing of individual image frames from different cameras can be processed in parallel by multiple image signal processors. Simultaneous processing from multiple cameras using infrared measurements (and / or other data about image frames) can improve image quality in multi-camera HDR fusion techniques and / or subsequent post-processing of files in photo gallery applications to reconstruct the field of view (FOV) by capturing image frames from multiple cameras (even when the cameras are contained in different devices at different locations). Simultaneous processing from multiple cameras can also be used to enhance the user experience when interacting with image capturing devices, such as by masking photometric effects when switching between cameras during zoom changes during preview or video capture. Simultaneous processing can reduce the effects of discontinuous color changes when zooming in or out of video by causing the image capturing device to alter sensor settings.

[0023] In an additional embodiment of this application, an apparatus is disclosed, comprising at least one processor and memory coupled to the at least one processor. The at least one processor is configured to perform any of the methods or techniques described herein. For example, the at least one processor may be configured to perform steps including receiving a first image frame and a set of corresponding infrared measurements. The processor may also be configured to determine a first portion and a second portion of the first image frame based on the set of infrared measurements. The processor may be further configured to determine a second image frame based on the first image frame by performing the following operations: applying a first white balance to the first portion of the first image frame based on image content in the first portion; and applying a second white balance to the second portion of the first image frame based on image content in the second portion.

[0024] At least one processor may include an image signal processor or a processor that includes specific functions for camera control and / or processing (such as enabling or disabling time filtering and / or motion compensation). At least one processor may also or alternatively include an application processor. The methods and techniques described herein may be performed entirely by an image signal processor or an application processor, or various operations may be partitioned between an image signal processor and an application processor, and in some cases, various operations may be partitioned across additional processors.

[0025] The device may include one, two, or more image sensors, such as a first image sensor. When multiple image sensors are present, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or a different dynamic range than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor, and the second image sensor may be a telephoto image sensor. In another example, the first sensor is configured to acquire an image via a first lens having a first optical axis, and the second sensor is configured to acquire an image via a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification, and the second lens may have a second magnification different from the first magnification. This configuration may occur in the case of a lens cluster on a mobile device, such as where multiple image sensors and associated lenses are located at offset positions on the front or rear side of the mobile device. Additional image sensors with larger, smaller, or the same field of view may be included. The image correction techniques described herein can be applied to image frames captured from any image sensor in a multi-sensor device.

[0026] In another aspect of this case, an apparatus configured for image processing and / or image capture is disclosed. The apparatus includes components for capturing image frames. The apparatus further includes one or more components for capturing data representing a scene, such as image sensors (including charge-coupled devices (CCDs), Bayer filter sensors, infrared (IR) detectors, ultraviolet (UV) detectors, complementary metal-oxide-semiconductor (CMOS) sensors), and time-of-flight detectors. The apparatus may further include one or more components (including simple lenses, compound lenses, spherical lenses, and aspherical lenses) for accumulating and / or focusing light into one or more image sensors. These elements can be controlled to capture first and / or second image frames input to the image processing techniques described herein.

[0027] In an additional embodiment of this case, a non-transitory computer-readable media storage instruction, when executed by a processor, causes the processor to perform operations including those described in the methods and techniques described herein. For example, the operation may include receiving a first image frame and a corresponding set of infrared measurements. The operation may also include determining a first portion and a second portion of the first image frame based on the set of infrared measurements. The operation may further include determining a second image frame based on the first image frame by performing the following operations: applying a first white balance to the first portion of the first image frame based on image content in the first portion; and applying a second white balance to the second portion of the first image frame based on image content in the second portion.

[0028] Other specifications, features, and implementations will become apparent to those skilled in the art when examining the following specification specifications in conjunction with the accompanying drawings and specific exemplary specifications. While features may be discussed with respect to certain specifications and drawings below, each specification may include one or more of the advantageous features discussed herein. In other words, while one or more specifications may be discussed as having certain advantageous features, one or more of these features may also be used according to each specification. Similarly, while exemplary specifications may be discussed below as device, system, or method specifications, exemplary specifications may be implemented in various devices, systems, and methods.

[0029] The method can be embedded as computer program code in a computer-readable medium, the computer program code including instructions to cause a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device including: a first network interface card configured to transmit data, such as images or videos as recorded or streamed data, via a first network connection among a plurality of network connections; and a processor coupled to the first network interface card and memory.

[0030] Certain features and technical advantages of embodiments of the present invention have been summarized quite extensively above to provide a better understanding of the following detailed description. Additional features and advantages that form the subject matter of the claims will be described below. Those skilled in the art will understand that the disclosed concepts and specific embodiments can be readily used as the basis for modifications or the design of other structures for achieving the same or similar purposes. Those skilled in the art will also recognize that such equivalent constructions do not depart from the spirit and scope of the invention as set forth in the appended claims. Additional features will be better understood from the following description when considered in conjunction with the accompanying drawings. However, it should be clearly understood that each drawing is provided for illustrative and explanatory purposes only and is not intended to limit the invention.

Implementation Method

[0042] The detailed description below, set forth in conjunction with the accompanying drawings, is intended as a specification of various configurations and is not intended to limit the scope of the subject matter. Rather, the detailed description includes specific details for the purpose of providing a thorough understanding of the subject matter. It will be apparent to those skilled in the art that these specific details are not necessary in every case, and in some cases, well-known structures and elements are illustrated in block diagram form for clarity.

[0043] This application provides systems, apparatus, methods, and computer-readable media that support image processing of captured image frames for photography and video. Specific embodiments of the subject matter described in this invention may be implemented to achieve potential advantages or benefits, such as improving image quality by improving color accuracy in photographic or video sequences of image frames. The systems, apparatus, methods, and computer-readable media may be embedded in image capturing devices, such as mobile phones, tablet computing devices, laptop computing devices, other computing devices, or digital cameras.

[0044] An example device (e.g., a smartphone) for capturing image frames using one or more image sensors may include a configuration of two, three, four, or more cameras on the rear side (e.g., the side opposite the user's display) or front side (e.g., the same side as the user's display). Devices with multiple image sensors include one or more image signal processors (ISPs), computer vision processors (CVPs), or other suitable circuitry for processing images captured by the image sensors. One or more image signal processors may provide the processed image frames to memory and / or processors (such as application processors, image front-ends (IFEs), image processing engines (IPEs), or other suitable processing circuitry) for further processing, such as encoding, storage, transmission, or other manipulation.

[0045] As used herein, an image sensor can refer to the image sensor itself and any other elements coupled to the image sensor, which generates image frames for processing by an image signal processor or other logic circuitry or for storage in memory, whether a short-term buffer or long-term non-volatile memory. For example, an image sensor may include other elements of a camera, including a shutter, buffer, or other readout circuitry for accessing individual pixels of the image sensor. The image sensor may also include an analog front-end or other circuitry for converting analog signals into a digital representation of the image frames, which is provided to digital circuitry coupled to the image sensor.

[0046] Numerous specific details, such as examples of specific components, circuits, and procedures, are set forth in the following specification to provide a thorough understanding of the present invention. As used herein, the term "coupling" means a direct connection or a connection via one or more intermediate components or circuits. Furthermore, specific nomenclature is set forth in the following specification and for illustrative purposes to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that these specific details may not be necessary to practice the teachings disclosed herein. In other instances, well-known circuits and devices are illustrated in block diagram form to avoid obscuring the teachings of the present invention.

[0047] Some parts of the following detailed description are presented in terms of programs, logic blocks, processes, and other symbolic representations of operations on data bits within computer memory. In this context, programs, logic blocks, procedures, etc., are considered as a self-consistent sequence of steps or instructions that lead to the desired result. These steps are those that require physical manipulation of physical quantities. Typically, although not essential, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated in a computer system.

[0048] In the accompanying drawings, a single block can be described as performing one or more functions. The one or more functions performed by this block can be performed in a single element or across multiple elements, and / or can be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability between hardware and software, various illustrative elements, blocks, modules, circuits, and steps are generally described below according to their functions. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention. Furthermore, the example device may include elements other than those shown, including known components such as processors and memory.

[0049] The various forms of this invention are applicable to any suitable electronic device that includes or is coupled to two or more image sensors capable of capturing image frames (or "frames"). Furthermore, the various forms of this invention can be implemented in devices having or being coupled to image sensors having the same or different capabilities and characteristics (such as resolution, shutter speed, sensor type, etc.). Additionally, the various forms of this invention can be implemented in devices for processing image frames, regardless of whether the device includes or is coupled to image sensors, such as processing devices capable of capturing stored images for processing, including processing devices present in cloud computing systems.

[0050] Unless otherwise stated, it will be apparent from the following discussion that, throughout this case, the use of terms such as “access,” “receive,” “send,” “use,” “select,” “determine,” “normalize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” “export,” “establish,” and “generate” refers to the actions and procedures of a computer system or similar electronic computing device that manipulate and convert data represented as physical (electronic) quantities in the registers and memory of the computer system into other data represented as physical quantities in the registers, memory, or other such information storage, transmission, or display devices of the computer system.

[0051] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (such as a smartphone, a camera controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that can implement at least some parts of the present invention. Although the following description and examples use the term "device" to describe various aspects of the present invention, the term "device" is not limited to a specific configuration, type, or number of objects. As used herein, an apparatus can include a device or part of a device for performing the described operations.

[0052] FIG1 illustrates a block diagram of an example device 100 for performing image capture from one or more image sensors. Device 100 may include or be otherwise coupled to an image signal processor 112 for processing image frames from one or more image sensors (e.g., a first image sensor 101, a second image sensor 102, and a depth sensor 140). Image sensors 101 and / or 102 may be, for example, charge-coupled devices (CCDs) or CMOS sensors. Depth sensor 140 may be, for example, a single-point or multi-point detector, including one or more of time-of-flight (ToF) detectors (such as flood illumination ToF or direct ToF detectors), optical distance and ranging (LiDAR) devices, infrared sensors, and / or ultraviolet sensors. In some embodiments, device 100 also includes or is coupled to a processor 104 and a memory 106 storing instructions 108. Device 100 may also include or be coupled to display 114 and multiple input / output (I / O) elements 116, such as touch screen interfaces and / or physical buttons. Device 100 may further include or be coupled to a power source 118, such as a battery or a component that couples device 100 to an energy source. Device 100 may also include or be coupled to additional features or elements not shown in FIG. 1. In one example, a wireless interface for a wireless communication device may be included, which may include one or more transceivers and a baseband processor. In another example, an analog front end (AFE) for converting analog image frame data into digital image frame data may be coupled between image sensors 101 and 102 and image signal processor 112.

[0053] The device may include or be coupled to a sensor hub 150, which is used to engage with sensors to receive data about movement of the device 100, data about the environment surrounding the device 100, and / or other non-camera sensor data. In some embodiments, such non-camera sensors may be integrated into the device 100. One example non-camera sensor is a gyroscope, which is a device configured to measure rotation, orientation, and / or angular velocity to generate motion data. Another example non-camera sensor is an accelerometer, which is a device configured to measure acceleration, and may also be used to determine the speed and distance of travel by appropriately integrating the measured acceleration, and one or more of acceleration, speed, and / or distance may be included in the generated motion data. In some examples, a gyroscope in an electronic image stabilization system (EIS) may be coupled to a sensor hub or directly coupled to an image signal processor 112. In another example, a non-camera sensor may be a Global Positioning System (GPS) receiver.

[0054] The image signal processor 112 can receive image data, such as data used to form image frames. In one embodiment, a region bus connection couples the image signal processor 112 to image sensors 101 and 102 of a first camera and a second camera, respectively. In another embodiment, a wired interface couples the image signal processor 112 to an external image sensor. In another embodiment, a wireless interface couples the image signal processor 112 to image sensors 101 and 102.

[0055] The first camera may include a first image sensor 101 and a corresponding first lens 131. The second camera may include a second image sensor 102 and a corresponding second lens 132. Each of lenses 131 and 132 may have associated autofocus (AF) systems 133 and 134, which adjust lenses 131 and 132 to focus on a specific focal plane at a specific scene depth from image sensors 101 and 102. AF systems 133 and 134 may be assisted by a depth sensor 140. The depth of focus of AF systems 133 and 134 may provide depth information about the image scene to other elements of device 100, such as an ISP 112, via relay data associated with image frames captured by image sensors 101 and 102. Device 100 may perform image processing on image data from a combination of image sensors located within or separate from device 100.

[0056] The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. Lenses 131 and 132 focus light onto image sensors 101 and 102 via one or more apertures for receiving light, one or more shutters for blocking light outside the exposure window, one or more color filter arrays (CFAs) for filtering light outside a specific frequency range, one or more analog front ends for converting analog measurements into digital information, and / or other suitable elements for imaging. The first lens 131 and the second lens 132 may have different fields of view to capture different representations of the scene. For example, the first lens 131 may be an ultra-wide (UW) lens, and the second lens 132 may be a wide (W) lens. The multiple image sensors may include combinations of ultra-wide (high field of view (FOV)), wide, telephoto, and ultra-telephoto (low FOV) sensors. That is, each image sensor can be configured via hardware settings and / or software settings to obtain different but overlapping fields of view. In one configuration, the image sensor is equipped with different lenses having different magnifications resulting in different fields of view. The sensor can be configured such that the UW sensor has a larger field of view (FOV) than the W sensor, the W sensor has a larger FOV than the T sensor, and the T sensor has a larger FOV than the UT sensor. For example, a sensor configured for a wide FOV can capture a field of view in the range of 64-84 degrees, a sensor configured for an ultra-side FOV can capture a field of view in the range of 100-140 degrees, a sensor configured for a long-range FOV can capture a field of view in the range of 10-30 degrees, and a sensor configured for an ultra-long-range FOV can capture a field of view in the range of 1-8 degrees.

[0057] Image signal processor 112 processes image frames captured by image sensors 101 and 102. Although FIG. 1 illustrates device 100 as including two image sensors 101 and 102 coupled to image signal processor 112, any number (e.g., one, two, three, four, five, six, etc.) of image sensors may be coupled to image signal processor 112. In some embodiments, a depth sensor (e.g., depth sensor 140) may be coupled to image signal processor 112 and its output may be processed in a manner similar to that of image sensors 101 and 102. Additionally, any number of additional image sensors or image signal processors may be present for device 100. In some embodiments, image signal processor 112 may execute instructions from memory, such as instructions 108 from memory 106, instructions stored in or included in a separate memory coupled to image signal processor 112, or instructions provided by processor 104. Alternatively, the image signal processor 112 may include specific hardware (e.g., one or more integrated circuits (ICs)) configured to perform one or more operations described in this invention. For example, the image signal processor 112 may include circuitry specifically configured to perform time filtering on two or more image frames, circuitry specifically configured to implement an infinite impulse response (IIR) filter, and / or circuitry specifically configured to apply motion compensation to the image frames.

[0058] In some embodiments, memory 106 may include a non-transient or non-temporary computer-readable medium storing computer-executable instructions 108 to perform all or part of one or more of the operations described herein. In some embodiments, instructions 108 include a camera application (or other suitable application) to be executed by device 100 for generating images or videos. Instructions 108 may also include other applications or programs executed by device 100, such as operating systems and specific applications other than those for generating images or videos. For example, execution of a camera application by processor 104 may cause device 100 to generate images using image sensors 101 and 102 and image signal processor 112. Memory 106 may also be accessed by image signal processor 112 to store processed frames, or may be accessed by processor 104 to obtain processed frames. In some embodiments, device 100 does not include memory 106. For example, device 100 may be circuitry including image signal processor 112, and memory may be external to device 100. Device 100 can be coupled to external memory and is configured to access the memory to write output frames for display or long-term storage. In some embodiments, device 100 is a system-on-a-chip (SoC) that integrates an image signal processor 112, a processor 104, a sensor hub 150, memory 106, and input / output elements 116 into a single package.

[0059] In some embodiments, at least one of the image signal processor 112 or processor 104 executes instructions to perform various operations described herein, including combined operations of temporal filtering. For example, execution of the instructions may instruct the image signal processor 112 to begin or end capturing image frames or sequences of image frames, wherein capturing includes temporal filtering as described in the embodiments herein. In some embodiments, processor 104 may include one or more general-purpose processors capable of executing scripts or instructions (e.g., instructions 108 stored in memory 106) of one or more software programs. For example, processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or video) stored in memory 106. When executing a camera application, processor 104 may be configured to instruct the image signal processor 112 to perform one or more operations with reference to image sensors 101 or 102. For example, the camera application may receive a capture command on which to capture and process video including a sequence of image frames. Temporal filtering may be applied to one or more image frames in the sequence. The camera application may allow enabling and disabling temporal filtering and / or configuring parameters for temporal filtering, such as the number of image frames to be combined in the temporal filtering, parameters for determining the application of the temporal filtering, and / or the absolute or relative number of random resets to be performed for each image frame. The execution of instructions 108 by processor 104 outside the camera application may also cause device 100 to perform any number of functions or operations. In some embodiments, processor 104 may include, in addition to the ability to execute software to cause device 100 to perform multiple functions or operations (e.g., those described herein), ICs or other hardware. In some other embodiments, device 100 may not include processor 104, for example, when all the described functionalities are configured in image signal processor 112.

[0060] In some embodiments, display 114 may include one or more suitable displays or screens that allow user interaction and / or present the user with a preview of an item (such as an image frame captured by image sensors 101 and 102). In some embodiments, display 114 is a touch-sensitive display. I / O element 116 may be or include any suitable mechanism, interface, or device for receiving input (such as commands) from the user and providing output to the user via display 114. For example, I / O element 116 may include (but is not limited to) a graphical user interface (GUI), keyboard, mouse, microphone, speaker, squeezable bezel, one or more buttons (such as a power button), slider, switch, etc.

[0061] Although shown as coupled to each other via processor 104, components (such as processor 104, memory 106, image signal processor 112, display 114, and I / O elements 116) may be coupled to each other in various other arrangements, such as via one or more regional buses, which are not shown for simplicity. Although image signal processor 112 is described as separate from processor 104, image signal processor 112 may be the core of processor 104, which is an application processor unit (APU) included in a system-on-a-chip (SoC) or otherwise included with processor 104. Although device 100 is mentioned in the examples herein for carrying out the present invention, some device elements may not be shown in FIG1 to avoid obscuring the present invention. In addition, other elements, multiple elements, or combinations of elements may be included in suitable devices for carrying out the present invention. Therefore, the present invention is not limited to a particular configuration of devices or elements, including device 100.

[0062] When capturing image frames containing different light sources with different spectra, applying white balance (e.g., automatic white balance (AWB)) to the image frames may produce undesirable results. For example, if the captured image frames include light sources with different light temperatures, the white balance may be inaccurate when the AWB counts statistics across the entire captured image frame because the statistics across the entire frame are averaged across different light sources. Statistical averages across different light sources result in any white balance based on such statistical results leading to a white balance image with undesirable colors. The disadvantages mentioned herein are merely representative and are included to highlight the problems that the inventors have identified and sought to improve with respect to existing devices. The embodiments of the devices described below can address some or all of these disadvantages, as well as other disadvantages known in the art. The improved embodiments of the devices described herein may offer additional benefits beyond those described above and can be used in applications other than those described above.

[0063] In one configuration of device 100, image frames captured from one or more of image sensors 101 and 102 can be modified, for example, by utilizing automatic white balance based on information about different portions of the image frames illuminated by different light sources. For example, infrared measurements can be used to distinguish between a portion of an image frame outdoors and therefore illuminated by sunlight and a portion of an image frame indoors and therefore illuminated by artificial lighting. Automatic white balance can, for example, use infrared measurements to determine the presence of these two different portions and apply different white balances to these two portions. One portion can be adjusted based on a first correlated color temperature (CCT), and another portion can be adjusted based on a second correlated color temperature (CCT). For example, the indoor portion (which may have a lower infrared measurement value) can be white-balanced to a lower CCT, and the outdoor portion (which may have a higher infrared measurement value) can be white-balanced to a higher CCT. These and other embodiments of white balance operation are described with reference to the configurations of Figures 2, 3, 4, 5, 6, 7A, 7B, and 8.

[0064] FIG2 is a flowchart illustrating a method for applying different white balances to different portions of an image frame according to some embodiments of the present invention. Method 200 includes receiving a first image frame and a set of corresponding infrared measurements at block 202. The first image frame may be received from memory, wherein the image frame is stored earlier, such as after the image frame is captured by an image sensor. Alternatively, the first image frame may be received from an image sensor for real-time white balance as part of a procedure for generating a preview image display. Infrared measurements may be received from a file stored in memory, relay data in the first image frame, and / or real-time from an infrared detector together with the first image frame. Infrared measurements may be measured using multiple detectors in parallel, multiple detectors in series, or single detectors in series.

[0065] Method 200 continues to process the first image frame based on corresponding infrared measurements, which may include, for example, the processing described in blocks 204 and 206. At block 204, the processing may include determining one or more portions of the first image frame based on the set of infrared measurements. For example, a first portion and a second portion of the image frame may be identified via infrared measurements. For example, determining multiple portions of the first image frame may include determining an indoor area as the first portion and an outdoor area as the second portion. In another example, determining multiple portions of the first image frame may include determining a first area illuminated by a first light source as the first portion and a second area illuminated by a second light source as the second portion.

[0066] By applying a threshold to infrared measurements to divide them into two or more groups, different portions can be determined based on the infrared measurements. The threshold can be a predetermined value or can be determined based on the infrared measurements of a specific image frame. For example, statistical analysis of infrared measurements can identify a bimodal distribution of the values, and the value associated with each peak can be determined as the value associated with a first and second portion of the image frame. Similarly, statistical analysis can identify multiple peaks or multiple value clusters, and the value associated with each peak or cluster can be considered a different portion.

[0067] The determination of different portions can also, or alternatively, be based on the expected shape of the portions of the scene. For example, pixels in a scene illuminated by the same light source can be expected to be adjacent to other pixels illuminated by the same light source. The determination of portions of an image frame can be based on other criteria, such as thresholding, but then pixels between two regions that meet the criteria are associated with the two regions to form a first portion of the image frame. In some embodiments, this can be performed using a flood fill function that aggregates pixels associated with thresholded infrared measurements into consecutive portions. The flood fill function can use the color of pixels to determine which pixels to include in a portion. For example, flood fill can use the difference in color between adjacent pixels or the rate of color change between adjacent pixels when defining a portion. In one application, pixels near a threshold infrared measurement can only be aggregated if the colors of adjacent pixels are within a threshold difference from each other or the rate of color change between adjacent pixels is less than a threshold amount. Computer vision analysis can also be used to identify features that can help define consecutive regions around a threshold when determining the first, second, third, or more portions of an image frame, features described in more detail below with reference to Figures 7A, 7B, and 8.

[0068] The determination of different parts can also, or alternatively, be based on relay data about the image frame. For example, the time of day and / or the date the image frame was captured can indicate whether an outdoor area of ​​the scene is illuminated by direct sunlight, evening sunlight, or moonlight, each of which can have a different color temperature for white balance. As another example, the location where the image frame was captured can indicate whether the image scene is an outdoor scene, for example, when the location corresponds to the presence of buildings, or can indicate whether the image scene is a mixed scene, for example, when the location corresponds to the presence of buildings.

[0069] The processing of the first image frame in method 200 may include determining a second image frame (e.g., a corrected image frame) at block 206 by applying different white balances to different portions of the first image frame identified at block 204. Different white balances may be applied to different portions based on raw statistics for each portion. In some embodiments, the white balance of block 206 may use automatic white balance (AWB) based on the content of the portion. Raw statistics may be calculated for pixels in the first portion of the image frame, and automatic white balance (AWB) may be applied to the first portion of the image based on the raw statistics calculated for the first portion of the image. In some embodiments, the white balance of block 206 may be based on user input. For example, the user may specify preferred color temperatures for different lighting sources, and apply white balance based on those preferences. As another example, after determining the portion at block 204, the user may be prompted to specify a white balance operation for each portion, such as by presenting the user with a slider specifying temperatures from warm to cool.

[0070] The white balance operation can be repeated for each additional portion of the image frame determined at block 204 based on statistical data associated with the pixels associated with the determined region. White balancing different portions of the image frame individually can improve the appearance of the image by producing more natural colors within the image frame, especially in scenes with multiple lighting sources with different color temperatures. This improves the image quality compared to white balancing the entire image frame based on statistical data averaged across the entire image frame, which can result in color shifts from one part of the image frame to another.

[0071] In some examples, the white balance of block 206 may include applying a first white balance to a first portion by applying a lower correlated color temperature (CCT) to the indoor portion, and applying a second white balance to a second portion by applying a higher correlated color temperature (CCT) to the outdoor portion. The first white balance operation and the second white balance operation may apply different weighting values ​​to the color intensity of pixels in different portions. For example, the white balance of block 206 may include applying a lower CCT to the indoor portion by applying a first set of weighting values ​​to the color intensity of pixels corresponding to the indoor portion, and applying a higher CCT to the outdoor portion by applying a second set of weighting values ​​to the color intensity of pixels corresponding to the outdoor portion.

[0072] An application of an embodiment of the method 200 of FIG2 is illustrated with reference to FIGS. 3, 4, and 5. FIG3 is a block diagram illustrating an example portion of an image frame that can be determined according to some embodiments of the present invention. An image frame 300 and a set of corresponding infrared measurements can be received at block 202 of method 200. At block 204 of method 200, a first portion 302 and a second portion 304 can be determined from within the image frame 300 based on one or more of the infrared measurements, computer vision (CV) analysis, pixel color, and / or other relay data associated with the image frame. An example infrared measurement corresponding to the image frame 300 is shown in FIG4. FIG4 is a block diagram illustrating example infrared measurements that can be used to determine portions of an image frame according to some embodiments of the present invention. A threshold 50 applied to the infrared measurements of FIG4 can be used to determine shaded areas corresponding to the first portion 302 with infrared measurements higher than 50 and the second portion 304 with infrared measurements lower than 50.

[0073] The infrared measurements in Figure 4 can correspond to the photograph shown in Figure 5. Figure 5 is a line drawing illustrating a scene with indoor and outdoor areas according to some embodiments of the present invention, which can be determined as having a first part and a second part based on infrared measurements. Infrared measurements above the threshold used to determine the first part 302 can correspond to a viewport 502 in a scene 500 with mixed lighting sources. Objects in the viewport 502 corresponding to the first part 302 can be illuminated by sunlight, while objects around the viewport 502 inside the building can be illuminated by artificial lighting sources. A different white balance than that of the second part 304 can be applied to the first part 302 to produce a more natural photograph of the mixed lighting source scene 500.

[0074] FIG6 is a flowchart illustrating a method for distinguishing a first portion and a second portion of an image frame based on thresholded infrared measurements according to some embodiments of the present invention. Method 600 includes receiving a set of infrared measurements corresponding to the first image frame at block 602. For each infrared measurement, the value is compared with a threshold at block 604. If the value is below the threshold, method 600 includes associating the pixel in the first image frame corresponding to the infrared measurement with the first portion of the image frame.

[0075] When infrared measurements are performed at a resolution lower than that of the image frame, the association of block 606 can result in more than one pixel being associated with the first portion. For example, if infrared measurements are sampled in an 8×8 array for a 4032×2268 image frame, each infrared measurement value can be associated with a 504×283 block in the image frame. The number of pixels associated with the first portion at block 606 based on the infrared measurement thresholding at block 604 can also vary with the value based on a refinement of the definition of the first portion based on other characteristics (e.g., features identified via computer vision (CV) analysis, relay data, and pixel color, as described above). If the value is higher than the threshold, method 600 includes associating the pixel in the first image frame corresponding to the infrared measurement value with a second portion of the image frame at block 608. After the image frame is segmented into two or more portions at blocks 606 and 608 based on the thresholding at block 604, white balance is applied. At block 610, a first white balance is applied to the first portion; and at block 612, a second white balance is applied to the second portion. A corrected image frame is generated from the white balance portion, and the corrected image frame may have the same or similar resolution as the first image frame.

[0076] Figure 7A illustrates a block diagram for processing image frames using computer vision (CV) analysis and / or multi-point infrared measurements with an image signal processor. Figure 7A is a block diagram illustrating different white balance operations for different portions of an image frame according to some embodiments of the present invention. System 700 may include an image signal processor 112 coupled to an image sensor 101, a multi-point IR sensor 702, and a computer vision processor 704. The multi-point IR sensor 702 may provide infrared measurement values ​​to the image signal processor 112. The image sensor 101 may provide a first image frame to the computer vision processor 704 and the image signal processor 112.

[0077] The computer vision processor 704 can analyze image frames and generate feature maps provided to the image signal processor. The feature map may include a list of detected features and their locations. For example, detected features may include envelopes of objects expected to be outdoors (e.g., trees, umbrellas, bushes, etc.) and envelopes of objects expected to be indoors (e.g., tables, lamps, televisions). The locations of indoor and outdoor objects can be used to determine portions of the image frame corresponding to indoor and outdoor areas. For example, the feature map may be used to confirm and / or refine portions of the image frame identified from infrared measurements. In some embodiments, the computer vision processor 704 can determine the presence and location of a lighting source (e.g., the sun, moon, fluorescent light, table lamp, etc.) within the image frame. The determination of the location and direction of the lighting source can be used to determine a portion of the scene illuminated by that lighting source and to define a first portion and a second portion.

[0078] The image signal processor 112 can process data received from the multi-point IR sensor 702, the image sensor 101, the computer vision processor 704, and / or relay data received from other sensors to generate a corrected image frame from the first image frame. The corrected image frame can have improved coloration, which looks more natural to the human eye in scenes with mixed lighting sources. The image signal processor 112 can perform region identification 722 on the first image frame to determine a first portion, a second portion, and / or additional portions based on feature maps, infrared measurements, and / or relay data. The first and second portions of the image frame can be output from the region identification 722 to the first AWB process 712 and the second AWB process 714, respectively. The region identification 722 is not limited to two portions, but can be extended to include identifying N portions, where N is a configuration value and / or a value determined based on the content of the first image frame, the feature map, the infrared measurements, and / or the relay data. The additional N parts can each be processed by the additional AWB processing 716 and combined to form a corrected image frame.

[0079] AWB processes 712 and 714 can apply white balance by adjusting the weighting of the color of each pixel. One pixel format is RGB, where each pixel has a red value, a green value, and a blue value. During white balance, the weighting of each of the red, green, and blue values ​​can be adjusted to modify the color temperature. In one embodiment of the white balance procedure, a portion of the image frame input to AWB processes 712 and 714 can be grayscale filtered to determine areas that may be gray regions. These regions can then be divided into multiple clusters, and the selected regions can be mapped onto a predetermined coordinate system. The centroid of each cluster can be calculated within the coordinate system. One or more reference luminescent points can be located within the coordinate system. The distance between each centroid of a cluster and each reference luminescent point can be determined. The light source corresponding to each cluster can be estimated, and the final light source can be determined based on the estimation. The white balance gain can be determined based on the light source and the white balance gain applied to multiple portions. Other embodiments of the white balance technique can apply different AWB processes 712 and 714 to the determined portions.

[0080] The results of the first and second portions, processed differently, are combined to produce a corrected image frame. This combination may result in the pixels of the corrected image frame being pixels from the corrected first portion output from AWB processing 712 or pixels from the corrected second portion output from AWB processing 714. The combination may also include blending pixels at the boundary between the first and second portions to reduce the appearance of hard edges caused by the different white balances of the first and second portions. Blending may include a weighted average of pixels within a specific distance of the boundary between the first and second portions with neighboring pixels or other pixels within a specific distance of the boundary. Blending may also, or alternatively, include a weighted average of pixels within a specific distance of the boundary with the original pixels in the first image frame before white balance, alpha blending, and / or anisotropic blending based on scene content (e.g., edges of parts, edges of detected objects, local contrast, and / or features).

[0081] Figure 8 illustrates a method for correcting an image using the system of Figure 7A. Figure 8 is a flowchart illustrating a method for correcting white balance in different portions of an image frame using infrared measurements and computer vision according to some embodiments of the present invention. Method 800 includes receiving a first image frame and a set of corresponding infrared measurements. At block 804, a feature map of the first image frame can be determined using computer vision (CV) analysis. At block 806, a first portion and a second portion can be segmented based on a threshold of the infrared measurements.

[0082] At block 808, the boundaries of the first and second portions within the first image frame can be modified based on feature maps. For example, the boundary of the first portion associated with infrared measurements below a threshold can be extended to include some pixels associated with infrared measurements above a threshold, such as pixels near the boundary between the first and second portions. For example, the boundary can be extended by identifying features that may be illuminated by the same light source across the first and second portions. Extending the boundary of the first or second portion in this feature map-based manner can improve the natural appearance of the lighting of objects in the scene. At block 810, the boundaries of the first and second portions can be modified, similar to that at block 808, but based on pixel color. At block 812, raw statistics of the first and second portions can be analyzed to determine the first and second white balance gains based on the raw statistics. Those white balance gains can be applied at block 814 to adjust the white balance in the first and second portions.

[0083] The embodiments of Figures 7A and 8 describe applying individual white balance to different portions of an image frame. However, in addition to or as an alternative to local white balance for individual portions, a global white balance operation can be performed. Global white balance is described with reference to Figure 7B. Figure 7B is a block diagram illustrating different white balance operations for different portions of an image frame according to some embodiments of the present invention. The global white balance processing of block 750 receives a first image frame and infrared measurements as input at image content weighting 752. Weighting 752 determines the weighted adjustment of the white balance operation, such as the application of automatic white balance (AWB). The weights can be based on infrared measurements, which indicate different characteristics within the first image frame, such as the possible presence of different indoor and outdoor areas within the first image frame. The weights can also be based on other characteristics, or alternatively. Weights can be provided to perform white balance based on weighting 754, which applies a single white balance operation to the entire first image frame based on weighting to produce a corrected image frame. General global white balance can be performed based on a weighted average of the image frame content, where the image frame is divided into grid cells, and each grid cell has a weight that contributes to the final white balance.

[0084] In one or more states, the technique for processing an image frame may include using infrared measurements to enhance an image or perform computational photography, such as any single state or any combination of states described below or in conjunction with one or more other programs or devices described elsewhere herein. In one or more states, performing the method may include receiving a first image frame and a corresponding set of infrared measurements (and / or other data about the image frame). The method may also include processing the first image frame by applying white balance to the first image frame based on the corresponding set of infrared measurements. Additionally, the method may be performed by means of a wireless device including a user equipment (UE). In some embodiments, the means may include at least one processor and memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the means. In some other embodiments, the method may be embedded in program code recorded thereon on a non-transitory computer-readable medium, and the program code may be executed by a computer to cause the computer to perform the operations described herein with reference to the means. In some embodiments, the method may be performed by one or more components configured to perform the operations described herein. In some implementations, a wireless communication method may include one or more operations described herein with reference to the apparatus.

[0085] In the second state, in conjunction with the first state, the method may include one or more of the following: determining a first portion and a second portion of the first image frame based on the set of infrared measurements; applying a first white balance to the first portion of the first image frame in the first portion; and / or applying a second white balance to the second portion of the first image frame in the second portion.

[0086] In the third state sample, in combination with one or more of the first state sample or the second state sample, the method may include: determining a first region illuminated by a first light source as a first part of a first image frame; and / or determining a second region illuminated by a second light source as a second part of the first image frame.

[0087] In the fourth state sample, in combination with one or more of the first to third state samples, the method may include: determining an indoor area as a first part of a first image frame; determining an outdoor area as a second part of a first image frame; applying a lower correlated color temperature (CCT) to the indoor area; and / or applying a higher correlated color temperature (CCT) to the outdoor area.

[0088] In the fifth state sample, in combination with one or more of the first to fourth state samples, the method may include: determining a portion of the first image frame having a corresponding infrared measurement value below a first threshold; and / or determining a portion of the first image frame having a corresponding infrared measurement value above the first threshold.

[0089] In the sixth state sample, in combination with one or more of the first to fifth state samples, the method may include: determining a first pixel region based on pixels in the first region having color values ​​within a threshold distance; and / or determining a second pixel region based on pixels in the second region having color values ​​within a threshold distance.

[0090] In the seventh state sample, in combination with one or more of the first to sixth state samples, the method may include: applying a first white balance including applying a first white balance based on the image content of a first portion; and / or applying a second white balance including applying a second white balance based on the image content of the first portion.

[0091] In the eighth state sample, in combination with one or more of the first to seventh state samples, the method may include performing computer vision analysis of the first image frame to identify a plurality of features in the first image frame; identifying a first set of pixels of the first image frame as a first part based on identifying a first continuous set of corresponding infrared measurements below a first threshold and a first set of a plurality of features corresponding to the first continuous set; and / or identifying a second set of pixels of the first image frame as a second part based on identifying a second continuous set of corresponding infrared measurements above a first threshold and a second set of a plurality of features corresponding to the second continuous set.

[0092] In the ninth state sample, in combination with one or more of the first to eighth state samples, the method may include receiving a set of ToF measurements from the time-of-flight (ToF) sensor as infrared measurements.

[0093] In the tenth state sample, in combination with one or more of the first to ninth state samples, the method may include receiving a set of LiDAR measurements from an optical detection and ranging (LiDAR) sensor as infrared measurements.

[0094] In one or more states, a technique for supporting a device is provided, the device including: a processor; and memory coupled to the processor and storing instructions, which, when executed by the processor, cause the device to perform operations that may include additional states, such as any single state or any combination of states, such as those described below or in conjunction with one or more other programs or devices described elsewhere herein. In an eleventh state, using infrared measurements to support image processing may include means configured to receive a first image frame and a corresponding set of infrared measurements (and / or other data about the image frame). The means may also be configured to process the first image frame by applying white balance to the first image frame based on the corresponding set of infrared measurements. Additionally, the means may perform or operate according to one or more states as described below. In some implementations, the means includes a wireless device (such as a user equipment (UE) or base station (BS)) or an infrastructure element (such as a cloud-based server). In some embodiments, the means may include at least one processor and memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the means. In some other embodiments, the device may include a non-transitory computer-readable medium having program code recorded thereon, the program code being executable by a computer to cause the computer to perform the operations described in the reference device herein. In some embodiments, the device may include one or more components configured to perform the operations described herein.

[0095] In the twelfth state sample, in conjunction with the eleventh state sample, the device can be configured to determine a first portion and a second portion of the first image frame based on the set of infrared measurements; apply a first white balance to the first portion of the first image frame in the first portion; and / or apply a second white balance to the second portion of the first image frame in the second portion.

[0096] In the thirteenth state sample, in conjunction with the eleventh to twelfth state samples, the device can be configured to determine a first region illuminated by a first light source as a first part of a first image frame; and / or determine a second region illuminated by a second light source as a second part of a first image frame.

[0097] In the fourteenth state sample, in combination with the eleventh to thirteenth state samples, the device can be configured to: determine an indoor area as a first part of a first image frame; determine an outdoor area as a second part of a first image frame; apply a lower correlated color temperature (CCT) to the indoor area; and / or apply a higher correlated color temperature (CCT) to the outdoor area.

[0098] In the fifteenth state sample, in conjunction with the eleventh to fourteenth state samples, the device can be configured to: determine a portion of the first image frame having a corresponding infrared measurement value below a first threshold; and / or determine a portion of the first image frame having a corresponding infrared measurement value above the first threshold.

[0099] In the sixteenth state sample, in conjunction with the eleventh to fifteenth state samples, the device can be configured to: determine a first pixel region based on pixels in a first region having color values ​​within a threshold distance; and / or determine a second pixel region based on pixels in a second region having color values ​​within a threshold distance.

[0100] In the seventeenth state sample, in conjunction with the eleventh to sixteenth state samples, the device can be configured to apply a first white balance including applying the first white balance based on the image content of the first portion; and / or apply a second white balance including applying the second white balance based on the image content of the first portion.

[0101] In the eighteenth state sample, in combination with the eleventh to seventeenth state samples, the device can be configured to perform computer vision analysis of the first image frame to identify a plurality of features in the first image frame; based on a first continuous set of infrared measurements corresponding to a first threshold and a first set of features corresponding to the first continuous set, a first set of pixels of the first image frame is identified as a first part; and / or based on a second continuous set of infrared measurements corresponding to a first threshold and a second set of features corresponding to the second continuous set, a second set of pixels of the first image frame is identified as a second part.

[0102] In the nineteenth state sample, in combination with the eleventh to eighteenth state samples, the device can be configured to receive a set of ToF measurements from the Time-of-Flight (ToF) sensor as infrared measurements.

[0103] In the twentieth state sample, in combination with the eleventh to nineteenth state samples, the device can be configured to receive a set of LiDAR measurements from the optical detection and ranging (LiDAR) sensor as infrared measurement values.

[0104] In one or more states, the technology for supporting non-transitory computer-readable media storing instructions may include other states, such as any single state or any combination of states of one or more other programs or devices described below or in conjunction with those described elsewhere herein, wherein the instructions, when executed by the device's processor, cause the device to perform operations. In the twenty-first state, supporting image processing using infrared measurements may include non-transitory computer-readable media storing instructions, wherein the instructions, when executed by the device's processor, cause the device to perform operations including: receiving a first image frame and a corresponding set of infrared measurements (and / or other data about the image frame); and / or processing the first image frame by applying white balance to the first image frame based on the corresponding set of infrared measurements. Additionally, the instructions cause the device to perform or operate according to one or more states as described below. In some implementations, the device includes a wireless device, such as a base station (BS) or user equipment (UE), or includes infrastructure equipment, such as a cloud-based server. In some embodiments, the device may include at least one processor and memory coupled to the processor. In some embodiments, the processor is an image signal processor, which further includes circuitry configured to perform other image functions described herein. The processor may be configured to perform the operations described herein with respect to the apparatus. In some other embodiments, a non-transitory computer-readable medium on which program code is recorded is used, and the program code is executable by a computer to cause the computer to perform the operations described herein with reference to the apparatus.

[0105] In the twenty-second state sample, in conjunction with the twenty-first state sample, the instruction can cause the device to perform the following operations: determining a first part and a second part of a first image frame based on a set of infrared measurements; applying a first white balance to the first part of the first image frame in the first part; and / or applying a second white balance to the second part of the first image frame in the second part.

[0106] In the 23rd state sample, in conjunction with the 21st and 22nd state samples, the instruction can cause the device to perform the following operations: determining a first area illuminated by a first light source as a first part of a first image frame; and / or determining a second area illuminated by a second light source as a second part of a first image frame.

[0107] In the 24th state sample, in conjunction with the 21st to 23rd state samples, the instruction can cause the device to perform the following operations: determining an indoor area as a first part of a first image frame; determining an outdoor area as a second part of the first image frame; applying a lower correlated color temperature (CCT) to the indoor area; and / or applying a higher correlated color temperature (CCT) to the outdoor area.

[0108] In the 25th state sample, in conjunction with the 21st to 24th state samples, the instruction can cause the device to perform the following operations: determine a portion of the first image frame having a corresponding infrared measurement value lower than the first threshold; and / or determine a portion of the first image frame having a corresponding infrared measurement value higher than the first threshold.

[0109] In the 26th state sample, in conjunction with the 21st to 25th state samples, the instruction can cause the device to perform operations, including: determining a first pixel region based on pixels in the first region having color values ​​within a threshold distance; and / or determining a second pixel region based on pixels in the second region having color values ​​within a threshold distance.

[0110] In the twenty-seventh state sample, in conjunction with the twenty-first to twenty-sixth state samples, the instruction can cause the device to perform an operation, the operation including: applying a first white balance including applying a first white balance based on the image content of a first portion; and / or applying a second white balance including applying a second white balance based on the image content of the first portion.

[0111] In the twenty-eighth state sample, in conjunction with the twenty-first to twenty-seventh state samples, the instruction causes the device to perform the following operations: performing computer vision analysis of the first image frame to identify a plurality of features in the first image frame; identifying a first set of pixels of the first image frame as a first part based on a first continuous set of infrared measurements corresponding to the first threshold and a first set of features corresponding to the first continuous set; and / or identifying a second set of pixels of the first image frame as a second part based on a second continuous set of infrared measurements corresponding to the first threshold and a second set of features corresponding to the second continuous set.

[0112] In the twenty-ninth state sample, in conjunction with the twenty-first to twenty-eighth state samples, the instruction can enable the device to perform an operation including receiving a set of ToF measurements from the Time-of-Flight (ToF) sensor as infrared measurement values.

[0113] In the thirtieth state sample, in conjunction with the twenty-first to twenty-ninth state samples, the instruction can enable the device to perform an operation including receiving a set of LiDAR measurements from the light detection and ranging (LiDAR) sensor as infrared measurement values.

[0114] In one or more states, techniques for supporting the processing of image frames based on infrared measurements of a scene captured in the image frame may be implemented in or by a device, the device including a first image sensor configured with a first field of view, a processor coupled to the first image sensor, and memory coupled to the processor. The processor is configured to perform steps including additional states, such as any single state or any combination of states described below, or in combination with one or more other programs or devices described elsewhere herein. In a thirty-first state, supporting image processing may include a device configured to receive the first image frame captured at a first time. The device may also include a multi-point IR detector and is further configured to receive a set of corresponding infrared measurements from the multi-point IR detector, and to process the first image frame by applying white balance to the first image frame via the set of corresponding infrared measurements using a processor (e.g., an image signal processor). Additionally, the device may perform or operate according to one or more states as described below. In some embodiments, the device includes a wireless device, such as a base station (BS) or user equipment (UE), or infrastructure equipment, such as a cloud-based server. In some embodiments, the device may include at least one processor and memory coupled to the processor, wherein the processor may be configured to perform the operations described herein with respect to the device. In some other embodiments, the device may include a non-transitory computer-readable medium on which program code is recorded, and the program code may be executed by the device to cause the device to perform the operations described herein with reference to the device. In some embodiments, the device may include one or more components configured to perform the operations described herein.

[0115] In the thirty-second state sample, in conjunction with the thirty-first state sample, the processor is further configured to determine a first portion and a second portion of the first image frame based on the set of infrared measurements; apply a first white balance to the first portion of the first image frame in the first portion; and / or apply a second white balance to the second portion of the first image frame in the second portion.

[0116] In the thirty-third state sample, in conjunction with one or more of the thirty-first to thirty-second state samples, the processor is further configured to determine the first region illuminated by the first light source as the first part of the first image frame; and / or determine the second region illuminated by the second light source as the second part of the first image frame.

[0117] In the thirty-fourth state sample, in combination with one or more of the thirty-first to thirty-third state samples, the processor is further configured to determine the indoor area as the first part of the first image frame; determine the outdoor area as the second part of the first image frame; apply a lower correlated color temperature (CCT) to the indoor area; and / or apply a higher correlated color temperature (CCT) to the outdoor area.

[0118] In the thirty-fifth state sample, in conjunction with one or more of the thirty-first to thirty-fourth state samples, the processor is further configured to: determine a portion of the first image frame having a corresponding infrared measurement value below the first threshold; and / or determine a portion of the first image frame having a corresponding infrared measurement value above the first threshold.

[0119] In the thirty-sixth state sample, in combination with one or more of the thirty-first to thirty-fifth state samples, the processor is further configured to determine the first pixel region based on pixels in the first region having color values ​​within a threshold distance; and / or determine the second pixel region based on pixels in the second region having color values ​​within a threshold distance.

[0120] In the thirty-seventh state sample, in conjunction with one or more of the thirty-first to thirty-sixth state samples, the processor is further configured to apply a first white balance including applying the first white balance based on the image content of the first portion; and / or apply a second white balance including applying the second white balance based on the image content of the first portion.

[0121] In the thirty-eighth state sample, in combination with one or more of the thirty-first to thirty-seventh state samples, the processor is further configured to perform computer vision analysis of the first image frame to identify a plurality of features in the first image frame; based on a first continuous set of infrared measurements corresponding to the first threshold and a first set of features corresponding to the first continuous set, a first set of pixels of the first image frame is identified as a first part; and / or based on a second continuous set of infrared measurements corresponding to the first threshold and a second set of features corresponding to the second continuous set, a second set of pixels of the first image frame is identified as a second part.

[0122] In the thirty-ninth state sample, in combination with one or more of the thirty-first to thirty-eighth state samples, the device also includes a time-of-flight (ToF) measurement device as a multi-point IR detector, and the processor is further configured to receive a set of ToF measurements as infrared measurements from the time-of-flight (ToF) sensor.

[0123] In the fortieth state sample, in combination with one or more of the thirty-first to thirty-ninth state samples, the device also includes a light detection and ranging (LiDAR) device as a multi-point IR detector, and the processor is further configured to receive a set of LiDAR measurements from the light detection and ranging (LiDAR) sensor as infrared measurement values.

[0124] Those skilled in the art will understand that information and signals can be represented using any of a variety of different techniques and skills. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the foregoing specification can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or light particles, or any combination thereof.

[0125] The elements, functional blocks, and modules described herein with respect to Figures 1 and 7A include processors, electronic devices, hardware devices, electronic components, logic circuits, memory, software code, firmware code, and other examples, or any combination thereof. Furthermore, the features discussed herein may be implemented via dedicated processor circuitry, via executable instructions, or a combination thereof.

[0126] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various illustrative elements, blocks, modules, circuits, and steps have been described above in general terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention. Those skilled in the art will also readily recognize that the order or combination of elements, methods, or interactions described herein are merely examples, and various forms of elements, methods, or interactions within this scope may be combined or performed in ways different from those shown and described herein.

[0127] The various illustrative logics, logic blocks, modules, circuits, and algorithms described in conjunction with the implementation schemes disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. The interchangeability of hardware and software has been largely described in the functionality section and in the various illustrative elements, blocks, modules, circuits, and programs described above. Whether this functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the entire system.

[0128] Hardware and data processing apparatuses for implementing the various illustrative logics, logic blocks, modules, and circuits described herein can be implemented or executed using general-purpose single-core or multi-core processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable or fixed logic devices, individual gate or transistor logic, individual hardware elements, or any combination thereof. A general-purpose processor can be a microprocessor or any processor, controller, microcontroller, or state machine. In some embodiments, the processor can be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors incorporating DSP cores, or any other such configuration. In some embodiments, specific programs and methods can be executed by circuitry specific to a given function.

[0129] In one or more embodiments, the described functions can be implemented using hardware, digital electronic circuits, computer software, firmware (including the structures disclosed in this specification and their equivalents) or any combination thereof. Embodiments of the subject matter described in this specification can also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device.

[0130] If implemented in software, the functionality can be stored as one or more instructions or codes on or transmitted via a computer-readable medium. The program of the methods or algorithms disclosed herein can be implemented in a processor-executable software module that can reside on a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, wherein communication media includes any media that can be enabled to transfer a computer program from one location to another. Storage media can be any available media accessible by a computer. By way of example, and not limitation, such computer-readable media can include random access memory (RAM), read-only memory (ROM), electronically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. Furthermore, any connection can be appropriately referred to as a computer-readable medium. As used herein, magnetic disks and optical disks include CDs, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs, wherein magnetic disks typically reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media. Furthermore, the operation of methods or algorithms may reside as one or any combination or set of codes and instructions on machine-readable and computer-readable media, which may be incorporated into computer program products.

[0131] Various modifications to the implementations described herein will be apparent to those skilled in the art, and the general principles defined herein can be applied to other implementations without departing from the spirit or scope of this document. Therefore, the claim is not intended to be limited to the implementations shown herein, but rather to be consistent with the widest scope of this document, the principles disclosed herein, and the novel features.

[0132] In addition, those skilled in the art will readily understand that the terms "upper" and "lower" are sometimes used for the convenience of describing the drawings and indicate relative positions corresponding to the orientation of the drawings on a properly oriented page, and may not reflect the proper orientation of any implemented device.

[0133] Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable subgroup combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may be for subgroup combinations or variations of subgroup combinations.

[0134] Similarly, although operations are illustrated in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or to perform all illustrated operations to achieve the desired result. Furthermore, the drawings may schematically illustrate one or more example programs in the form of flowcharts. However, other operations not illustrated may be incorporated into the schematically illustrated example programs. For example, one or more additional operations may be performed before, after, simultaneously with, or between any illustrated operations. In some cases, multiplexing and parallel processing may be advantageous. Furthermore, the separation of various system elements in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program elements and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other embodiments are within the scope of the appended claims. In some cases, the actions described in the claims may be performed in a different sequence and still achieve the desired result.

[0135] As used herein (including the claims), the term "or," when used in a list of two or more items, means that any one of the listed items may be used alone, or any combination of two or more of the listed items may be used. For example, if a composition is described as containing elements A, B, or C, the composition may contain only A; only B; only C; a combination of A and B; a combination of A and C; a combination of B and C; or a combination of A, B, and C. Furthermore, as used herein, including in the claims, "or" in a list of items beginning with "at least one of..." indicates a separate list, such that a list such as "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any one of these or any combination thereof. As will be understood by those generally skilled in the art, the term "substantially" is defined as mostly, but not necessarily entirely, of the specified content (and includes the specified content; for example, substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel). In any disclosed implementation, the term "substantially" may be replaced by the specified "within [percentage]", where the percentage includes 0.1%, 1%, 5%, or 10%.

[0136] The prior description of this application is provided to enable anyone skilled in the art to make or use this application. Various modifications to the content of this application will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the spirit or scope of this application. Therefore, the content of this application is not intended to be limited to the examples and designs described herein, but rather to be consistent with the widest scope of the principles and novel features disclosed herein. [Simplified Explanation of the Diagram]

[0031] A further understanding of the nature and advantages of this invention can be achieved by referring to the following figures. In the figures, similar elements or features may have the same element symbol. Furthermore, various elements of the same type may be distinguished by a dash following the element symbol and a second mark distinguishing them from similar elements. If only the first element symbol is used in the specification, the specification applies to any similar element having the same first element symbol, regardless of the second element symbol.

[0032] Figure 1 is a block diagram of one or more computing devices configured to perform the example techniques described in this invention.

[0033] FIG2 is a flowchart illustrating a method of applying different white balances to different parts of an image frame according to some embodiments of the present invention.

[0034] FIG3 is a block diagram illustrating an example portion of an image frame that may be determined according to some embodiments of the present invention.

[0035] Figure 4 is a block diagram illustrating example infrared measurements that can be used to determine portions of an image frame according to some embodiments of the present invention.

[0036] Figure 5 is a line drawing illustrating a scene with indoor and outdoor areas according to some embodiments of the present invention, which can be determined to have a first part and a second part based on infrared measurements.

[0037] Figure 6 is a flowchart illustrating a method for distinguishing a first part and a second part of an image frame based on thresholded infrared measurements according to some embodiments of the present invention.

[0038] Figure 7A is a block diagram illustrating different white balance operations on different parts of an image frame according to some embodiments of the present invention.

[0039] Figure 7B is a block diagram illustrating different white balance operations on different parts of an image frame according to some embodiments of the present invention.

[0040] Figure 8 is a flowchart illustrating a method for correcting white balance in different parts of an image frame using infrared measurements and computer vision according to some embodiments of the present invention.

[0041] The same element symbols and names in the various figures indicate the same elements. [Biomaterial Storage]

[0138] Domestic storage information (please note in order of storage institution, date, and number): None. International storage information (please note in order of storage country, institution, date, and number): None.

Claims

1. A method for image processing, comprising: A first image frame and a set of corresponding infrared measurements are received; the first image frame is divided into a plurality of parts based on the equal values ​​included in the set of infrared measurements, the plurality of parts including a first part illuminated by a first light source and a second part illuminated by a second light source, wherein the first part has infrared measurements of the set of corresponding infrared measurements below a first threshold, and the second part has infrared measurements of the set of corresponding infrared measurements above the first threshold; and based on the division, the first image frame is processed as a whole by applying a first white balance to the first part and a second white balance to the second part.

2. According to the method of request item 1, wherein: The division includes: defining an indoor area as the first portion of the first image frame; and defining an outdoor area as the second portion of the first image frame; applying the first white balance to the first portion includes applying a lower correlated color temperature (CCT) to the indoor area; and applying the second white balance to the second portion includes applying a higher correlated color temperature (CCT) to the outdoor area.

3. According to the method of request item 1, wherein: The pixels in the first part have color values ​​within a first threshold distance; and the pixels in the second part have color values ​​within a second threshold distance.

4. According to the method of request item 1, wherein: Applying the first white balance includes applying the first white balance based on the image content of the first portion; and applying the second white balance includes applying the second white balance based on the image content of the first portion.

5. According to the method of request item 1, the partition includes: Perform a computer vision analysis of the first image frame to identify a plurality of features in the first image frame; Based on a first continuous set of infrared measurements corresponding to values ​​below the first threshold and a first set of features corresponding to the first continuous set, a first set of pixels of the first image frame is identified as the first part of the first image frame; and based on a second continuous set of infrared measurements corresponding to values ​​above the first threshold and a second set of features corresponding to the second continuous set, a second set of pixels of the first image frame is identified as the second part of the first image frame.

6. According to the method of request item 1, wherein the processing includes: The white balance weight of the first image frame is determined based on the set of infrared measurements. And apply the corresponding white balance based on these white balance weights.

7. According to the method of request item 1, the set of receiving the corresponding infrared measurements includes: Receive a set of LiDAR measurements from a light detection and ranging (LiDAR) sensor; or receive a set of ToF measurements from a time-of-flight (ToF) sensor.

8. An apparatus for image processing, comprising: One processor; The device also includes a memory coupled to the processor and storing instructions that, when executed by the processor, cause the device to perform operations including: receiving a first image frame and a set of corresponding infrared measurements; dividing the first image frame into a plurality of portions based on the values ​​included in the set of infrared measurements, the plurality of portions including a first portion illuminated by a first light source and a second portion illuminated by a second light source, wherein the first portion has infrared measurements of the set of corresponding infrared measurements below a first threshold, and the second portion has infrared measurements of the set of corresponding infrared measurements above the first threshold; and processing the first image frame based on the division by applying a first white balance to the first portion and a second white balance to the second portion.

9. The device according to request item 8, wherein: The division includes: defining an indoor area as the first portion of the first image frame; and defining an outdoor area as the second portion of the first image frame; applying the first white balance to the first portion includes applying a lower correlated color temperature (CCT) to the indoor area; and applying the second white balance to the second portion includes applying a higher correlated color temperature (CCT) to the outdoor area.

10. The device according to request item 8, wherein: The pixels in the first part have color values ​​within a first threshold distance; and the pixels in the second part have color values ​​within a second threshold distance.

11. The device according to request item 8, wherein: Applying the first white balance includes applying the first white balance based on the image content of the first portion; and applying the second white balance includes applying the second white balance based on the image content of the first portion.

12. The device according to request item 8, wherein the partition includes: Perform a computer vision analysis of the first image frame to identify a plurality of features in the first image frame; Based on a first continuous set of infrared measurements corresponding to values ​​below the first threshold and a first set of features corresponding to the first continuous set, a first set of pixels of the first image frame is identified as the first part of the first image frame; and based on a second continuous set of infrared measurements corresponding to values ​​above the first threshold and a second set of features corresponding to the second continuous set, a second set of pixels of the first image frame is identified as the second part of the first image frame.

13. The device according to claim 8, wherein processing the first image frame includes: The white balance weight of the first image frame is determined based on the set of infrared measurements. And apply the corresponding white balance based on the white balance weight.

14. The device according to claim 8, wherein the set of receiving the corresponding infrared measurements includes: Receive a set of LiDAR measurements from a light detection and ranging (LiDAR) sensor; or receive a set of ToF measurements from a time-of-flight (ToF) sensor.

15. The device according to claim 8, wherein at least two of the plurality of portions have different numbers of pixels from each other.

16. A non-transitory computer-readable medium storing instructions, which, when executed by a processor of a device, cause the device to perform operations including: receiving a first image frame and a set of corresponding infrared measurements; dividing the first image frame into a plurality of portions based on the values ​​included in the set of infrared measurements, the plurality of portions including a first portion illuminated by a first light source and a second portion illuminated by a second light source, wherein the first portion has infrared measurements of the set of corresponding infrared measurements below a first threshold, and the second portion has infrared measurements of the set of corresponding infrared measurements above the first threshold; and processing the first image frame as a whole by applying a first white balance to the first portion and a second white balance to the second portion based on the division.

17. The non-transitory computer-readable media pursuant to claim 16, wherein: The division includes: defining an indoor area as the first portion of the first image frame; and defining an outdoor area as the second portion of the first image frame; applying the first white balance to the first portion includes applying a lower correlated color temperature (CCT) to the indoor area; and applying the second white balance to the second portion includes applying a higher correlated color temperature (CCT) to the outdoor area.

18. The non-transitory computer-readable media according to claim 16, wherein: The pixels in the first part have color values ​​within a first threshold distance; and the pixels in the second part have color values ​​within a second threshold distance.

19. The non-transitory computer-readable medium pursuant to claim 16, wherein: Applying the first white balance includes applying the first white balance based on the image content of the first portion; and applying the second white balance includes applying the second white balance based on the image content of the first portion.

20. The non-transitory computer-readable medium according to claim 16, wherein the division includes: Perform a computer vision analysis of the first image frame to identify a plurality of features in the first image frame; Based on a first continuous set of infrared measurements corresponding to values ​​below the first threshold and a first set of features corresponding to the first continuous set, a first set of pixels of the first image frame is identified as the first part of the first image frame; and based on a second continuous set of infrared measurements corresponding to values ​​above the first threshold and a second set of features corresponding to the second continuous set, a second set of pixels of the first image frame is identified as the second part of the first image frame.

21. The non-transitory computer-readable medium according to claim 16, wherein processing the first image frame includes: The white balance weight of the first image frame is determined based on the set of infrared measurements. And apply the corresponding white balance based on the white balance weight.

22. The non-transitory computer-readable medium according to claim 16, wherein the set of receiving the corresponding infrared measurements includes: Receive a set of LiDAR measurements from a light detection and ranging (LiDAR) sensor; or receive a set of ToF measurements from a time-of-flight (ToF) sensor.

23. An apparatus for image processing, comprising: A first image sensor; A first multi-point infrared (IR) detector; A processor coupled to the first image sensor and the first multi-point IR detector; and a memory coupled to the processor, wherein the processor is configured to perform the following steps: receiving a first image frame from the first image sensor and a set of corresponding infrared measurements from the first multi-point IR detector; dividing the first image frame into a plurality of portions based on the equal values ​​included in the set of infrared measurements, the plurality of portions including a first portion illuminated by a first light source and a second portion illuminated by a second light source, wherein the first portion has infrared measurements of the set of corresponding infrared measurements below a first threshold, and the second portion has infrared measurements of the set of corresponding infrared measurements above the first threshold; and processing the first image frame as a whole by applying a first white balance to the first portion and a second white balance to the second portion based on the division.

24. The device according to request item 23, wherein: The division includes: defining an indoor area as the first portion of the first image frame; and defining an outdoor area as the second portion of the first image frame; applying the first white balance to the first portion includes applying a lower correlated color temperature (CCT) to the indoor area; and applying the second white balance to the second portion includes applying a higher correlated color temperature (CCT) to the outdoor area.

25. The device according to request item 23, wherein: The pixels in the first part have color values ​​within a first threshold distance; and the pixels in the second part have color values ​​within a second threshold distance.

26. The device according to request item 23, wherein: Applying the first white balance includes applying the first white balance based on the image content of the first portion; and applying the second white balance includes applying the second white balance based on the image content of the first portion.

27. The device according to claim 23, wherein the partition includes: Perform a computer vision analysis of the first image frame to identify a plurality of features in the first image frame; Based on a first continuous set of infrared measurements corresponding to values ​​below the first threshold and a first set of features corresponding to the first continuous set, a first set of pixels of the first image frame is identified as the first part of the first image frame; and based on a second continuous set of infrared measurements corresponding to values ​​above the first threshold and a second set of features corresponding to the second continuous set, a second set of pixels of the first image frame is identified as the second part of the first image frame.

28. The device according to claim 23, wherein the multi-point IR detector includes a time-of-flight (ToF) measurement device, wherein the set of receiving the corresponding infrared measurements includes a set of ToF measurements received from the time-of-flight measurement device.

29. The device according to claim 23, wherein the multi-point IR detector includes a light detection and ranging (LiDAR) device, wherein the set of receiving the corresponding infrared measurements includes a set of LiDAR measurements received from the light detection and ranging (LiDAR) device.

Images