Image tone adjustment

Abstract

One embodiment provides a method comprising: receiving an image of a scene at a hue adjustment system; using a hue adjustment system to identify a light source that provides illumination within the image; and adjusting the hue of the image based on the characteristics of the light source using a hue adjustment system. Other aspects are claimed and described.

Description

Technical Field

[0001] The present disclosure relates to methods, systems, and products for image tone adjustment. Background Art

[0002] Video conferencing or video calling is becoming common. Within a video conference or video call, images are captured rapidly and continuously by an image capture device for transmission to other participants in the video conference or video call. Additionally, many people use image capture devices to take images or videos. The image capture device performs processing on the images or frames within the video such that the images appear like real-world objects being depicted within the images or frames. In the case of a video conference or video call, the processed images are placed within a video stream or video feed along with other processed frames and transmitted to other participants within the video conference or video call. Technology enables such processing to occur very rapidly, such that the video conference or video call appears to be live or substantially real-time. In the case where a user takes an image or video independent of a video conference or video call, the image or frame can be processed and then stored in an accessible data storage location. Summary of the Invention

[0003] In summary, one aspect provides a method that includes: receiving an image of a scene at a tone adjustment system; using the tone adjustment system to identify a light source providing illumination within the image; and using the tone adjustment system to adjust the tone of the image based on characteristics of the light source.

[0004] Another aspect provides a system that includes: a processor; a memory device storing instructions that, when executed by the processor, cause the system to perform the following operations: receive an image of a scene at a tone adjustment system; use the tone adjustment system to identify a light source providing illumination within the image; and use the tone adjustment system to adjust the tone of the image based on characteristics of the light source.

[0005] Yet another aspect provides a product that includes: a computer-readable storage device storing executable code that, when executed by a processor, causes the product to perform the following operations: receive an image of a scene at a tone adjustment system; use the tone adjustment system to identify a light source providing illumination within the image; and use the tone adjustment system to adjust the tone of the image based on characteristics of the light source.

[0006] The foregoing is an overview and thus may contain simplifications, generalizations, and omissions of details; accordingly, those skilled in the art will recognize that the Summary of the Invention is merely illustrative and is not intended to be limiting in any way.

[0007] For a better understanding of the embodiments and their other and further features and advantages, reference is made to the following description taken in conjunction with the accompanying drawings. The scope of the invention will be pointed out in the appended claims. Description of the Drawings

[0008] Figure 1 An example of the circuit system of the information processing device is shown.

[0009] Figure 2 Another example of the circuit system of the information processing device is shown.

[0010] Figure 3 An example method for adjusting the hue of an image based on the characteristics of a light source that provides illumination within the image is shown. Detailed Description of Embodiments

[0011] It will be readily understood that, in addition to the described example embodiments, the components of the embodiments generally described and shown in the accompanying drawings herein can be arranged and designed in a wide variety of different configurations. Accordingly, the following more detailed description of the example embodiments shown in the drawings is not intended to limit the scope of the claimed embodiments, but merely represents example embodiments.

[0012] Throughout this specification, references to "one embodiment" or "an embodiment" (etc.) mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the phrases "in one embodiment" or "in an embodiment" that appear in various places throughout this specification are not necessarily all referring to the same embodiment.

[0013] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments. However, those skilled in the relevant art will recognize that various embodiments can be practiced without one or more of the specific details, or using other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring the description.

[0014] The processing performed on an image or video frame is typically an automatic process. The image capture device attempts to process the image in a way that makes the image or frame appear to accurately capture the real-world scene and the objects within the scene. To perform such processing, the image capture device not only captures the image or frame but also measures different values of the environment or the image capture device while capturing the image or frame. An example of such captured sensor information is the color values of red, green, and blue within the environment. The image capture device includes a sensor array that captures these sensor values while capturing the image or frame. The image capture device then obtains these values and recombines the colors so that the output image or frame appears correct.

[0015] Since different light sources (which can provide illumination for the scene) can have different color values, one technique employed by image capture devices is automatic white balance (AWB). Automatic white balance changes the hue of the image presented to the user to account for the different color temperatures of the light sources. For example, some light sources can have an orange hue, a blue hue, a white hue, etc. The automatic white balance of the image capture device adjusts the red-green-blue (RGB) values of the image so that the image appears to more accurately represent a neutral-illumination scene captured within the image or frame. However, the image capture device makes assumptions about what the image should look like to determine the RGB values of the different channels when performing automatic white balance. These assumptions are typically made during the manufacture of the camera device sensor and are based on the curves between the intensity values across all color channels and the luminance level. These curves typically keep the RGB values within a certain range for a given overall intensity value.

[0016] The basic method is based on the "gray world" theory, which assumes that across the entire frame, the average color should be gray, i.e., a balance of blue:yellow, green:magenta, and red:cyan. More sophisticated methods are based on the retinex theory, which identifies a correction ratio for the brightest part of the frame and then linearly scales the entire frame. However, these techniques assume the color averages that should be identified and then applied across the entire frame and also rely on the color curves defined during the manufacture of the camera device sensor. Thus, if a part of the image has a high value for a particular color, that color will be compensated across the entire image even if it causes the image to not look like the real scene. Some techniques that help account for differences in light source color temperature require the use of separate devices. For example, a separate photometer can be used to directly measure the color temperature of the light emitter. However, this requires special hardware and that the photometer be able to communicate directly with the image capture device.

[0017] Some professional photographers may also carry a card or other item with a known white or gray color and a uniform spectral response. The photographer can use this item to determine the color temperature of the illuminant and manually adjust the white balance. However, this requires a special item and the photographer to set the image capture device based on this item. In the case where the light source or color temperature changes, or if the light source or color temperature changes, the photographer must use the special item to reset the image capture device. This takes additional time and requires the photographer to identify when the color temperature has changed.

[0018] Accordingly, the described systems and methods provide techniques for adjusting the hue of an image based on the characteristics of the light source that provides illumination within the image. The hue adjustment system receives an image. For example, in response to a user taking a picture or video, due to the activation of video conferencing or video calling software, and / or any other input that activates the image capture device, the image can be captured by the image capture device. The image can also be received from components other than the image capture device (e.g., a data storage location, a downstream processor of the image capture device, etc.). The hue adjustment system identifies the light source that provides illumination within the image. The light source may or may not be visible within the image, but the illumination caused by the light source will be visible within the image. The light source can be any type of illuminant, including non-natural light sources (e.g., a display, a light bulb, etc.) and natural light sources (e.g., sunlight, moonlight, stars, etc.). Some of the light sources can be controlled by the hue adjustment system, which means the system can provide instructions to change the illumination. Other light sources may not be controllable by the system.

[0019] When identifying the light source, the system can also identify the characteristics of the light source. These characteristics can help determine the color temperature provided by the light source. Then, the hue adjustment system can adjust the hue of the image based on the characteristics of the light source. For example, if the system determines that the light source provides red light, the system can adjust the hue of the image to process the red hue while ignoring the normal curve used to adjust the red hue of the image. In a conventional system, due to the normal balance between red and cyan, the adjustment for that red hue may result in an image with an overall cyan hue.

[0020] Accordingly, the system provides a technical improvement over traditional methods for tone adjustment within an image. The tone adjustment system does not rely on the color curves established during the manufacture of the image capture device sensor, but can identify the light source providing illumination within the image and adjust the image based on the characteristics of that particular light source. This provides a more accurate technique for adjusting the tone of an image compared to conventional techniques that rely on color curves and then apply a color adjustment ratio across the entire image. Additionally, the described system and techniques do not rely on the use of auxiliary items to correctly identify the hue of the illuminant or light source, thus providing a system that is simpler and requires less user input compared to conventional techniques. The described system and method improve the functionality of the image capture device by creating a more accurate and consistent white balance technique without requiring more user input than conventional techniques.

[0021] The illustrated example embodiments will be best understood by reference to the accompanying drawings. The following description is intended only as an example and simply shows certain example embodiments.

[0022] Although various other circuits, circuit systems or components may be used in an information processing device, with respect to the smart phone and / or tablet circuit system 100, Figure 1 the example shown includes, for example, a system-on-a-chip design found in a tablet or other mobile computing platform. The software and the processor are combined in a single chip 110. As is well known in the art, a processor includes internal arithmetic units, registers, cache memories, buses, input / output (I / O) ports, etc. The internal buses, etc. depend on different vendors, but essentially all peripheral devices (120) can be attached to the single chip 110. The circuit system 100 combines the processor, memory control and I / O controller hub all into a single chip 110. Additionally, this type of system 100 generally does not use Serial Advanced Technology Attachment (SATA) or Peripheral Component Interconnect (PCI) or Low Pin Count (LPC). For example, common interfaces include Secure Digital Input / Output (SDIO) and Inter-Integrated Circuit (I2C).

[0023] There is a power management chip 130 such as a Battery Management Unit (BMU) that manages the power supplied, for example, via a rechargeable battery 140, which can be recharged by connection to a power source (not shown). In at least one design, a single chip, such as 110, is used to provide Basic Input / Output System (BIOS)-like functionality and Dynamic Random Access Memory (DRAM) memory.

[0024] System 100 generally includes one or more of a wireless wide area network (WWAN) transceiver 150 and a wireless local area network (WLAN) transceiver 160 for connecting to various networks such as telecommunications networks and wireless Internet devices such as access points. Additionally, a device 120 such as a wireless communication device, an external storage device, etc. is generally included. System 100 generally includes a touch screen 170 for data input and display / rendering. System 100 also generally includes various storage devices such as a flash memory 180 and a synchronous dynamic random access memory (SDRAM) 190.

[0025] Figure 2 A block diagram depicting another example of an information processing device circuit, circuitry, or component. Figure 2 The example depicted in may correspond to a computing system such as a personal computer or other device. As is apparent from the description herein, embodiments may include Figure 2 other features of the example shown in or only some of the features.

[0026] Figure 2 Examples of include the so-called chipset 210 (a group of integrated circuits or chips, a chipset, that work together), and the chipset 210 has an architecture that may vary according to the manufacturer. The architecture of the chipset 210 includes a core and a memory control group 220 and an I / O controller hub 250 that exchange information (e.g., data, signals, commands, etc.) via a direct management interface (DMI) 242 or a link controller 244. In Figure 2 , the DMI 242 is a chip-to-chip interface (sometimes referred to as the link between the "north bridge" and the "south bridge"). The core and memory control group 220 includes one or more processors 222 (e.g., single-core or multi-core) and a memory controller hub 226 that exchange information via a front-side bus (FSB) 224; note that the components of group 220 may be integrated in a chip that replaces the conventional "north bridge"-type architecture. One or more processors 222 include internal arithmetic units, registers, cache memories, buses, I / O ports, etc. as are well known in the art.

[0027] In Figure 2In it, the memory controller hub 226 interfaces with the memory 240 (e.g., provides support for a type of random access memory (RAM) that can be referred to as "system memory" or "memory"). The memory controller hub 226 also includes a Low Voltage Differential Signaling (LVDS) interface 232 for a display device 292 (e.g., cathode ray tube (CRT), flat panel, touch screen, etc.). Block 238 includes some technologies that can be supported via the Low Voltage Differential Signaling (LVDS) interface 232 (e.g., Serial Digital Video, High-Definition Multimedia Interface / Digital Visual Interface (HDMI / DVI), DisplayPort). The memory controller hub 226 also includes a Peripheral Component Interconnect Express (PCI-E) interface 234 that can support a discrete graphics card 236.

[0028] In Figure 2 it, the I / O hub controller 250 includes a Serial ATA (SATA) interface 251 (e.g., for a hard disk drive (HDD), solid state drive (SSD), etc., 280), a PCI-E interface 252 (e.g., for a wireless connection 282), a Universal Serial Bus (USB) interface 253 (e.g., for devices 284 such as a digitizer, keyboard, mouse, imaging device, telephone, microphone, storage device, other connected devices, etc.), a network interface 254 (e.g., Local Area Network (LAN)), a General Purpose Input / Output (GPIO) interface 255, an LPC interface 270 (for an Application Specific Integrated Circuit (ASIC) 271, a Trusted Platform Module (TPM) 272, a Super I / O 273, a Firmware Hub 274, BIOS support 275, and various types of memory 276 such as Read Only Memory (ROM) 277, Flash Memory 278, and Non-Volatile Random Access Memory (NVRAM) 279), a power management interface 261, a clock generator interface 262, an audio interface 263 (e.g., for a speaker 294), a Time Control Operation (TCO) interface 264, a System Management Bus interface 265, and a Serial Peripheral Interface (SPI) Flash 266 that can include a BIOS 268 and boot code 290. The I / O hub controller 250 can include Gigabit Ethernet support.

[0029] Upon power-up, the system can be configured to execute the boot code 290 for the BIOS 268 stored in the SPI Flash 266 and then process data under the control of one or more operating systems and application software (e.g., stored in the system memory 240). The operating system can be stored in any of various locations and is accessed, for example, according to the instructions of the BIOS 268. As described herein, the device can include fewer or more features compared to the features shown in the Figure 2 system.

[0030] For example Figure 1 orFigure 2 The information processing device circuitry outlined in can be used in devices such as tablet computers, smart phones, typically personal computer devices, and / or electronic devices, which can be used in devices or systems for capturing images, processing images, analyzing images, transmitting images, etc. For example, Figure 1 the circuitry outlined in can be implemented in tablet computer or smart phone embodiments, while Figure 2 the circuitry outlined in can be implemented in personal computer embodiments.

[0031] Figure 3 An example method for adjusting the hue of an image based on the characteristics of a light source providing illumination within the image is shown. The method can be implemented on a system including a processor, a memory device, an output device (e.g., a display device, a printer, etc.), an input device (e.g., a keyboard, a touch screen, a mouse, a microphone, a sensor, a biometric scanner, etc.), an image capture device, and / or other components such as those combined with Figure 1 and / or Figure 2 discussed components. Although the system can include known hardware and software components and / or hardware and software components developed in the future, the system itself is specifically programmed to perform the functions described herein to adjust the hue within the image based on the light source providing illumination within the image. Additionally, the hue adjustment system includes modules and functions specific to the system.

[0032] Activation of the hue adjustment system can be manual, where the user provides an input indicating that the hue adjustment system should be activated; or it can be automatic, where the hue adjustment system detects a trigger event indicating that the system should be activated. Example trigger events include: activation of a device or component of the hue adjustment system, activation of a software or application using the hue adjustment system (e.g., a video conferencing application, a video call application, an image capture system, an image manipulation software, other applications or software using an image capture device, applications using or transmitting a video stream or feed, etc.). For example, the hue adjustment system can be used with video conferencing or call software or image capture software. When the video conferencing or video call or image capture software is activated, the hue adjustment system can also be activated. Although most of this discussion will describe using the hue adjustment system in conjunction with image capture software. However, this is a non-limiting example as the hue adjustment system can be used with any application or software that captures, transmits, and / or receives images.

[0033] The system can be used with any image that can be captured by an image capture device, transmitted to the tone adjustment system, and / or otherwise accessible to the tone adjustment system. The image can include a still image or a frame from a video (such as a video within a video conferencing application). The tone adjustment system can also be used in a variety of different environments including but not limited to personal environments, commercial environments, public environments, office environments, global environments, etc. In the case where the image is captured as a video frame, the processing of the image by the tone adjustment system can occur substantially in real time as the frames of the video are captured, processed, and transmitted within the video stream. Thus, processing can be performed very quickly on each frame within the video or any other image, which may or may not be part of a video feed or video stream.

[0034] Additionally, processing or tone adjustment can be used to adjust frames or images that have not yet been captured. In other words, the tone adjustment system can provide or identify adjustments that need to be made to correct or adjust the tone. These adjustments can be used not only for the received frames or images, but also for subsequent frames or images captured by the image capture device. In a video, tone adjustment identification can be performed for each frame, or can be performed every few frames, where the information identified for a previous frame is used to adjust a frame for which an exact adjustment has not been identified. Similarly, tone adjustment identification can be performed for each static image captured, or can be performed for images at a specific frequency. Adjustment information identified previously can be used to adjust an image for which an exact adjustment has not been identified.

[0035] The tone adjustment system can be a stand-alone system, can be accessible via other computing devices, and / or a combination thereof. For example, the tone adjustment system can be a stand-alone system accessible by a user, and / or can be or provide an application accessible by the user on another computing device. The tone adjustment system can be accessible using any type of computing device (such as a personal computer, laptop computer, smart phone, tablet computer, smart watch, head-mounted display, smart TV, or other smart appliance, etc.).

[0036] The tone adjustment system can have an associated graphical user interface. The graphical user interface can be provided on a display or monitor, which can be associated with the tone adjustment system or not associated with the tone adjustment system. In other words, the tone adjustment system can have a dedicated display or monitor, or can be accessible using any display or monitor. In either case, the tone adjustment system can provide instructions to generate and display the graphical user interface on the display device used to access the tone adjustment system. The graphical user interface can also be updated and managed based on instructions provided by the tone adjustment system. In other words, the tone adjustment system generates and transmits instructions to create and update the graphical user interface.

[0037] The graphical user interface may include multiple tabs, windows, and / or unique interfaces. The graphical user interface may include graphical user interface icons or elements. The graphical user interface icons or elements may include: static non-selectable elements (e.g., headers, footers, logos, global information areas, graphics, etc.), dynamic non-selectable elements (e.g., local information areas applied to specific elements, dynamic graphics, information areas updated based on the information provided therein, indicators, statistical displays, etc.), static selectable elements (e.g., radio buttons, menu icons, selectable indicators, etc.), dynamic selectable elements (e.g., form field input areas, drop-down menus, pop-up windows, etc.), and / or any other elements that may be found in the graphical user interface.

[0038] The graphical user interface may enable a user to provide an input identifying parameters to be used by the tone adjustment system. For example, the tone adjustment system may have different parameters for light source identification, characteristics identified for a specific light source, tone adjustments assigned to or learned for a specific light source or light source characteristics, identification of the application using the tone adjustment system, etc. The user may use any type of input modality to provide the input, and the any type of input modality includes but is not limited to mechanical input (e.g., keyboard input, mouse input, etc.), touch input, auditory or voice input, gesture input, tactile input, etc. The graphical user interface may also provide a display identifying the parameters of the tone adjustment system. It should be noted that the parameters may be different for different applications, different computing systems, different users, etc. Thus, the parameters of the tone adjustment system are not always the same. However, the tone adjustment system may have default or system-wide settings that are the same across different users, systems, applications, etc., until the parameters are adjusted or otherwise changed.

[0039] It should be noted that different users may configure the graphical user interface according to their preferences. Thus, the graphical user interface layout and configuration may vary among users. The extent to which a user may configure the layout may be restricted or set by a system administrator, etc. Additionally, different users or different user roles may have different access levels, which may also change the way and content of information display. Thus, the system may display different graphical user interfaces.

[0040] The tone adjustment system may use one or more artificial intelligence models to identify the light source, the characteristics of the light source, and the tone adjustment that should be performed. For ease of reading, most of the description will refer to a single artificial intelligence model. However, it should be noted that a collection of artificial intelligence models or multiple artificial intelligence models may be used. Additionally, the term artificial intelligence model in this application encompasses neural networks, machine learning models, deep learning models, artificial intelligence models or systems, and / or any other type of computer learning algorithm or artificial intelligence model that can be used currently or created in the future.

[0041] The artificial intelligence model can be a pre-trained model fine-tuned for the tone adjustment system or can be a model created from scratch. Since the tone adjustment system is used in combination with images and videos, some of the models that the system can use are image analysis models, image generation models, image classification models, etc. One or more training data sets can be used to train the model. Additionally, when the model is deployed, it can receive feedback to become more accurate over time. The feedback can be automatically ingested by the model when it is deployed. For example, when the model is used to identify a light source, the characteristics of the light source, and the adjustments that should be made to the tone of the image, if the user modifies the identification of the light source, the characteristics of the light source, or the tone of the image, or otherwise provides some indication that the light source and / or the light source characteristics or tone adjustment may be incorrect, the model ingests this feedback to improve the model.

[0042] On the other hand, when the model identifies a light source and / or the characteristics of the light source or a tone adjustment and no changes are made to the identified light source and / or the characteristics of the light source or the tone adjustment, the model can use this as feedback to further improve the model. The model can be trained in any one of a number of ways, including but not limited to supervised learning, unsupervised learning, semi-supervised learning, training / validation / testing learning, etc. Additionally, the model or models can be used in combination with computer vision to understand, interpret, and manipulate visual data.

[0043] As previously mentioned, a collection of models or multiple models can also be used. Some example models that can be used are: variational autoencoders, generative adversarial networks, recurrent neural networks, convolutional neural networks, deep neural networks, autoencoders, random forests, decision trees, gradient boosting machines, extreme gradient boosting, multimodal machine learning, unsupervised learning models, deep learning models, transformer models, inference models, etc., including models that may be developed in the future. The selected model structure can depend on the specific task that the model will be used to perform.

[0044] A tone adjustment system may include different components for performing different functions of the system, including different steps to be executed. A component that may be part of the tone adjustment system or may simply communicate with the tone adjustment system is an image capture device and associated components. The image capture device may be a camera device or other device or component that can be used to capture images or frames, such as, for example, an infrared image capture device, a network camera device, a thermal imaging image capture device, a conventional camera device, a night vision image capture device, a camera device integrated into an information processing device (such as a laptop computer, a tablet computer, a smart watch, a smart phone, a smart appliance, a television, a smart TV, etc.). The image capture device (referred to as a camera device for ease of reading) may include integrated or otherwise associated components. These associated components may be components that perform functions only for the camera device. For example, the associated components may include a camera device module, an ISP, a camera device driver, a camera device processor, etc.

[0045] Although the image capture device may have components or modules that can perform automatic white balance, due to the design and manufacture of the camera device sensor, this automatic white balance may be incorrect or result in an image that does not correctly reflect the real-world objects within the scene. Although in some cases, the tone adjustment system works as a module or component on the image capture device and in combination with the image capture device sensor, in other cases, the tone adjustment system may use components external to the image capture device to perform functions.

[0046] In other words, the tone adjustment system may be installed on the image capture device or otherwise accessible by the image capture device, such that the image capture device can work as a single system without communicating with other components external to the image capture device. On the other hand, the tone adjustment system may be implemented such that a downstream processor, driver, or other application accesses and activates the tone adjustment system. The downstream processing components (such as a processor, a driver, other applications, etc.) are downstream relative to the image capture device and the processing flow of the image capture device. Therefore, the term downstream is relative to the processing flow.

[0047] The downstream processing component can be any component or software that can be used to adjust the hue of an image. For example, a GPU, CPU, NPU, processing driver, artificial intelligence model, etc. The downstream processor, driver, or other application can have better processing capabilities and / or algorithms than the camera device module of the image capture device. For example, the downstream processor may be able to monitor more types of conditions or scenarios than the camera device module of the image capture device. As another example, compared with the image capture device, the downstream processor may be able to perform more advanced image processing functions. Therefore, it may be beneficial to use the downstream processor to perform at least some of the functions of the hue adjustment system. However, this is not strictly required, and all of the functions of the hue adjustment system can be performed using the image capture device and associated processors, drivers, and / or algorithms.

[0048] At 301, the hue adjustment system receives an image. The image can be a still image, an image obtained from a video (also referred to as a frame), an image from a series of still images, etc. The image can include a scene with one or more objects. For example, the scene can include people, landscapes, buildings, devices, cars, objects, etc. (collectively referred to as "objects" for ease of reading). The reception of the image can occur after the image is captured by the image capture device but before the image is transmitted to another location, such as within a video feed to another user, within a video feed in a video conference transmission, to a data storage location, etc. The reception of the image can also be from a data storage location, a component other than the image capture device, etc. The reception of the image can include: the user providing the image to the hue adjustment system, the hue adjustment system automatically receiving the image from a component such as the image capture device, the hue adjustment system accessing a data storage location and extracting the image, and / or any other technique for obtaining the image.

[0049] How the image is received or otherwise obtained can be set within the graphical user interface, including by accepting instructions within the graphical user interface to access a location that can store one or more images from the user. As another example, the user can set a data storage location within the graphical user interface, and the hue adjustment system can automatically obtain any images stored at that location, even when new images are added to the data storage location. The user can also identify one or more applications within the graphical user interface that interface with the hue adjustment system. If these applications are activated, the hue adjustment system can also be activated and work in conjunction with these applications to perform the described functions. For example, if a video conferencing application is opened, the hue adjustment system can also be activated, and when the image capture device captures an image or frame, the hue adjustment system performs the described functions.

[0050] Additionally, the predetermined frequency at which the tone adjustment system receives images can be set within the graphical user interface. For example, the user can identify that the system should check for new images at a predetermined time interval. As another example, in the case of a video feed or a series of still images, the user can identify the frequency at which frames or images should be transmitted to the tone adjustment system. For example, the frequency can be set such that every other frame, every two frames, every three frames, etc. are transmitted to the tone adjustment system instead of sending every frame captured by the image capture device during a video conference.

[0051] The predetermined frequency can also be based on a trigger event, which can indicate that an image requires tone adjustment. For example, the trigger event can be: the user provides an input within the application via the selection of an auto white balance icon or a user input area indicating that the user wishes to adjust the tone of the image, the user provides an auditory input or a gesture input indicating that the video feed or the image requires tone adjustment, etc. The trigger event can also occur when a change between images is detected, especially for the illumination within the image. For example, if a light source is added or removed between images of a scene, the tone adjustment system can be triggered. As another example, if the color or color temperature of the illumination of a light source changes between images of a scene, the tone adjustment system can be triggered. Thus, changes in the environment or context within the scene may cause the tone adjustment system to be triggered.

[0052] At 302, the tone adjustment system can determine whether a light source providing illumination within the image can be identified. To determine whether a light source can be identified, the system may attempt to identify the light source that can provide illumination. The identification of the light source can include identifying a specific light source (e.g., the model of the light source, the identifier of the light source, etc.) and / or general information that can identify the light source (e.g., the type of the light source, etc.). For simplicity, the discussion will refer to a single light source providing illumination within the image. However, it should be readily understood that there may be multiple light sources providing illumination within the image, and the described systems and methods can be applied to each of these light sources. In the case of multiple light sources, additional functions may also be available. It should be noted that the light source does not need to be included in the image. In other words, the source of the illumination within the image is not necessarily included in the image. Additionally, the term light source refers to any object that provides illumination.

[0053] The light source can be a natural light source (e.g., the sun, moon, stars, fire, etc.) or an artificial light source (e.g., a light bulb, a display screen of different devices, a flash of an image capture device, etc.). Additionally, some light sources or the illumination caused by the light source can be controlled by a color tone adjustment system. For example, the color tone adjustment system can communicate with the Internet of Things (IoT) environment, where different devices within the environment communicate with each other and can be controlled by instructions. In this case, the color tone adjustment system can provide instructions to IoT devices to adjust the illumination caused by the IoT devices. As another example, the color tone adjustment system can be part of a system or an information processing device and can provide instructions or communicate with a light source within the system (e.g., the display of the system or the information processing device). As a final non-limiting example, the system can provide instructions to other devices or objects that can be part of the IoT environment, which result in a modification of the illumination provided by the light source. For example, the system can communicate with blinds or other window coverings and can provide instructions to close or open the window coverings, thereby adjusting or changing the amount of illumination caused by the sun or other external light sources.

[0054] Even if the color tone adjustment system cannot transmit instructions or control devices, the color tone adjustment system can still receive transmissions from the devices. For example, the color tone adjustment system can receive information from a light source, which can provide illumination identification characteristics, parameters, settings, etc. of the light source. However, the system may not be able to provide instructions to the light source to change any of these characteristics, parameters, settings, etc.

[0055] To determine whether a light source can be identified, the color tone adjustment system can use information corresponding to an image. For example, the image can include metadata that identifies the geographical location where the image was captured, the time when the image was captured, the user who captured the image or caused the image to be captured, etc. Therefore, the system can use the metadata corresponding to the image to identify the information corresponding to the image. This information can be used to identify the light source that provides illumination within the image. For example, the time of day can help determine whether the natural light source is the moon or the sun. As another example, knowing the user who captured the image can provide information related to the environment where the user was located when the image was captured. The system can store information related to the user's environment, including: different light sources within the environment, the size of the environment, other objects within the environment, the identification of controllable objects within the environment, etc. In other words, the system can access a database or other data storage location that identifies the light sources within a known environment. When new information about the environment is learned, the system or the data storage location can be updated using the new information so that the new information can be used at a later time.

[0056] The system can use one or more artificial intelligence models, algorithms, classifiers, parsing tools, etc. to identify situations and environments from images. In other words, the system can use analytical techniques to analyze images and identify the information contained within the scenes of the images. As an example, the system can use an image parser or an artificial intelligence model to identify objects within the image, and then use a classifier to identify the types of the identified objects. For example, the system can use analytical techniques to identify light sources within the image (e.g., light bulbs, lights, general displays, device-specific displays, light-emitting diode (LED) objects, etc.). It should be noted that not all light sources or objects that generate illumination within the image can be actually captured in the image. For example, although it may be the sun providing illumination within the image, the scene captured in the image may not include the sun.

[0057] The tone adjustment system can also use auxiliary information sources to identify information corresponding to the image. Auxiliary information sources include any information sources that can provide information related to the image. Example auxiliary information sources include, but are not limited to: the Internet, data storage locations, the user's applications or accounts (e.g., calendars, social media accounts, text messaging applications, email applications, etc.), environmental databases, etc. Accessing the auxiliary information sources can include using the information already identified about the image to query the auxiliary information sources. For example, if the metadata indicates the user who caused the image to be captured, the system can access auxiliary information sources such as the user's calendar or schedule, the user's email account, the user's text messaging or social media accounts, etc. These auxiliary information sources can identify the user's location, the applications the user is using, the location where the user was when the image was captured, etc. Other auxiliary information sources can be the Internet or applications. As an example, if the metadata indicates the geographical location and time when the image was taken, the system can query the Internet or a weather application to identify the environmental conditions at the indicated geographical location at the indicated time.

[0058] The tone adjustment system can also use other objects within the environment to identify light sources. The use of other objects can include: querying other objects or systems within the environment for information related to light sources that may or may not provide illumination within the environment. For example, the system can query IoT objects or other objects within the environment that communicate with the tone adjustment system to identify light sources within the environment. As an example, the tone adjustment system can provide instructions to a system with a display to determine whether the display of the system is off or on. As another example, the tone adjustment system can request information from a hub or controller that communicates with multiple devices within the environment, requesting information identifying which devices have displays and are currently on and thus providing illumination within the image.

[0059] The system can also identify any characteristics of the light source. The characteristics of the light source are those that directly or indirectly identify the color temperature of the light source and the influence of the light source. Examples of characteristics can include: the lumen or nit value of the light source, the brightness of the light source, the color temperature of the light source, the age of the light source, etc. The color temperature can be defined on a scale (e.g., warm color, cold color, deep, shallow, etc.), defined as a specific color or color value (e.g., the RGB value of a color, red, blue, etc.), defined as a hue (e.g., gray hue, magenta hue, cyan hue, etc.), color ratio (e.g., green:magenta, blue:yellow, red:cyan, etc.), etc. The tone adjustment system can identify the output color of the light source, and based on the identified color, can identify the color temperature of the illumination provided by the light source.

[0060] The identification of the color temperature of the light source can be direct measurement or direct information identifying the color temperature, or information that can be used to infer the color temperature information. Examples of direct measurement or information will be information that specifically identifies the color temperature (e.g., the color output by the display device, light source information identifying the color temperature (e.g., bulb information), etc.). Examples of information that can be used to infer the color temperature information is the weather, which will be used to infer the color temperature of the natural light entering through the window.

[0061] The influence of the light source is the identification or measurement of the area within the image that is illuminated by the light source. In other words, this influence identifies how much of the image is illuminated by the light source and where within the image the illumination by the light source occurs. One technique that can be used to identify the influence of the light source is that the tone adjustment system can identify the brightness value of the light source. The brightness value can be used to calculate the illumination range from the light source and thus determine the size or area within the image that is affected by the illumination of the light source. Using another technique to identify the influence of the light source, the system can analyze the image to determine the areas within the image that are close to a specific light source or which areas appear brighter than other areas within the image. Looking for the brightest locations can enable the system to move outwards and identify those locations within the image that are brighter than other locations or have a different color temperature compared to other locations. It should be noted that the influence of the light source can be the entire image. Other techniques for identifying the influence of the light source are possible and expected, including the use of artificial intelligence models, the use of historical information, the use of information provided by the user, etc. Additionally, combinations of these techniques can be used to identify the influence.

[0062] To identify the characteristics of a light source, the system can access auxiliary information sources, which can be the same as or different from the auxiliary information sources used to identify information related to an image, and these auxiliary information sources can provide information related to the characteristics of the light source providing illumination within the image. The tone adjustment system can use the identification of the light source to query the auxiliary information source to identify the characteristics of the light source. As an example, the system can use the identifier or identification information of the light source to query the auxiliary information source (e.g., the Internet, databases, data storage locations, device or object manuals or specifications, software applications, user notes, etc.). Then, the auxiliary source can return the characteristics related to the light source. As a non-limiting example, the tone adjustment system can use the identification information of a specific display providing illumination within the image to query a database of storage devices or object manuals. A manual corresponding to the display can be returned, and the system can further parse the manual to identify the characteristics of the display, such as average brightness or brightness range, color temperature of the display, size of the display, etc.

[0063] The tone adjustment system can also use an artificial intelligence model to identify the characteristics of the light source. A training data set including the light source and the characteristics of the light source can be used to train the model. When the tone adjustment system can identify the light source, the model can digest this information and make predictions about the identified light source based on the training of the model. As new information is received and identified, the model can automatically ingest the new information as well as any predictions or feedback related to the new information to become more refined and accurate over time.

[0064] One or more of the techniques described in connection with identifying the light source itself can also be used to identify the characteristics of the light source. For example, when identifying the characteristics of the light source, the system can use other objects within the environment. As an example, to determine the characteristics of an uncontrollable light source, the system can provide instructions to control or change the characteristics of a controllable light source. By controlling the controllable light source or the object controlling the illumination of the light source (e.g., window coverings, etc.), the system can identify the change in tone and then attribute the remaining tone to a different light source. For example, the system can cause the display to show a blue image providing a blue hue to the scene. Since the system now knows the exact color temperature or tone attributed to one light source, the system can infer that the remaining tone belongs to other light sources providing illumination within the scene. The system can also iteratively control the object to further refine or infer information related to other light sources.

[0065] It should be noted that other techniques can be used to identify the light source and / or the characteristics of the light source. Additionally, the described techniques can be used in combination to identify the light source providing illumination within the image and its characteristics.

[0066] If a light source providing illumination within the image cannot be identified at 302, the tone adjustment system can use conventional techniques to adjust the tone of the image at 304. This means that the tone adjustment system or other processor will use conventional techniques to automatically white balance without input based on the light source. In the presence of multiple light sources, if the system can identify one light source but not another, the automatic white balance can consider the information of one light source and make adjustments related to the influence of that light source, and then use conventional techniques for each part of the image affected by the unknown light source.

[0067] On the other hand, if the tone adjustment system can identify a light source providing illumination within the image, at 303 the system can adjust the tone of the image based on the characteristics of the light source. Adjusting the tone can include providing information related to the light source and its characteristics to an automatic white balance algorithm or mechanism. Then, the AWB algorithm can consider this information when determining the red, green, and blue gain values of the image or the parts of the image affected by the illumination of the light source. Thus, the tone adjustment system can result in a change in the color values of only a part of the image rather than the entire image. Additionally, the tone adjustment system can cause a change in the value of a single color based on the characteristics. In conventional techniques, when one color changes, the corresponding colors also change at a programmed ratio. However, since the tone adjustment system knows the actual color temperature and influence of the illumination of the light source, the tone adjustment system can change a single or individual RGB gain value based on the color change actually required, regardless of which colors are related within the programmed color curve or ratio. In other words, the described system can adjust the color values within the image regardless of the values of other colors within the image.

[0068] The adjustment of the tone of the image can be performed very quickly such that the adjusted image can be included in the transmission of a video feed occurring substantially in real time. In other words, the described system and method can be used even during the transmission of a live video stream or video feed. Additionally or alternatively, the identified or made adjustments to the image can be used to adjust subsequent images or frames captured by the image capture device. In other words, the adjustments made to the image or frame can be identified and provided within instructions for adjusting subsequent captured images or frames of the scene.

[0069] In addition, the described systems and methods can be used in a post - processing step. The post - processing step means that the tone - adjustment system receives an image, the image is adjusted and then subsequently stored, rather than transmitted back to the image - capture device. For example, a user can select an image stored in a data - storage location and perform processing on the image to make it look closer to the real - world scene captured within the image. Selecting the auto - white - balance icon or otherwise providing an input to perform auto - white - balance can cause the tone - adjustment system to perform the described steps to adjust the tone of the image in a more sophisticated way than conventional techniques, thus providing an output image that is more likely to look closer to the real - world scene than that presented by conventional techniques. Once processed, the image can be stored, printed, transmitted to a location selected by the user, further adjusted or processed, etc.

[0070] As a non - limiting overall example of the described system, the tone - adjustment system receives frames of video being captured from video - conference participants and the surrounding environment. Within the image, there are multiple light sources providing illumination within the image. In this example, there will be two light sources: a display screen showing an image with a red hue, and a window through which natural sunlight shines. Although the display device is visible within the image, the sun is not visible. However, the illumination from both the display device and the sun is visible within the scene and the image. The tone - adjustment system identifies that the light sources are the display device and the sun.

[0071] The tone - adjustment system also identifies the characteristics of the light sources. In this example, the display device is operably coupled or in communication with the tone - adjustment system. This operable coupling or connection can be any type of connection that allows communication transmission, such as a wireless connection, a wired connection, communication through other devices, etc. Thus, to identify the characteristics of the display device, the tone - adjustment system queries the display device or otherwise communicates with the display device to determine what is currently being displayed on the display device. Based on this communication, the tone - adjustment system identifies that an image with a red hue is currently being displayed on the display device. The tone - adjustment system can also learn the exact color of the image, the size of the image, whether any other content is being displayed on the display, the brightness level of the display device, and / or information that directly or indirectly identifies details or characteristics about the light source and can identify the color temperature of the light source. The tone - adjustment system also identifies the area within the image affected by the illumination of the display device or the impact of the display - device illumination.

[0072] The sun does not communicate directly with the tone adjustment system, so the tone adjustment system cannot query the sun to determine the sun's characteristics. However, the tone adjustment system can query one or more auxiliary sources to determine characteristics regarding the sun. For example, the tone adjustment system can query a weather application or an Internet source to determine the weather at a location. This enables the tone adjustment system to determine whether it is sunny, cloudy, or raining, etc. The tone adjustment system can also use this information to determine the time of day. The weather and / or the time of day affect the tone of the illumination from the sun. The tone adjustment system can also send instructions to the display device to change the tone generated by the display device to better approximate the tone of the illumination from sunlight. By changing the illumination of the display device, the system can also identify the impact of sunlight within an image.

[0073] Once the color temperature and impact of the display device and sunlight are identified, the tone adjustment system can provide instructions to perform automatic white balance in view of these characteristics. In other words, the system adjusts the tone of the image based on the characteristics of the light source. In this example, the instructions can include modifying the RGB values to compensate for the red color or hue generated by the display device in the areas of the image affected by the display device. The instructions can also include modifying the RGB values to compensate for the color temperature and impact of sunlight. Different from conventional techniques that use automatic white balance based on programmed color ratios and apply these color ratio modifications to the entire image, the described system determines the specific tone adjustments needed to compensate for the characteristics of the illumination source. Additionally, the tone adjustment can be performed not across the entire image, but only on the respective parts of the image affected by the illumination of the light source. Then, the adjusted image can be transmitted to some locations, for example, transmitted through the transmission medium in a video conference, transmitted to a data storage location, transmitted back to the image capture device for additional processing, etc.

[0074] As those skilled in the art will recognize, aspects can be implemented as a system, method, or apparatus program product. Thus, aspects can take the form of an entirely hardware implementation or an implementation including software, which can generally be referred to herein as "circuitry", "module", or "system". Additionally, aspects can take the form of an apparatus program product implemented in one or more device-readable media having device-readable program code embodied therewith.

[0075] It should be noted that the various functions described herein can be implemented using instructions executed by a processor stored on a device-readable storage medium such as a non-signal storage device. The storage device can be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the storage medium will include the following: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the context of this document, a storage device is not a signal and should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through an optical fiber cable), or electrical signals transmitted through a wire. Additionally, the term "non-transitory" includes all media other than signal media.

[0076] The program code implemented on the storage medium can be transmitted using any suitable medium (including but not limited to wireless, wired, fiber optic cable, radio frequency, etc., or any suitable combination of the foregoing).

[0077] The program code for performing the operations can be written in any combination of one or more programming languages. The program code can be executed entirely on a single device, partially on a single device, executed as a stand-alone software package, partially on a single device and partially on another device, or entirely on another device. In some cases, the devices can be connected by any type of connection or network including a local area network (LAN) or a wide area network (WAN), or can be connected through other devices (e.g., through the Internet using an Internet service provider), through a wireless connection such as near field communication, or through a hardwired connection such as a USB connection.

[0078] Example embodiments are described herein with reference to the accompanying drawings, which illustrate example methods, apparatuses, and program products according to various example embodiments. It will be understood that the actions and functions can be at least partially implemented by program instructions. These program instructions can be provided to the processor of a device, a dedicated information processing device, or other programmable data processing device of a manufacturing machine, such that the instructions executed via the processor of the device implement the specified function / action.

[0079] Note that while specific boxes are used in the figures and a specific ordering of the boxes is shown, these are non-limiting examples. In some contexts, two or more boxes may be combined, one box may be divided into two or more boxes, or certain boxes may be appropriately reordered or reorganized, as the explicitly shown examples are for illustrative purposes only and should not be construed as restrictive.

[0080] As used herein, unless otherwise expressly indicated, the singular forms “a” and “an” may be construed to include the plural “one or more.”

[0081] The present disclosure has been presented for purposes of illustration and description, but the present disclosure is not intended to be exhaustive or limiting. Many modifications and variations will be apparent to those of ordinary skill in the art. The example embodiments were chosen and described to illustrate the principles and practical applications and to enable others of ordinary skill in the art to understand the disclosure of various embodiments with various modifications that are suited to the particular uses contemplated.

[0082] Accordingly, although illustrative example embodiments have been described herein with reference to the figures, it is to be understood that the description is not restrictive, and that various other changes and modifications may be made therein by those of ordinary skill in the art without departing from the scope or spirit of the present disclosure.

Claims

1. A method, comprising: receiving an image of a scene at a tone adjustment system; identifying, using the tone adjustment system, a light source providing illumination within the image; as well as The hue adjustment system is used to adjust the hue of the image based on characteristics of the light source.

2. The method according to claim 1, wherein: Identifying the light source includes accessing a data storage location that identifies the light source within the environment.

3. The method according to claim 1, wherein: Identifying the light source includes identifying characteristics of the light source.

4. The method according to claim 3, wherein: Identifying the characteristics of the light source includes accessing a data storage location storing light sources and characteristics of the light sources and querying the data storage location for the identified light source.

5. The method according to claim 1, wherein: Identifying the light source includes querying a device operably coupled to the color tone adjustment system.

6. The method according to claim 1, wherein: Identifying the light source includes identifying an output color of the light source.

7. The method according to claim 1, wherein: The adjusting includes adjusting a value of a color of a portion of the image.

8. The method according to claim 1, wherein: The adjusting includes adjusting the value of one color of the image without regard to the values ​​of other colors of the image.

9. The method according to claim 1, wherein: Identifying the light source includes controlling the lighting within the scene by providing instructions to a controllable device to identify characteristics of an uncontrollable light source providing illumination within the image.

10. The method according to claim 1, wherein: Identifying the light source includes querying an auxiliary information source for characteristics related to the light source.

11. A system, comprising: processor, a memory device storing instructions that, when executed by the processor, cause the system to: receiving an image of a scene at a tone adjustment system; identifying, using the tone adjustment system, a light source providing illumination within the image; as well as The hue adjustment system is used to adjust the hue of the image based on characteristics of the light source.

12. The system according to claim 11, wherein: Identifying the light source includes accessing a data storage location that identifies the light source within the environment.

13. The system according to claim 11, wherein: Identifying the light source includes identifying characteristics of the light source.

14. The system according to claim 13, wherein: Identifying the characteristics of the light source includes accessing a data storage location storing light sources and characteristics of the light sources and querying the data storage location for the identified light source.

15. The system according to claim 11, wherein: Identifying the light source includes querying a device operably coupled to the color tone adjustment system.

16. The system according to claim 11, wherein: Identifying the light source includes identifying an output color of the light source.

17. The system of claim 11, wherein: The adjusting includes adjusting a value of a color of a portion of the image.

18. The system of claim 11, wherein: The adjusting includes adjusting the value of one color of the image without regard to the values ​​of other colors of the image.

19. The system of claim 11, wherein: Identifying the light source includes controlling the lighting within the scene by providing instructions to a controllable device to identify characteristics of an uncontrollable light source providing illumination within the image.

20. A product, comprising: A computer readable storage device storing executable code that, when executed by a processor, causes the product to: receiving an image of a scene at a tone adjustment system; identifying, using the tone adjustment system, a light source providing illumination within the image; as well as The hue adjustment system is used to adjust the hue of the image based on characteristics of the light source.

Images