Camera auto white balance: under-the-sea true color algorithm

Abstract

A method includes receiving an image from a camera. The method also includes, based on an indication that the camera is underwater, segmenting the image to determine a plurality of regions of the image and a corresponding classification for each region of the plurality of regions. The method further includes, for each region of the plurality of regions, determining one or more color gain parameters based on the corresponding classification of the region. The method additionally includes adjusting the image based on the one or more color gain parameters determined for each region of the plurality of regions.

Description

Camera Auto White Balance: Under-the-Sea True Color AlgorithmBACKGROUND

[0001] Many modem computing devices, including mobile phones, personal computers, and tablets, include image capturing devices. Some image capturing devices are configured with telephoto capabilities.SUMMARY

[0002] In an embodiment, a method includes receiving an image from a camera. The method also includes, based on an indication that the camera is underwater, segmenting the image to determine a plurality of regions of the image and a corresponding classification for each region of the plurality of regions. The method further includes, for each region of the plurality of regions, determining one or more color gain parameters based on the corresponding classification of the region. The method additionally includes adjusting the image based on the one or more color gain parameters determined for each region of the plurality of regions.

[0003] In another embodiment, a computing device includes a control system. The control system is configured to receive an image from a camera. The control system is also configured to, based on an indication that the camera is underwater, segment the image to determine a plurality of regions of the image and a corresponding classification for each region of the plurality of regions. The control system is further configured to, for each region of the plurality of regions, determine one or more color gain parameters based on the corresponding classification of the region. The control system is additionally configured to adjust the image based on the one or more color gain parameters determined for each region of the plurality of regions.

[0004] In a further embodiment, a non-transitory computer readable medium stores program instructions executable by one or more processors to cause the one or more processors to perform operations. The operations include receiving an image from a camera. The operations also include, based on an indication that the camera is underwater, segmenting the image to determine a plurality of regions of the image and a corresponding classification for each region of the plurality of regions. The operations further include, for each region of the plurality of regions, determining one or more color gain parameters based on the corresponding classification of the region. The operations additionally include adjusting the image based on the one or more color gain parameters determined for each region of the plurality of regions.

[0005] In another embodiment, a system is provided that includes means for receiving an image from a camera. The system also includes means for, based on an indication that thecamera is underwater, segmenting the image to determine a plurality of regions of the image and a corresponding classification for each region of the plurality of regions. The system further includes means for, for each region of the plurality of regions, determining one or more color gain parameters based on the corresponding classification of the region. The system additionally includes means for adjusting the image based on the one or more color gain parameters determined for each region of the plurality of regions.

[0006] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the figures and the following detailed description and the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 illustrates an example computing device, in accordance with example embodiments.

[0008] Figure 2 is a simplified block diagram showing some of the components of an example computing system.

[0009] Figure 3 is a diagram illustrating a training phase and an inference phase of one or more trained machine learning models in accordance with example embodiments.

[0010] Figure 4 is a flow chart of a method, in accordance with example embodiments.

[0011] Figure 5 depicts an image, in accordance with example embodiments.

[0012] Figure 6 depicts a segmented image, in accordance with example embodiments.

[0013] Figure 7 depicts capturing an image, in accordance with example embodiments.

[0014] Figure 8 depicts a displayed image, in accordance with example embodiments.

[0015] Figure 9 depicts a color correction process, in accordance with example embodiments.

[0016] Figure 10 depicts calculations for the color correction process, in accordance with example embodiments.DETAILED DESCRIPTION

[0017] Example methods, devices, and systems are described herein. It should be understood that the words “example” and “exemplary” are used herein to mean “serving as an example, instance, or illustration.” Any embodiment or feature described herein as being an “example” or “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or features unless indicated as such. Other embodiments can be utilized,and other changes can be made, without departing from the scope of the subject matter presented herein.

[0018] Thus, the example embodiments described herein are not meant to be limiting. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0019] Throughout this description, the articles “a” or “an” are used to introduce elements of the example embodiments. Any reference to “a” or “an” refers to “at least one,” and any reference to “the” refers to “the at least one,” unless otherwise specified, or unless the context clearly dictates otherwise. The intent of using the conjunction “or” within a described list of at least two terms is to indicate any of the listed terms or any combination of the listed terms.

[0020] The use of ordinal numbers such as “first,” “second,” “third” and so on is to distinguish respective elements rather than to denote a particular order of those elements. For the purpose of this description, the terms “multiple” and “a plurality of’ refer to “two or more” or “more than one.”

[0021] Further, unless context suggests otherwise, the features illustrated in each of the figures may be used in combination with one another. Thus, the figures should be generally viewed as component aspects of one or more overall embodiments, with the understanding that not all illustrated features are necessary for each embodiment. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. Further, unless otherwise noted, figures are not drawn to scale and are used for illustrative purposes only. Moreover, the figures are representational only and not all components are shown. For example, additional structural or restraining components might not be shown.

[0022] Additionally, any enumeration of elements, blocks, or steps in this specification or the claims is for purposes of clarity. Thus, such enumeration should not be interpreted to require or imply that these elements, blocks, or steps adhere to a particular arrangement or are carried out in a particular order.I. Overview

[0023] A computing system may be configured to alter images to make the scene represented by an image appear more similar to how the scene is perceived by a human. In particular, the computing system may adjust color gain parameters of an image to make the colors of the image appear closer to the colors perceived by a human. For example, an image taken of a scene with fluorescent lighting may appear to have a yellow tint. To make the scenein the image appear more similar to the scene observed by a human, the computing system may increase the blue color gain parameter.

[0024] Other environmental situations may also affect how color is captured in an image. For instance, an image taken underwater may have a blue tint, which may be due to the diffraction of light through water. As a result, the image may not accurately reflect the colors perceived by the human. However, due to complicated lighting conditions underwater, the computing system may be unable to determine color gain parameters using traditional color adjustment methods. The transmission of light through water may vary based on water type, depth, and other factors, all of which may affect an image taken underwater and how the scene is perceived to a human. Further, even if the computing system adjusts the color gain parameter, the adjusted image may not appear accurate. For example, a user may expect the water to be blue in an image taken underwater. However, after adjusting the color of subjects in the image to be more accurate, the water may appear to be white. Accordingly, due to the effects of color correction on various regions of the image, an approach that adjusts different regions of an image by different color correction gains may result in an image that appears closer to how the image would appear to a human than an approach that adjusts different regions of an image by the same color correction gain.

[0025] Provided herein are underwater color correction methods to adjust color gain parameters for an image taken underwater so that the image appears more similar to how the scene is perceived by a human. The computing system may segment the image into various regions with corresponding classifications. For each region, the computing system may determine one or more color gain parameters based on the classification of the region. The computing system may then adjust the image based on the color gain parameters.

[0026] To trigger these underwater color correction methods, the computing system may first determine whether an image is taken underwater. In some examples, the computing system may receive a signal from a depth sensor, which may be attached to the computing system or other device responsible for capturing the image. Additionally and / or alternatively, the computing system may apply a machine learning model to the image to determine whether the image is taken underwater. If the computing system determines that the image is taken underwater, the computing system may apply the methods as described herein. If the computing system determines that the image is not taken underwater, the computing system may apply other color correction methods.

[0027] The computing system may segment the image into a plurality of regions with corresponding classifications. In some examples, the computing system may segment theimage into one or more foreground regions and / or one or more background regions. For example, for an image of a fish, the computing system may segment and classify the region depicting the fish as the foreground region and the water around the fish as the background region.

[0028] To segment the image, the computing system may apply a machine learning model and / or edge detection techniques. For instance, the computing system may apply a segmentation machine learning model, which may output regions and classifications for the image. Additionally and / or alternatively, the computing system may apply Canny edge detection by calculating the difference between neighboring pixels in the image, and the computing system may then classify the identified regions. Other edge detection methods may also be used. In some examples, applying an edge detection technique rather than a machine learning model may help facilitate efficient processing of the image, so that the computing system uses less computing power and / or energy.

[0029] Based on the identified regions, the computing system may calculate one or more color gain parameters for each of the regions. For example, the computing system may determine the average red and green statistics of the image, and the computing system may determine the red color gain parameter based on this average. Additionally and / or alternatively, the computing may determine the average of the blue and green statistics of the image, which the computing system may use to determine the color gain parameters of the image.

[0030] These color gain parameters may then be used to adjust the image. In some examples, the computing system may combine the color gain parameters determined for each region. For example, the computing system may multiply the foreground color gain parameter by a foreground weight and the background color gain parameter by a background weight, and the computing system may take the sum of the products to determine a color gain parameter to apply to the image.

[0031] In some examples, the computing system may also calculate a trust level, which may also be used to determine the color gain parameters to apply. The trust level may depend on a depth measurement (e.g., a distance to water level measurement), the classifications of the regions in the image, and other factors. For example, the computing system may determine a count of regions having a particular classification. If the count is above a threshold number, the computing system may apply the underwater color correction methods described herein.II. Example Systems and Methods

[0032] Figure 1 illustrates an example computing device 100. In examples described herein, computing device 100 may be an image capturing device and / or a video capturingdevice. Computing device 100 is shown in the form factor of a mobile phone. However, computing device 100 may be alternatively implemented as a laptop computer, a tablet computer, and / or a wearable computing device, among other possibilities. Computing device 100 may include various elements, such as body 102, display 106, and buttons 108 and 110. Computing device 100 may further include one or more cameras, such as front-facing camera 104 and at least one rear-facing camera 112. In examples with multiple rear-facing cameras such as illustrated in Figure 1, each of the rear-facing cameras may have a different field of view. For example, the rear facing cameras may include a wide angle camera, a main camera, and a telephoto camera. The wide angle camera may capture a larger portion of the environment compared to the main camera and the telephoto camera, and the telephoto camera may capture more detailed images of a smaller portion of the environment compared to the main camera and the wide angle camera.

[0033] Front-facing camera 104 may be positioned on a side of body 102 typically facing a user while in operation (e.g., on the same side as display 106). Rear-facing camera 112 may be positioned on a side of body 102 opposite front-facing camera 104. Referring to the cameras as front and rear facing is arbitrary, and computing device 100 may include multiple cameras positioned on various sides of body 102.

[0034] Display 106 could represent a cathode ray tube (CRT) display, a light emitting diode (LED) display, a liquid crystal (LCD) display, a plasma display, an organic light emitting diode (OLED) display, or any other type of display known in the art. In some examples, display 106 may display a digital representation of the current image being captured by front-facing camera 104 and / or rear-facing camera 112, an image that could be captured by one or more of these cameras, an image that was recently captured by one or more of these cameras, and / or a modified version of one or more of these images. Thus, display 106 may serve as a viewfinder for the cameras. Display 106 may also support touchscreen functions that may be able to adjust the settings and / or configuration of one or more aspects of computing device 100.

[0035] Front-facing camera 104 may include an image sensor and associated optical elements such as lenses. Front-facing camera 104 may offer zoom capabilities or could have a fixed focal length. In other examples, interchangeable lenses could be used with front-facing camera 104. Front-facing camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. Front-facing camera 104 also could be configured to capture still images, video images, or both. Further, front-facing camera 104 could represent, for example, a monoscopic, stereoscopic, or multiscopic camera. Rear-facing camera 112 maybe similarly or differently arranged. Additionally, one or more of front-facing camera 104 and / or rear-facing camera 112 may be an array of one or more cameras.

[0036] One or more of front-facing camera 104 and / or rear-facing camera 112 may include or be associated with an illumination component that provides a light field to illuminate a target object. For instance, an illumination component could provide flash or constant illumination of the target object. An illumination component could also be configured to provide a light field that includes one or more of structured light, polarized light, and light with specific spectral content. Other types of light fields known and used to recover three- dimensional (3D) models from an object are possible within the context of the examples herein.

[0037] Computing device 100 may also include an ambient light sensor that may continuously or from time to time determine the ambient brightness of a scene that cameras 104 and / or 112 can capture. In some implementations, the ambient light sensor can be used to adjust the display brightness of display 106. Additionally, the ambient light sensor may be used to determine an exposure length of one or more of cameras 104 or 112, or to help in this determination.

[0038] Computing device 100 could be configured to use display 106 and front-facing camera 104 and / or rear-facing camera 112 to capture images of a target object. The captured images could be a plurality of still images or a video stream. The image capture could be triggered by activating button 108, pressing a softkey on display 106, or by some other mechanism. Depending upon the implementation, the images could be captured automatically at a specific time interval, for example, upon pressing button 108, upon appropriate lighting conditions of the target object, upon moving computing device 100 a predetermined distance, or according to a predetermined capture schedule.

[0039] Figure 2 is a simplified block diagram showing some of the components of an example computing system 200, such as an image capturing device and / or a video capturing device. By way of example and without limitation, computing system 200 may be a cellular mobile telephone (e.g., a smartphone), a computer (such as a desktop, notebook, tablet, server, or handheld computer), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a gaming console, a robotic device, a vehicle, or some other type of device. Computing system 200 may represent, for example, aspects of computing device 100.

[0040] As shown in Figure 2, computing system 200 may include communication interface 202, user interface 204, processor 206, data storage 208, and camera components 224, all of which may be communicatively linked together by a system bus, network, or otherconnection mechanism 210. Computing system 200 may be equipped with at least some image capture and / or image processing capabilities. It should be understood that computing system 200 may represent a physical image processing system, a particular physical hardware platform on which an image sensing and / or processing application operates in software, or other combinations of hardware and software that are configured to carry out image capture and / or processing functions.

[0041] Communication interface 202 may allow computing system 200 to communicate, using analog or digital modulation, with other devices, access networks, and / or transport networks. Thus, communication interface 202 may facilitate circuit-switched and / or packet-switched communication, such as plain old telephone service (POTS) communication and / or Internet protocol (IP) or other packetized communication. For instance, communication interface 202 may include a chipset and antenna arranged for wireless communication with a radio access network or an access point. Also, communication interface 202 may take the form of or include a wireline interface, such as an Ethernet, Universal Serial Bus (USB), or High- Definition Multimedia Interface (HDMI) port, among other possibilities. Communication interface 202 may also take the form of or include a wireless interface, such as a Wi-Fi, BLUETOOTH®, global positioning system (GPS), or wide-area wireless interface (e.g., WiMAX or 3GPP Long-Term Evolution (LTE)), among other possibilities. However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used over communication interface 202. Furthermore, communication interface 202 may comprise multiple physical communication interfaces (e.g., a Wi-Fi interface, a BLUETOOTH® interface, and a wide-area wireless interface).

[0042] User interface 204 may function to allow computing system 200 to interact with a human or non-human user, such as to receive input from a user and to provide output to the user. Thus, user interface 204 may include input components such as a keypad, keyboard, touch-sensitive panel, computer mouse, trackball, joystick, microphone, and so on. User interface 204 may also include one or more output components such as a display screen, which, for example, may be combined with a touch-sensitive panel. The display screen may be based on CRT, LCD, LED, and / or OLED technologies, or other technologies now known or later developed. User interface 204 may also be configured to generate audible output(s), via a speaker, speaker jack, audio output port, audio output device, earphones, and / or other similar devices. User interface 204 may also be configured to receive and / or capture audible utterance(s), noise(s), and / or signal(s) by way of a microphone and / or other similar devices.

[0043] In some examples, user interface 204 may include a display that serves as a viewfinder for still camera and / or video camera functions supported by computing system 200. Additionally, user interface 204 may include one or more buttons, switches, knobs, and / or dials that facilitate the configuration and focusing of a camera function and the capturing of images. It may be possible that some or all of these buttons, switches, knobs, and / or dials are implemented by way of a touch-sensitive panel.

[0044] Processor 206 may comprise one or more general purpose processors - e.g., microprocessors - and / or one or more special purpose processors - e.g., digital signal processors (DSPs), graphics processing units (GPUs), floating point units (FPUs), network processors, or application-specific integrated circuits (ASICs). In some instances, special purpose processors may be capable of image processing, image alignment, and merging images, among other possibilities. Data storage 208 may include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash, or organic storage, and may be integrated in whole or in part with processor 206. Data storage 208 may include removable and / or non-removable components.

[0045] Processor 206 may be capable of executing program instructions 218 (e.g., compiled or non-compiled program logic and / or machine code) stored in data storage 208 to carry out the various functions described herein. Therefore, data storage 208 may include a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by computing system 200, cause computing system 200 to carry out any of the methods, processes, or operations disclosed in this specification and / or the accompanying drawings. The execution of program instructions 218 by processor 206 may result in processor 206 using data 212.

[0046] By way of example, program instructions 218 may include an operating system 222 (e.g., an operating system kernel, device driver(s), and / or other modules) and one or more application programs 220 (e.g., camera functions, address book, email, web browsing, social networking, audio-to-text functions, text translation functions, and / or gaming applications) installed on computing system 200. Similarly, data 212 may include operating system data 216 and application data 214. Operating system data 216 may be accessible primarily to operating system 222, and application data 214 may be accessible primarily to one or more of application programs 220. Application data 214 may be arranged in a file system that is visible to or hidden from a user of computing system 200.

[0047] Application programs 220 may communicate with operating system 222 through one or more application programming interfaces (APIs). These APIs may facilitate,for instance, application programs 220 reading and / or writing application data 214, transmitting or receiving information via communication interface 202, receiving and / or displaying information on user interface 204, and so on.

[0048] In some cases, application programs 220 may be referred to as “apps” for short. Additionally, application programs 220 may be downloadable to computing system 200 through one or more online application stores or application markets. However, application programs can also be installed on computing system 200 in other ways, such as via a web browser or through a physical interface (e.g., a USB port) on computing system 200.

[0049] Camera components 224 may include, but are not limited to, an aperture, shutter, recording surface (e.g., photographic film and / or an image sensor), lens, shutter button, infrared projectors, and / or visible-light projectors. Camera components 224 may include components configured for capturing of images in the visible-light spectrum (e.g., electromagnetic radiation having a wavelength of 380 - 700 nanometers) and / or components configured for capturing of images in the infrared light spectrum (e.g., electromagnetic radiation having a wavelength of 701 nanometers - 1 millimeter), among other possibilities. Camera components 224 may be controlled at least in part by software executed by processor 206.

[0050] Histogram processing algorithm(s) 226 may include one or more stored algorithms programmed to process histogram information to facilitate autofocus as described herein. In some examples, histogram processing algorithm(s) 226 may include one or more trained machine learning models. In other examples, histogram processing algorithm(s) 226 may be based on heuristics without the use of machine learning. In further examples, a combination of different types of histogram processing algorithm(s) 226 may be used as well.

[0051] In further examples, one or more remote cameras 230 may be controlled by computing system 200. For instance, computing system 200 may transmit control signals to the one or more remote cameras 230 through a wireless or wired connection. Such signals may be transmitted as part of an ambient computing environment. In such examples, inputs received at the computing system 200 (for instance, physical movements of a wearable device) may be mapped to movements or other functions of the one or more remote cameras 230. Images captured by the one or more remote cameras 230 may be transmitted to the computing system 200 for further processing. Such images may be treated as images captured by cameras physically located on the computing system 200.

[0052] Figure 3 shows diagram 300 illustrating a training phase 302 and an inference phase 304 of trained machine learning model(s) 332, in accordance with exampleembodiments. Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions about (patterns in the) training data. The resulting trained machine learning algorithm can be termed as a trained machine learning model. For example, Figure 3 shows training phase 302 where one or more machine learning algorithms 320 are being trained on training data 310 to become trained machine learning model 332. Producing trained machine learning model(s) 332 during training phase 302 may involve determining one or more hyperparameters, such as one or more stride values for one or more layers of a machine learning model as described herein. Then, during inference phase 304, trained machine learning model 332 can receive input data 330 and one or more inference / prediction requests 340 (perhaps as part of input data 330) and responsively provide as an output one or more inferences and / or predictions 350. The one or more inferences and / or predictions 350 may be based in part on one or more learned hyperparameters, such as one or more learned stride values for one or more layers of a machine learning model as described herein

[0053] As such, trained machine learning model(s) 332 can include one or more models of one or more machine learning algorithms 320. Machine learning algorithm(s) 320 may include, but are not limited to: an artificial neural network (e.g., a herein-described convolutional neural networks, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system). Machine learning algorithm(s) 120 may be supervised or unsupervised, and may implement any suitable combination of online and offline learning.

[0054] In some examples, machine learning algorithm(s) 320 and / or trained machine learning model(s) 332 can be accelerated using on-device coprocessors, such as graphic processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application specific integrated circuits (ASICs). Such on-device coprocessors can be used to speed up machine learning algorithm(s) 320 and / or trained machine learning model(s) 332. In some examples, trained machine learning model(s) 332 can be trained, reside and execute to provide inferences on a particular computing device, and / or otherwise can make inferences for the particular computing device.

[0055] During training phase 302, machine learning algorithm(s) 320 can be trained by providing at least training data 310 as training input using unsupervised, supervised, semisupervised, and / or reinforcement learning techniques. Unsupervised learning involvesproviding a portion (or all) of training data 310 to machine learning algorithm(s) 320 and machine learning algorithm(s) 320 determining one or more output inferences based on the provided portion (or all) of training data 310. Supervised learning involves providing a portion of training data 310 to machine learning algorithm(s) 320, with machine learning algorithm(s) 320 determining one or more output inferences based on the provided portion of training data 310, and the output inference(s) are either accepted or corrected based on correct results associated with training data 310. In some examples, supervised learning of machine learning algorithm(s) 320 can be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or set of labels may be used to correct inferences of machine learning algorithm(s) 320.

[0056] Semi-supervised learning involves having correct results for part, but not all, of training data 310. During semi-supervised learning, supervised learning is used for a portion of training data 310 having correct results, and unsupervised learning is used for a portion of training data 310 not having correct results.

[0057] Reinforcement learning involves machine learning algorithm(s) 320 receiving a reward signal regarding a prior inference, where the reward signal can be a numerical value. During reinforcement learning, machine learning algorithm(s) 320 can output an inference and receive a reward signal in response, where machine learning algorithm(s) 320 are configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing an expected total of the numerical values provided by the reward signal over time. In some examples, machine learning algorithm(s) 320 and / or trained machine learning model(s) 332 can be trained using other machine learning techniques, including but not limited to, incremental learning and curriculum learning.

[0058] In some examples, machine learning algorithm(s) 320 and / or trained machine learning model(s) 332 can use transfer learning techniques. For example, transfer learning techniques can involve trained machine learning model(s) 332 being pre-trained on one set of data and additionally trained using training data 310. More particularly, machine learning algorithm(s) 320 can be pre-trained on data from one or more computing devices and a resulting trained machine learning model provided to computing device CD1, where CD1 is intended to execute the trained machine learning model during inference phase 304. Then, during training phase 302, the pre-trained machine learning model can be additionally trained using training data 310. This further training of the machine learning algorithm(s) 320 and / or the pre-trained machine learning model using training data 310 of CDl’s data can be performed using eithersupervised or unsupervised learning. Once machine learning algorithm(s) 320 and / or the pretrained machine learning model has been trained on at least training data 310, training phase 302 can be completed. The trained resulting machine learning model can be utilized as at least one of trained machine learning model(s) 332.

[0059] In particular, once training phase 302 has been completed, trained machine learning model(s) 332 can be provided to a computing device, if not already on the computing device. Inference phase 304 can begin after trained machine learning model(s) 332 are provided to computing device CD1.

[0060] During inference phase 304, trained machine learning model(s) 332 can receive input data 330 and generate and output one or more corresponding inferences and / or predictions 350 about input data 330. As such, input data 330 can be used as an input to trained machine learning model(s) 332 for providing corresponding inference(s) and / or prediction(s) 350. For example, trained machine learning model(s) 332 can generate inference(s) and / or predict! on(s) 350 in response to one or more inference / prediction requests 340. In some examples, trained machine learning model(s) 332 can be executed by a portion of other software. For example, trained machine learning model(s) 332 can be executed by an inference or prediction daemon to be readily available to provide inferences and / or predictions upon request. Input data 330 can include data from computing device CD1 executing trained machine learning model(s) 332 and / or input data from one or more computing devices other than CD1.

[0061] Figure 4 is a flow chart of a method, in accordance with example embodiments. Method 400 may be executed by one or more computing systems (e.g., computing system 200 of Figure 2) and / or one or more processors (e.g., processor 206 of Figure 2). Method 400 may be carried out on a computing system, such as computing system 100 of Figure 1.

[0062] In some examples, method 400 may be executed by a system that includes a mobile device. The mobile device may communicate with an outer casing, which may include a camera and / or other sensors that are used in the process described herein. Additionally and / or alternatively, the mobile device may communicate with one or more sensors remote from the device. The mobile device may receive information transmitted by the sensors, and the mobile device may then use the information for the process described herein.

[0063] At block 402, method 400 includes receiving an image from a camera. The computing system may receive the image from a camera on a mobile device or from a remote camera. In some examples, the mobile device may be covered by an outer casing system, and the camera may capture the image from inside the outer casing system. Additionally and / oralternatively, the mobile device may receive the image from a remote camera that is not physically attached to the mobile device using a wireless connection.

[0064] Figure 5 depicts an image 500, in accordance with example embodiments. Image 500 may be received by a computing device and may be taken by a camera. Image 500 may include coral 502, fish 504, and rock 506. Image 500 may also include water 508. Because image 500 is an image taken of a scene that is underwater, water 508 may affect the appearance of other objects in image 500, such as coral 502, fish 504, and rock 506. As mentioned above, water 508 may affect the transmission of light, causing the objects in image 500 to appear to have a blue tint. Images taken above the water may not have such an issue.

[0065] Accordingly, a computing system may determine whether image 500 or another received image is of an underwater scene, and based on that determination, determine whether to apply a color correction algorithm. To determine whether image 500 is of an underwater scene, the computing system may apply a machine learning model to image 500. Additionally and / or alternatively, the computing system may receive a depth measurement from a depth sensor communicating with the computing system. The depth sensor may be a depth sensor on a mobile device, a depth sensor attached to an outer casing system that is sending information to the computing system, and / or a depth sensor associated with image 500 that is sending information to the computing system. Upon determining that the camera is underwater, the computing system may generate an indication that the camera is underwater. Additionally and / or alternatively, a remote computing system may determine that the camera that took image 500 is underwater, generate an indication that the camera is underwater, and transmit the indication to the computing system.

[0066] Referring back to Figure 4, at block 404, method 400 includes, based on an indication that the camera is underwater, segmenting the image to determine a plurality of regions of the image and a corresponding classification for each region of the plurality of regions. If the indication is that the camera is underwater, the computing system may segment the image using a machine learning model or other process to determine regions for the image and corresponding classifications for the regions.

[0067] Figure 6 depicts segmented image 600, in accordance with example embodiments. The computing system may determine segmented image 600 from image 500 of Figure 5, perhaps by applying a machine learning model or other segmentation process. By applying a machine learning model to image 500 of Figure 5, the computing system may determine an output image that categorizes one or more pixels of image 500 as being a particular region with a particular classification.

[0068] Additionally and / or alternatively, the computing system may apply a texturebased segmentation process, such as Canny edge detection, to image 500 of Figure 5. For an image that includes many pixels, the texture-based segmentation process may involve calculating the differences between adjacent pixels in the image. The computing system may thus determine the edges of each region based on the calculated differences being above and / or below a threshold value. Other methods may also be used to segment the image into different regions with corresponding classifications.

[0069] In some examples, applying a non-machine learning model based segmentation process, such as Canny edge detection, may be more efficient than applying a machine learning model. A machine learning model based approach may use more computing power and energy than a non-machine learning model based segmentation algorithm. In particular, a machine learning model based approach may cause the computing system to determine convolutions and other computationally intensive calculations, whereas a non-machine learning model based segmentation process may involve less computationally intensive calculations.

[0070] By applying a segmentation process, the computing system may segment image 500 into foreground region 602 and background region 604. In some examples, the computing system may combine one or more regions into one region. For example, the computing system may determine that the coral, rock, and fish of image 500 are all different regions, and the computing system may combine the coral, rock, and fish regions of image 500 into a foreground region.

[0071] Segmenting the image into a foreground and background region may be particularly useful in the context of correcting underwater images, due to light transmission through water and a user’s expectations. For instance, as mentioned above, the transmission of light through water may cause objects in the image to appear to have a blue tint. Correcting for this blue tint in the entire image may cause the water to appear white. However, a user may expect that the water is blue. Accordingly, by having one or more foreground regions and one or more background regions, the computing system may selectively apply color correction to relevant regions. For instance, the computing system may more heavily correct (e.g., more aggressively remove) the blue tint in foreground region 602, whereas the computing system may allow the blue tint of background region 604 to remain and / or minimally correct the blue tint of background region 604 such that background region 604 retains the color.

[0072] Referring back to Figure 4, at block 406, method 400 includes, for each region of the plurality of regions, determining one or more color gain parameters based on the corresponding classification of the region. In some examples, determining the color gainparameters may be based on the depth at which the image was obtained. For instance, the computing system may determine attenuation from object to camera, attenuation from the surface of the water, and object radiation above water (i.e., the true color). The attenuation from object to the camera may be calculated based on a distance to the camera. The computing system may multiply the attenuation from object to camera by the attenuation from the surface of the water and the object radiation above water to determine color gain parameters.

[0073] Additionally and / or alternatively, the computing system may assume a two- toned image. One or more color gain parameters may include a blue color gain and a red color gain. In particular, for each color gain, the computing system may take the inverse of the average of the respective color. In some examples, this calculation may apply only to foreground regions. For background regions, the computing system may apply a similar computation, except multiplying the gain by a constant, which may vary based on color. For example, for a red color gain of a background region, the computing system may multiply by a constant less than one, while for a blue color gain of a background region, the computing system may multiply by a constant greater than one. This would help adjust the background to appear realistically blue, while maintaining a level of blue color that does not appear unnatural.

[0074] Additionally and / or alternatively, the computing system may apply a global color correction process. This global color correction process may involve linearly combining two color gains. For example, the computing system may combine the red color gain of the background region and the background color gain of the foreground region to achieve a gain that may be used for both the foreground and background regions of the image.

[0075] For example, for an image that includes one or more regions identified as a foreground region and one or more regions identified as a background region, the computing system may determine a relative percentage of the foreground region to the background region, and based on the relative percentage of that, determining one or more color gain parameters, which may then be used to adjust the image.

[0076] Figure 7 depicts capturing an image, in accordance with example embodiments. As shown in Figure 7, the depth at which the image was captured may also affect color gain parameters. As the image is taken at a greater and greater depth, the blue tint may become more apparent. The computing system may use a color correction chart, which may help determine how much to adjust the color. In particular, using a depth measurement may help facilitate dynamic determination of color correction gains. For example, as depth measurement 712 increases, the computing system may apply a greater blue tint adjustment to the image captured by camera 704. Further, the distance of camera 704 to subject 702 may cause the image toappear different, and the computing system may likewise use the distance measurement 714 to determine color gain parameters. To facilitate dynamic determination of color correction gains, the computing system may continuously adjust the color correction gains as the depth changes. Dynamic determination of color correction gains may result in images that appear closer to what a human observing the scene would perceive. In contrast, other approaches, such as using predetermined color correction gains may cause the computing system to determine adjusted images with color correction gains that are not optimal.

[0077] In some examples, the computing system may include sensors or may be communicating with sensors. From these sensors, the computing system may receive depth measurements 712 and / or distance measurement 714. For example, an outer casing system may include depth sensors and / or distance sensors, which may collect depth measurement 712 and distance measurement 714. The outer casing system may transmit these measurements to the computing system. Additionally and / or alternatively, the computing system may include distance and depth sensors, which may transmit data to the computing system. Upon receipt of the information, the computing system may determine how much to adjust the colors in the image.

[0078] Referring back to Figure 4, at block 408, method 400 includes adjusting the image based on the one or more color gain parameters determined for each region of the plurality of regions. Adjusting the image based on color gain parameters may be based on a variety of factors, including the number of identified regions and trust levels. Additionally and / or alternatively, the computing system may apply the one or more color gain parameters by averaging the color gain parameters for each identified region and applying the determined color gain parameter to the respective region. For example, the computing system may average the determined color gain parameter for the foreground region, and the computing system may average the color gain parameters for the foreground, and the computing system may use the average of the color gain parameters to adjust the image.

[0079] Additionally and / or alternatively, the computing system may adjust the color gain parameters based on trust levels. The trust level may be a measure of how likely the image taken by the camera is to be accurate to the true colors of the scene. Accordingly, the trust level may indicate at what depth the image was taken and / or at what distance the image was taken from the object. In particular, the trust level may be calculated based on a depth measurement from a depth sensor, a depth upper limit, and a depth lower limit. If the depth measurement is greater than the upper depth limit, the computing system may assign a trust level of one, which may indicate that the colors in the image are fairly accurate and the computing system shouldnot apply the color correction process. Whereas, if the depth measurement is less than the lower depth limit, the computing system may assign a trust level of zero, which may indicate that the colors in the image are fairly inaccurate and the computing system should apply the color correction process. When the depth measurement is between the upper depth limit and lower depth limit, the trust level may be a sliding scale from one to zero. When the trust level is one, the computing system may apply the color gain parameters determined by the color correction process. Whereas, when the trust level is zero, the computing system may apply color gain parameters not determined by the color correction process or may apply no color gain adjustment.

[0080] The trust level of an image may also depend on one or more additional factors. For example, the computing system may determine a count of one or more regions with a particular classification. The computing system may determine whether the count is less than a threshold value. If the count is less than a threshold value, the computing system may determine a trust level and proceed with adjusting the image. Additionally and / or alternatively, the computing system may determine trust level based on (i) a count of one or more regions of the plurality of regions with a particular classification and (ii) a count of one or more regions. Further additionally and / or alternatively, the computing system may determine a count of the one or more regions with a particular classification, and the computing system may determine a trust level based on the count of the one or more regions with the particular classification.

[0081] Figure 8 depicts a displayed image, in accordance with example embodiments. As shown in Figure 8, adjusting the image may cause the foreground (e.g., coral 802, rock 804, and fish 806) to have less of a blue tint, whereas the blue tint remains for water 808. Computing system 810 may display adjusted image 800 on a display, which may help a user determine whether the color correction process was applied reasonably.

[0082] Figure 9 depicts color correction process 900, in accordance with example embodiments. As shown in Figure 9, a computing system may receive image 902 from a camera. The computing system may execute scene detector 906, which may also use information from sensor input 904. Based on the information provided by image 902 and sensor input 904, the computing system may determine segmented image 910. Based on segmented image 910, the computing system may apply color correction algorithm 912 to image 902, and the computing system may determine adjusted image 914.

[0083] Figure 10 depicts calculations for the color correction process, in accordance with example embodiments. As shown in Figure 10, a computing system may use depth and / or GPS information at block 1002 to determine whether the scene is underwater by applyingunderwater scene detection at block 1004. If the scene is not underwater, the computing system may exit at block 1030. If the scene is underwater, the computing system may determine a frame summary at block 1006. To determine a frame summary at block 1006, the computing system may also incorporate background statistics at block 1022. Based on determining a frame summary at block 1006, the computing system may apply a frame region detection algorithm at block 1008. If the computing system determines that the region is underwater, the computing system may apply segmentation at block 1010. Applying segmentation at block 1010 may also be based on edge statistics 1022. From applying segmentation at block 1010, the computing system may determine gains for each region (e.g., region A gain calculation at block 1012 until region X gain calculation at block 1014), and the computing system may determine segmented statistics calculator at block 1016. Based on the calculated gains at blocks 1012 and 1014 and the calculated statistics at block 1016, the computing system may determine a gain calculation at block 1018.

[0084] In some examples, the color correction process described herein may depend on a depth value, and perhaps without using segmentation. For example, the computing system may receive an image from a camera and based on the image, determine whether the scene in the image is underwater. Based on an indication that the camera is underwater, the computing system may determine color gain parameters by using the process described herein without segmentation. In particular, the computing system may determine the attenuation from object to camera multiplied by attenuation from the surface of the water multiplied by the object radiation above water (i.e., the true color). The attenuation from object to the camera may be calculated based on a distance to the camera. In some examples, the computing system may obtain an estimate based on the average of the red and green statistics of the image based on the depth at which the image was obtained. Calculating the color gain parameters may be dynamic based on the depth that the image was taken, among other factors, such that various depth values may cause various color gain parameters to be determined. This dynamic process may help facilitate the determination of images that appear closer to what a user would perceive.

[0085] In some examples, a color correction process that does not use segmentation may also depend on trust levels. As mentioned above, the trust level may be a measure of how likely the image taken by the camera is to be accurate to the true colors of the scene and may be calculated based on a depth upper limit and a depth lower limit. If the depth measurement is greater than the upper limit, the computing system may assign a trust level of one, and if the depth measurement is less than the lower limit, the computing system may assign a trust levelof zero. When the depth measurement is between the upper depth limit and lower depth limit, the trust level may be a sliding scale from one to zero. When the trust level is one, the computing system may apply the color gain parameters determined by the color correction process. Whereas, when the trust level is zero, the computing system may apply color gain parameters not determined by the color correction process or may apply no color gain adjustment.

[0086] Referring back to Figure 4, in some examples, method 400 includes receiving a depth estimate from a depth sensor, based on the depth estimate, determining whether the camera is underwater, and based on determining that the camera is underwater, generating the indication that the camera is underwater.

[0087] In some examples, method 400 includes determining, based on applying a machine learning model, determining that the camera is underwater, and based on determining that the camera is underwater, generating the indication that the camera is underwater.

[0088] In some examples, segmenting the image to determine the plurality of regions of the image comprises applying a texture-based segmentation algorithm to the image, where the image comprises a plurality of pixels, where applying the texture-based segmentation algorithm comprises determining a difference between adjacent pixels of the plurality of pixels.

[0089] In some examples, the plurality of regions includes a foreground region and a background region.

[0090] In some examples, adjusting the image based on the one or more color gain parameters comprises determining a relative percentage of the foreground region to the background region, based on the relative percentage of the foreground region to the background region, determining one or more combined color gain parameters, and adjusting the image based on the combined color gain parameters.

[0091] In some examples, adjusting the image based on the one or more color gain parameters comprises determining one or more combined color gain parameters for the foreground region and one or more combined color gain parameters for the background region, adjusting the foreground region of the image based on the combined color gain parameters of the foreground region, and adjusting the background region of the image based on the combined color gain parameters of the background region.

[0092] In some examples, adjusting the image based on the one or more color gain parameters comprises combining the one or more color gain parameters based on the corresponding classification of the region and adjusting the image based on the combined one or more color gain parameters.

[0093] In some examples, the one or more color gain parameters for each region of the plurality of regions includes a blue color gain and a red color gain.

[0094] In some examples, adjusting the image based on the one or more color gain parameters determined for each region of the plurality of regions comprises determining one or more average color gain parameters for each region of the plurality of regions.

[0095] In some examples, method 400 further comprises determining, based on a depth value, a trust level, where adjusting the image based on the one or more color gain parameters determined for each region of the plurality of regions is based on the trust level.

[0096] In some examples, a high depth value is associated with a low trust level.

[0097] In some examples, method 400 further comprises determining a count of one or more regions of the plurality of regions with a particular corresponding classification and, based on the count of the one or more regions being less than a threshold value, determining a trust level, where adjusting the image based on the one or more color gain parameters determined for each region of the plurality of regions is based on the trust level.

[0098] In some examples, method 400 further comprises determining a count of one or more regions of the plurality of regions with a particular corresponding classification, wherein adjusting the image based on the one or more color gain parameters is based on the count of the one or more regions with the particular corresponding classification being below a threshold value.

[0099] In some examples, segmenting the image to determine the plurality of regions of the image is based on applying Canny edge detection to the image.

[0100] In some examples, segmenting the image to determine the plurality of regions of the image is based on applying a machine learning algorithm to the image.

[0101] In some examples, a computing device may comprise a control system configured to carry out the operations of method 400.

[0102] In some examples, the computing device further comprises a casing system, wherein the casing system includes a depth sensor, wherein the control system is configured to receive a depth measurement from the depth sensor of the casing system.

[0103] In some examples, determining the one or more color gain parameters are based on the depth measurement.

[0104] In some examples, a non-transitory computer readable medium may store program instructions executable by one or more processors that cause the one or more processors to perform operations comprising the operations of method 400.III. Conclusion

[0105] The present disclosure is not to be limited in terms of the particular embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those described herein, will be apparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims.

[0106] The above detailed description describes various features and operations of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The example embodiments described herein and in the figures are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0107] With respect to any or all of the message flow diagrams, scenarios, and flow charts in the figures and as discussed herein, each step, block, and / or communication can represent a processing of information and / or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages can be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved. Further, more or fewer blocks and / or operations can be used with any of the message flow diagrams, scenarios, and flow charts discussed herein, and these message flow diagrams, scenarios, and flow charts can be combined with one another, in part or in whole.

[0108] A step or block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical operations or actions in the method or technique.The program code and / or related data may be stored on any type of computer readable medium such as a storage device including random access memory (RAM), a disk drive, a solid state drive, or another storage medium.

[0109] The computer readable medium may also include non-transitory computer readable media such as computer readable media that store data for short periods of time like register memory, processor cache, and RAM. The computer readable media may also include non-transitory computer readable media that store program code and / or data for longer periods of time. Thus, the computer readable media may include secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, solid state drives, compactdisc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.

[0110] Moreover, a step or block that represents one or more information transmissions may correspond to information transmissions between software and / or hardware modules in the same physical device. However, other information transmissions may be between software modules and / or hardware modules in different physical devices.[OHl] The particular arrangements shown in the figures should not be viewed as limiting. It should be understood that other embodiments can include more or less of each element shown in a given figure. Further, some of the illustrated elements can be combined or omitted. Yet further, an example embodiment can include elements that are not illustrated in the figures.

[0112] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for the purpose of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

CLAIMS1. A method comprising: receiving an image from a camera; based on an indication that the camera is underwater, segmenting the image to determine a plurality of regions of the image and a corresponding classification for each region of the plurality of regions; for each region of the plurality of regions, determining one or more color gain parameters based on the corresponding classification of the region; and adjusting the image based on the one or more color gain parameters determined for each region of the plurality of regions.

2. The method of claim 1, further comprising: receiving a depth estimate from a depth sensor; based on the depth estimate, determining whether the camera is underwater; and based on determining that the camera is underwater, generating the indication that the camera is underwater.

3. The method of claim 1, further comprising: determining, based on applying a machine learning model, that the camera is underwater; and based on determining that the camera is underwater, generating the indication that the camera is underwater.

4. The method of claim 1 , wherein segmenting the image to determine the plurality of regions of the image comprises applying a texture-based segmentation algorithm to the image, wherein the image comprises a plurality of pixels, wherein applying the texture-based segmentation algorithm comprises determining a difference between adjacent pixels of the plurality of pixels.

5. The method of claim 1, wherein the plurality of regions includes a foreground region and a background region.

6. The method of claim 5, wherein adjusting the image based on the one or more color gain parameters comprises:determining a relative percentage of the foreground region to the background region; based on the relative percentage of the foreground region to the background region, determining one or more combined color gain parameters; and adjusting the image based on the combined color gain parameters.

7. The method of claim 5, wherein adjusting the image based on the one or more color gain parameters comprises: determining one or more combined color gain parameters for the foreground region and one or more combined color gain parameters for the background region; adjusting the foreground region of the image based on the combined color gain parameters of the foreground region; and adjusting the background region of the image based on the combined color gain parameters of the background region.

8. The method of claim 1, wherein adjusting the image based on the one or more color gain parameters comprises: combining the one or more color gain parameters based on the corresponding classification of the region; and adjusting the image based on the combined one or more color gain parameters.

9. The method of claim 1, wherein the one or more color gain parameters for each region of the plurality of regions includes a blue color gain and a red color gain.

10. The method of claim 1, wherein adjusting the image based on the one or more color gain parameters determined for each region of the plurality of regions comprises determining one or more average color gain parameters for each region of the plurality of regions.

11. The method of claim 1, further comprising: determining, based on a depth value, a trust level, wherein adjusting the image based on the one or more color gain parameters determined for each region of the plurality of regions is based on the trust level.

12. The method of claim 1, wherein adjusting the image based on the one or more color gain parameters is based on a depth estimate13. The method of claim 1, further comprising: determining a count of one or more regions of the plurality of regions with a particular corresponding classification; and based on the count of the one or more regions being less than a threshold value, determining a trust level, wherein adjusting the image based on the one or more color gain parameters determined for each region of the plurality of regions is based on the trust level.

14. The method of claim 1, further comprising: determining a count of one or more regions of the plurality of regions with a particular corresponding classification; determining a count of the plurality of regions; and based on a comparison of the count of the one or more regions with the particular corresponding classification and the count of the plurality of regions, determining a trust level, wherein adjusting the image based on the one or more color gain parameters determined for each region of the plurality of regions is based on the trust level.

15. The method of claim 1 , wherein segmenting the image to determine the plurality of regions of the image is based on applying Canny edge detection to the image.

16. The method of claim 1 , wherein segmenting the image to determine the plurality of regions of the image is based on applying a machine learning model to the image.

17. A computing device comprising: a control system configured to: receive an image from a camera; based on an indication that the camera is underwater, segment the image to determine a plurality of regions of the image and a corresponding classification for each region of the plurality of regions; for each region of the plurality of regions, determine one or more color gain parameters based on the corresponding classification of the region; andadjust the image based on the one or more color gain parameters determined for each region of the plurality of regions.

18. The computing device of claim 18, wherein the computing device further comprises a casing system, wherein the casing system includes a depth sensor, wherein the control system is configured to receive a depth measurement from the depth sensor of the casing system.

19. The computing device of claim 18, wherein determining the one or more color gain parameters are based on the depth measurement.

20. A non-transitory computer readable medium storing program instructions executable by one or more processors to cause the one or more processors to perform operations comprising: receiving an image from a camera; based on an indication that the camera is underwater, segmenting the image to determine a plurality of regions of the image and a corresponding classification for each region of the plurality of regions; for each region of the plurality of regions, determining one or more color gain parameters based on the corresponding classification of the region; and adjusting the image based on the one or more color gain parameters determined for each region of the plurality of regions.

Images