Learning-based lens flare removal

Abstract

A method includes obtaining an input image containing a particular lens flare representation, and generating a de-glare image by processing the input image by a machine learning model, the de-glare image including the input image with at least a portion of the particular lens flare representation removed. The machine learning (ML) model may be trained by generating training images that combine a corresponding reference image with a corresponding lens flare image. For each corresponding training image, a modified image may be determined by processing the corresponding training image by the ML model, and a loss value may be determined based on a loss function comparing the modified image to a corresponding reference image used to generate the corresponding training image. Parameters of the ML model may be adjusted based on the loss value determined for each corresponding training image and the loss function.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 084,377, filed on September 28, 2020, which is incorporated herein by reference as if fully set forth in this specification. Technical Field

[0003] The present application relates to learning-based lens flare removal. Background Art

[0004] Lens flare is a common artifact in photographs caused by pointing a camera at a strong light source. Lens flare can blur or interfere with the representation of parts of the environment, thereby reducing the image quality of the photograph. Summary of the invention

[0005] A machine learning model, such as a convolutional neural network, may be trained to determine lens flare representations in an image. Specifically, the machine learning model may be configured to process an input image including a lens flare representation to generate a de-flared version of the input image in which at least some of the lens flare representations are removed. The machine learning model may be trained based on a plurality of training images formed by combining a plurality of reference images lacking lens flare representations with a plurality of lens flare images representing lens flare. Some of the lens flare images may be captured experimentally using a physical camera, while others may be generated based on computer simulations of a camera.

[0006] In a first example embodiment, a method may include obtaining (i) a plurality of reference images and (ii) a plurality of lens flare images. The method may also include generating a plurality of training images by combining each respective reference image of the plurality of reference images with a respective lens flare image of the plurality of lens flare images. The method may also include determining a modified image for each respective training image of the plurality of training images by processing the respective training images by a machine learning model. The modified image may include the respective training image from which at least a portion of the respective lens flare representation is removed. The method may also include determining a loss value for each respective training image of the plurality of training images based on a loss function, the loss function being configured to compare the modified image with a respective reference image used to generate the respective training image. The method may also include adjusting one or more parameters of the machine learning model based on (i) the loss value determined for each respective training image and (ii) the loss function.

[0007] In a second example embodiment, a system may include a processor and a non-transitory computer-readable medium having instructions stored thereon, which, when executed by the processor, cause the processor to perform operations. The operations may include obtaining (i) a plurality of reference images and (ii) a plurality of lens flare images. The operations may also include generating a plurality of training images by combining each respective reference image in the plurality of reference images with a respective lens flare image in the plurality of lens flare images. The operations may also include, for each respective training image in the plurality of training images, determining a modified image by processing the respective training image by a machine learning model. The modified image may include the respective training image in which at least a portion of the respective lens flare representation is removed. The operations may also include determining a loss value for each respective training image in the plurality of training images based on a loss function, the loss function being configured to compare the modified image with a respective reference image used to generate the respective training image. The operations may also include adjusting one or more parameters of the machine learning model based on (i) the loss value determined for each respective training image and (ii) the loss function.

[0008] In a third example embodiment, an article of manufacture may include a non-transitory computer-readable medium having instructions stored thereon, which, when executed by a computing device, cause the computing device to perform operations. The operations may include obtaining (i) a plurality of reference images and (ii) a plurality of lens flare images. The operations may also include generating a plurality of training images by combining each respective reference image in the plurality of reference images with a respective lens flare image in the plurality of lens flare images. The operations may also include, for each respective training image in the plurality of training images, determining a modified image by processing the respective training image by a machine learning model. The modified image may include the respective training image in which at least a portion of the respective lens flare representation is removed. The operations may also include determining a loss value for each respective training image in the plurality of training images based on a loss function, the loss function being configured to compare the modified image with a respective reference image used to generate the respective training image. The operations may also include adjusting one or more parameters of the machine learning model based on (i) the loss value determined for each respective training image and (ii) the loss function.

[0009] In a fourth example embodiment, a system may include means for obtaining (i) a plurality of reference images and (ii) a plurality of lens flare images. The system may also include means for generating a plurality of training images by combining each respective reference image of the plurality of reference images with a respective lens flare image of the plurality of lens flare images. The system may also include means for determining a modified image for each respective training image of the plurality of training images by processing the respective training images by a machine learning model. The modified image may include a respective training image in which at least a portion of the respective lens flare representation is removed. The system may also include means for determining a loss value for each respective training image of the plurality of training images based on a loss function, the loss function being configured to compare the modified image with a respective reference image used to generate the respective training image. The operation may also include adjusting one or more parameters of the machine learning model based on (i) the loss value determined for each respective training image and (ii) the loss function.

[0010] In a fifth example embodiment, a method may include obtaining an input image containing a particular lens flare representation. The method may also include processing the input image by a machine learning model to generate a de-glare image, the de-glare image including the input image with at least a portion of the particular lens flare representation removed. The machine learning model may have been trained to determine the lens flare representation in an image. The method may also include outputting the de-glare image.

[0011] In a sixth example embodiment, a system may include a processor and a non-transitory computer-readable medium having instructions stored thereon, which, when executed by the processor, cause the processor to perform operations. The operations may include obtaining an input image containing a particular lens flare representation. The operations may also include processing the input image by a machine learning model to generate a de-glare image, the de-glare image including the input image with at least a portion of the particular lens flare representation removed. The machine learning model may have been trained to determine the lens flare representation in an image. The operations may also include outputting the de-glare image.

[0012] In a seventh example embodiment, an article of manufacture may include a non-transitory computer-readable medium having instructions stored thereon, which, when executed by a computing device, cause the computing device to perform operations. The operations may include obtaining an input image containing a particular lens flare representation. The operations may also include processing the input image by a machine learning model to generate a de-glare image, the de-glare image including the input image with at least a portion of the particular lens flare representation removed. The machine learning model may have been trained to determine the lens flare representation in an image. The operations may also include outputting the de-glare image.

[0013] In an eighth example embodiment, a system may include means for obtaining an input image containing a particular lens flare representation. The system may also include means for processing the input image by a machine learning model to generate a de-glare image, the de-glare image including the input image with at least a portion of the particular lens flare representation removed. The machine learning model may have been trained to determine the lens flare representation in an image. The system may also include means for outputting the de-glare image.

[0014] These and other embodiments, aspects, advantages and alternatives will become clear to those of ordinary skill in the art by reading the following detailed description and referring to the accompanying drawings as appropriate. In addition, the overview and other descriptions and drawings provided herein are intended to illustrate the embodiments only by way of example, and therefore, multiple variations are possible. For example, structural elements and process steps may be rearranged, combined, distributed, eliminated or otherwise changed while remaining within the scope of the claimed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A computing device according to examples described herein is shown.

[0016] Figure 2 A computing system according to examples described herein is shown.

[0017] Figure 3A A system according to the examples described herein is shown.

[0018] Figure 3B and Figure 3C An image according to examples described herein is shown.

[0019] Figure 4 A system for training a machine learning model according to the examples described herein is shown.

[0020] Figure 5 A system according to the examples described herein is shown.

[0021] Figure 6 A flow chart according to examples described herein is shown.

[0022] Figure 7 A flow chart according to examples described herein is shown. DETAILED DESCRIPTION

[0023] Example methods, devices, and systems are described herein. It should be understood that the words "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "example," "exemplary," and / or "illustrative" is not necessarily to be construed as preferred or advantageous over other embodiments or features unless so stated. Therefore, other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein.

[0024] Therefore, the example embodiments described herein are not meant to be limiting.It will be readily understood that aspects of the present disclosure, as generally described herein and illustrated in the accompanying drawings, may be arranged, substituted, combined, separated and designed in a variety of different configurations.

[0025] Furthermore, unless the context indicates otherwise, the features shown in each of the figures may be used in combination with each other. Figure 1 The illustrated features should generally be viewed as forming aspects of one or more overall embodiments, and it should be understood that not all illustrated features are necessary for every embodiment.

[0026] In addition, any enumeration of elements, blocks, or steps in this specification or claims is for the purpose of clarity. Therefore, such enumeration should not be interpreted as requiring or implying that these elements, blocks, or steps follow a specific arrangement or are performed in a specific order. Unless otherwise specified, the drawings are not drawn to scale.

[0027] 1. Overview

[0028] Lens flare is an optical phenomenon observed when a camera captures light emitted by a relatively strong / bright light source. Lens flare may cause and / or be represented as streaks, lines, and / or irregular spots in the image data, as well as other possible visual artifacts / patterns. The lens flare representation in the image data may be caused by internal reflections of light by the optical components (e.g., lens) of the camera, and the specific visual pattern of the representation may vary depending on the optical properties of the optical components. The lens flare representation in the image data may also be caused by scattering and / or diffraction of light caused by imperfections (imperfections) present in and / or on the optical components of the camera, and the specific visual pattern of the representation may vary depending on the specific type, size, and / or spatial distribution of the imperfections.

[0029] In general, light from multiple parts of a scene may experience such internal reflections, scattering, and / or diffraction. When the intensity of the incident light is relatively low, the internally reflected, scattered, and / or diffracted light may not result in a perceptible representation of lens flare. However, when the intensity of the incident light is relatively high, such as when there are strong / bright light sources within the scene, the internally reflected, scattered, and / or diffracted light may result in a perceptible representation of lens flare. In some cases, it may be desirable to remove the lens flare representation from the image to represent the corresponding scene as it would appear without the lens flare. In other cases, it may be desirable to adjust the lens flare representation by, for example, enhancing the representation, weakening the representation, and / or adjusting the color of the representation, as well as other possible ways.

[0030] For any particular imaging scene, lens flare may occur due to a variety of complex optical phenomena that are difficult to predict and / or control. Therefore, lens flare representations are typically removed through manual processing of the image, in which a user identifies lens flare in the image through visual inspection and modifies the image to remove or reduce the lens flare. However, such manual image processing is time consuming and requires skilled image manipulation. The challenges associated with controlling lens flare in imaging scenes mean that automated image processing is needed to remove lens flare representations in images. Due to the challenges associated with controlling lens flare in imaging scenes, existing lens flare techniques generally attempt to characterize shape properties of lens flare representations and use such properties to identify lens flare representations. The various optical phenomena that may cause lens flare mean that techniques that use characteristic properties of lens flare representations generally perform poorly.

[0031] Thus, a machine learning model, such as a convolutional neural network (CNN), may be trained to determine a lens flare representation in an image. Specifically, determining, by the machine learning model, a particular lens flare representation in an input image may include (i) generating a restored lens flare image representing lens flare and / or (ii) generating a de-glare image comprising the input image in which at least a portion of the particular lens flare representation is removed.

[0032] It may be difficult, time consuming, and / or infeasible to capture training image pairs, where each training image pair includes (i) a scene and a corresponding representation of lens flare that appears due to a light source that forms part of the scene, and (ii) a corresponding representation of the scene without a corresponding representation of the lens flare. In particular, it may be difficult, time consuming, and / or infeasible to capture such training image pairs because aspects of a camera device that cause lens flare may not be easily controllable, if controllable at all.

[0033] Thus, training data for a machine learning model may be generated by combining (i) a baseline image without glare and (ii) a lens flare image with only glare. The baseline image without glare may be captured / generated independently of the glare-only image. Thus, each training image in the training data may be explicitly separated into (i) a representation of the corresponding non-glare scene and (ii) a corresponding representation of the lens flare for providing supervision to the machine learning model. The lens flare images may be generated using computer simulations of a camera device and a light source and / or experimental / physical capture of images of a physical light source with a physical camera device.

[0034] In particular, the simulated lens flare image generated by the computer simulation may at least represent the lens flare caused by the scattering and / or diffraction of light caused by various optical imperfections. The optical imperfections may be difficult to control and / or vary on a physical camera, but may be relatively easier to simulate, thus allowing the deterministic generation of various visual patterns (e.g., a first lens flare pattern) caused by the scattering and / or diffraction. The physically captured image may at least represent the lens flare caused by the internal reflection of light by the optical components (as well as the scattering and / or diffraction of light caused by specific optical imperfections associated with the corresponding physical camera device). The internal reflections may be difficult to simulate because the simulation may be based on an accurate model of the optical components (which may not be publicly available), but may be relatively easier to capture physically, thus allowing the deterministic generation of various visual patterns (e.g., a second lens flare pattern) caused by the internal reflections.

[0035] In general, the simulated lens flare images and experimental lens flare images can provide representative samples of various possible representations of lens flare that may be present in the image. In addition, although lens flare is caused by the presence of a strong light source in the scene, the visual pattern represented by the lens flare is generally independent of the scene components because the presence and / or arrangement of objects in the scene (other than the light source) is unlikely to cause and / or affect the lens flare. Therefore, synthetic training data generated by combining different, independently generated reference images and lens flare images does not impair the ability of the machine learning model to extrapolate to real / non-synthetic scenes at inference time because the removal of lens flare is unlikely to depend on the scene components. In fact, because a variety of different scenes and glare combinations are available, synthetic training data generated in this manner may be desirable, which can improve the ability of the trained model to extrapolate to new real / non-synthetic scenes.

[0036] The trained machine learning model can process an input image including a representation of lens flare to generate a de-glare image. The de-glare image can correspond to the input image in which at least a portion of the representation of lens flare is removed. In one example, the machine learning model can be configured to directly generate the de-glare image (i.e., the de-glare image can be the output of the machine learning model). In another example, the machine learning model can be configured to generate a restored lens flare image, which can be subtracted from the input image to generate the de-glare image. In either case, the machine learning model can be configured to determine / identify a representation of lens flare within the input image.

[0037] Additionally or alternatively, a restored lens flare image can be generated by subtracting the de-flare image from the input image. Thus, the restored lens flare image can represent the portion of the input image that was a lens flare representation determined / identified by the machine learning model. In some cases, the intensity and / or color of various aspects of the restored lens flare can be modified and added back to the de-flare image, thereby generating a version of the input image in which the visual appearance of the lens flare representation has been altered in a particular manner.

[0038] The machine learning model may be configured to operate on monoscopic image data captured using a monoscopic camera. Thus, the machine learning model may not rely on the input images being captured using any particular type of hardware (e.g., a stereo camera) and / or including any annotations (e.g., disparity maps) other than the image data itself.

[0039] In some cases, the machine learning model may have been trained to determine the lens flare representation by determining a representation of the light source that causes the lens flare. Thus, the representation of the light source may be determined / identified along with the lens flare representation, and may therefore be removed from the input image when generating the de-glare image. Thus, a light source mask may be generated and used to copy the representation of the light source from the input image to the de-glare image, resulting in a modified version of the input image in which (i) the lens flare representation is removed and (ii) the light source representation is present. In some cases, as described above, the light source representation may be copied to a version of the input image in which its corresponding representation of lens flare has been modified.

[0040] II. Example Computing Devices and Systems

[0041] Figure 1An example computing device 100 is shown. Computing device 100 is shown in the form of a mobile phone. However, computing device 100 may alternatively be implemented as a laptop computer, a tablet computer, or a wearable computing device, among other possible devices. Computing device 100 may include various elements, such as a body 102, a display 106, and buttons 108 and 110. Computing device 100 may also include one or more cameras, such as a front-facing camera 104 and a rear-facing camera 112.

[0042] Front-facing camera 104 may be located on a side of body 102 that generally faces a user when in operation (e.g., on the same side as display 106). Rear-facing camera 112 may be located on a side of body 102 opposite front-facing camera 104. Referring to cameras as front-facing and rear-facing is arbitrary, and computing device 100 may include multiple cameras located on various sides of body 102.

[0043] Display 106 may represent a cathode ray tube (CRT) display, a light emitting diode (LED) display, a liquid crystal (LCD) display, a plasma display, an organic light emitting diode (OLED) display, or any other type of display known in the art. In some examples, display 106 may display a digital representation of a current image captured by front camera 104 and / or rear camera 112, and / or an image that can be captured or recently captured by one or more of these cameras. Thus, display 106 may be used as a viewfinder for a camera. Display 106 may also support a touch screen function that can adjust settings and / or configurations of any aspect of computing device 100.

[0044] The front camera 104 may include an image sensor and associated optical elements, such as a lens. The front camera 104 may provide zoom capabilities or may have a fixed focal length. In other examples, interchangeable lenses may be used with the front camera 104. The front camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. The front camera 104 may also be configured to capture still images, video images, or both. In addition, the front camera 104 may represent, for example, a single-view camera. The rear camera 112 may be arranged similarly or differently. In addition, one or more of the front camera 104 and / or the rear camera 112 may be an array of one or more cameras. In one example, the rear camera 112 may constitute and / or form part of a stereo camera.

[0045] One or more of the front camera 104 and / or the rear camera 112 may include or be associated with an illumination assembly that provides a light field to illuminate a target object. For example, the illumination assembly may provide flash illumination or constant illumination of the target object. The illumination assembly may also be configured to provide a light field that includes one or more of structured light, polarized light, and light with specific spectral content. Other types of light fields that are known and used to recover a three-dimensional (3D) model from an object are possible in the context of the examples herein.

[0046] The computing device 100 may also include an ambient light sensor that may continuously or from time to time determine the ambient brightness of a scene that may be captured by the cameras 104 and / or 112. In some implementations, the ambient light sensor may be used to adjust the display brightness of the display 106. Additionally, the ambient light sensor may be used to determine an exposure length for one or more of the cameras 104 or 112, or to assist in such determination.

[0047] The computing device 100 may be configured to capture images of a target object using the display 106 and the front camera 104 and / or the rear camera 112. The captured images may be a plurality of still images or a video stream. Image capture may be triggered by activating a button 108, pressing a softkey on the display 106, or by some other mechanism. Depending on the implementation, images may be captured automatically at specific time intervals (e.g., when the button 108 is pressed, when the target object is in appropriate lighting conditions, when the digital camera device 100 is moved a predetermined distance, or according to a predetermined capture schedule).

[0048] Figure 2 2 is a simplified block diagram illustrating some of the components of an example computing system 200. By way of example and not limitation, computing system 200 may be a cellular mobile phone (e.g., a smart phone), a computer (such as a desktop, notebook, tablet, or handheld computer), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a gaming console, a robotic device, a vehicle, or some other type of device. Computing system 200 may represent aspects of computing device 100, for example.

[0049] like Figure 2As shown, computing system 200 may include communication interface 202, user interface 204, processor 206, data storage 208, and camera assembly 224, all of which may be communicatively linked together via a system bus, network, or other connection mechanism 210. Computing system 200 may be equipped with at least some image capture and / or image processing capabilities. It should be understood that computing system 200 may represent a physical image processing system, a specific physical hardware platform on which image sensing and processing applications operate in software, or other combinations of hardware and software configured to perform image capture and / or processing functions.

[0050] The communication interface 202 may allow the computing system 200 to communicate with other devices, access networks, and / or transport networks using analog or digital modulation. Thus, the communication interface 202 may facilitate circuit-switched communications and / or packet-switched communications, such as plain old telephone service (POTS) communications and / or Internet Protocol (IP) or other packetized communications. For example, the communication interface 202 may include a chipset and an antenna arranged for wireless communication with a radio access network or access point. Furthermore, the communication interface 202 may take the form of or include a wired interface, such as an Ethernet, Universal Serial Bus (USB), or High Definition Multimedia Interface (HDMI) port. The communication interface 202 may also take the form of or include a wireless interface, such as Wi-Fi, The communication interface 202 may include a global positioning system (GPS) or a wide area wireless interface (e.g., WiMAX or 3GPP Long Term Evolution (LTE)). However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used on the communication interface 202. In addition, the communication interface 202 may include multiple physical communication interfaces (e.g., Wi-Fi interfaces, interface and wide area wireless interface).

[0051] User interface 204 can allow computing system 200 to interact with the human or non-human user, such as receiving input from the user and providing output to the user.Therefore, user interface 204 can include input components, such as keypad, keyboard, touch-sensitive panel, computer mouse, trackball, joystick, microphone etc.User interface 204 can also include one or more output components, such as display screen, which for example can be combined with touch-sensitive panel.Display screen can be based on CRT, LCD and / or LED technology or based on other technologies known now or developed later.User interface 204 can also be configured to generate auditory output via loudspeaker, speaker jack, audio output port, audio output device, earphone and / or other similar equipment.User interface 204 can also be configured to receive and / or capture audible speech, noise and / or signal by microphone and / or other similar equipment.

[0052] In some examples, user interface 204 may include a display that serves as a viewfinder for still camera functionality and / or video camera functionality supported by computing system 200. Additionally, user interface 204 may include one or more buttons, switches, knobs, and / or dials that facilitate configuration of the camera's focus functionality and the capture of images. Some or all of these buttons, switches, knobs, and / or dials may be implemented via a touch-sensitive panel.

[0053] Processor 206 may include one or more general purpose processors (e.g., microprocessors) and / or one or more special purpose processors (e.g., digital signal processors (DSPs), graphics processing units (GPUs), floating point units (FPUs), network processors, or application specific integrated circuits (ASICs)). In some cases, the special purpose processors may be capable of image processing, image alignment, and merging images, among other possible processing. Data storage 208 may include one or more volatile and / or non-volatile storage components, such as magnetic storage, optical storage, flash memory, or organic storage, and may be integrated in whole or in part with processor 206. Data storage 208 may include removable and / or non-removable components.

[0054] The processor 206 may be capable of executing program instructions 218 (e.g., compiled or non-compiled program logic and / or machine code) stored in the data store 208 to perform various functions described herein. Thus, the data store 208 may include a non-transitory computer-readable medium having program instructions stored thereon, which, when executed by the computing system 200, causes the computing system 200 to perform any method, process, or operation disclosed in this specification and / or the accompanying drawings. The execution of the program instructions 218 by the processor 206 may cause the processor 206 to use the data 212.

[0055] For example, program instructions 218 may include an operating system 222 (e.g., an operating system kernel, device drivers, and / or other modules) and one or more applications 220 (e.g., a camera function, an address book, email, web browsing, social networking, an audio-to-text function, a text translation function, and / or a gaming application) installed on computing system 200. Similarly, data 212 may include operating system data 216 and application data 214. Operating system data 216 may be primarily accessible by operating system 222, while application data 214 may be primarily accessible by one or more applications 220. Application data 214 may be arranged in a file system that is visible or hidden to a user of computing system 200.

[0056] Applications 220 may communicate with operating system 222 via one or more application programming interfaces (APIs). These APIs may facilitate, for example, application 220 reading and / or writing application data 214, sending or receiving information via communication interface 202, receiving and / or displaying information on user interface 204, and the like.

[0057] In some cases, the application 220 may be referred to simply as an "app". In addition, the application 220 may be downloadable to the computing system 200 through one or more online application stores or application markets. However, the application may also be installed on the computing system 200 in other ways, such as via a web browser or through a physical interface (e.g., a USB port) on the computing system 200.

[0058] The camera assembly 224 may include, but is not limited to, an aperture, a shutter, a recording surface (e.g., photographic film and / or an image sensor), a lens, a shutter button, an infrared projector, and / or a visible light projector. The camera assembly 224 may include components configured to capture images in the visible light spectrum (e.g., electromagnetic radiation having a wavelength of 380-700 nanometers) and components configured to capture images in the infrared light spectrum (e.g., electromagnetic radiation having a wavelength of 701 nanometers-1 millimeter). The camera assembly 224 may be controlled, at least in part, by software executed by the processor 206.

[0059] 3. Example system for removing lens flare representations from images

[0060] Figure 3A An example system for removing a representation of lens flare from an image is shown. Specifically, Figure 3ASystem 300 is shown, comprising machine learning model 304, glare controller 310, mask generator 314, and inverse operator 318. System 300 may form part of computing device 100 and / or computing system 200, etc. System 300 may represent hardware components (e.g., dedicated circuits), software instructions, or a combination thereof.

[0061] System 300 can be configured to receive input image 302 and, based thereon, generate modified input image 312, modified input image 320, and / or a combination of modified input images 312 and 320, each of which can be a respective modified version of input image 302. Specifically, input image 302 can include a lens flare representation therein. Machine learning model 304 can be configured to determine and / or recognize at least a portion of the lens flare representation within input image 302.

[0062] In one example, determining, by the machine learning model 304, the representation of lens flare in the input image 302 may include generating a new image that (i) lacks at least a portion of the representation of lens flare and (ii) retains a representation of the underlying scene of the input image 302. Thus, the machine learning model 304 may be configured to generate a de-glare image 306 corresponding to the input image 302, in which at least a portion of the representation of lens flare is removed from the input image 302. The machine learning model 304 may include, for example, a convolutional artificial neural network that may be arranged according to a U-Net architecture or a VGG-19 architecture, or the like.

[0063] The system 300 may also be configured to subtract the de-glare image 306 from the input image 302, thereby generating a restored lens flare image 308. The restored lens flare image 308 may contain a representation of the lens flare that the machine learning model 304 has determined in the input image 302 and that has been removed from the input image 302 when generating the de-glare image 306. Thus, the sum of the de-glare image 306 and the restored lens flare image 308 may be equal to the input image 302.

[0064] In a second example, determining, by the machine learning model 304, the lens flare representation in the input image 302 may alternatively include generating a new image that (i) represents at least a portion of the lens flare representation and (ii) lacks a representation of the underlying scene of the input image 302. That is, the machine learning model 304 may alternatively be configured to generate a restored lens flare image 308 that represents at least a portion of the lens flare representation determined / identified in the input image 302. Thus, the system 300 may alternatively be configured to subtract the restored lens flare image 308 from the input image 302 to generate the de-glare image 306.

[0065] The glare controller 310 may be configured to generate one or more glare modification values ​​that are multiplied with a plurality of pixels of the restored lens flare image 308 to modify the visual appearance of the lens flare representation. Specifically, when determining the de-glare image 306, some pixels in the input image 302 that do not represent lens flare may not be modified, and thus corresponding pixels in the restored lens flare image 308 may have a zero value. Thus, each non-zero pixel in the restored lens flare image 308 may be modified according to the one or more glare modification values ​​generated by the glare controller 310.

[0066] Specifically, the intensity and / or color of the lens flare representation in the restored lens flare image 308 may be modified by multiplying each color component (e.g., red, green, blue) of the pixels of the restored lens flare image 308 by a corresponding value. For example, the intensity of the lens flare representation may be increased by multiplying each color component by a value greater than 1. In another example, the color of the lens flare representation may be modified by multiplying each color component by a different value to obtain a desired color (e.g., multiplying the green and blue pixel components by 1 and the red pixel component by 2 to make the lens flare representation appear redder). The lens flare representation may be completely removed by multiplying each of the non-zero values ​​in the restored lens flare image 308 by zero, so that the modified input image 312 is equal to the de-glare image 306.

[0067] The system 300 may be configured to, after multiplying the pixels of the restored lens flare image 308 by one or more flare modification values, add the multiplied restored lens flare image 308 to the de-flare image 306, thereby generating a modified input image 312. The modified input image 312 may represent the input image 302 in which the lens flare representation is modified according to the one or more flare modification values. For example, the modified input image 312 may be equivalent to the input image 302 in which the lens flare representation is increased, decreased, and / or the color of the representation is modified, etc. Thus, by generating a restored lens flare image 308, modifying it using the flare controller 310, and adding the modified version back to the flare image 306, the system 300 may allow for various visual modifications to the lens flare representation in the input image 302.

[0068] In some implementations, the machine learning model 304 may also be configured to determine and / or identify one or more light source representations that cause lens flare within the input image 302 (e.g., the machine learning model 304 may treat the one or more light sources as the same as the lens flare). Thus, in such implementations, the de-glare image 306 may not represent the one or more light sources that cause the lens flare. However, in some cases, it may be desirable to remove the lens flare representation from the input image 302 while maintaining the corresponding one or more light source representations that cause the lens flare. Therefore, the system 300 may include a mask generator 314 configured to generate a light source mask 316 that can be used to add the one or more light source representations that cause the lens flare back to the glare image 306.

[0069] Specifically, the light source mask 316 may include, for example, a grayscale image including a plurality of pixels and having the same resolution as the input image 302 and / or the de-glare image 306. A pixel having a first value (e.g., 255) may indicate that the corresponding pixel in the input image 302 represents a light source, while a pixel having a second value (e.g., 0) may indicate that the corresponding pixel in the input image 302 does not represent a light source. Values ​​between the first value and the second value may be used to create a smooth and / or gradual transition between an area representing a light source and an area not representing a light source.

[0070] The mask generator 314 can be configured to determine that a particular pixel in the input image 302 represents a light source based on, for example, that a brightness value associated with the particular pixel exceeds a threshold brightness value (e.g., 0.95). Therefore, the mask generator 314 can be configured to generate the light source mask 316 by assigning a corresponding value (e.g., 255 or 0) to each pixel of the light source mask 316 based on whether the brightness value of the spatially corresponding pixel in the input image 302 exceeds the threshold brightness value.

[0071] The system 300 can be configured to multiply the light source mask 316 and the input image 302 to generate a first masked image that includes the light sources represented in the input image 302 but excludes other features represented therein. In addition, the system 300 can be configured to multiply the de-glare image 306 by the inverse of the light source mask 316 to generate a second masked image that excludes areas in the de-glare image 306 that correspond to the light sources but include all other features of the de-glare image 306. The system 300 can also be configured to add the first masked image and the second masked image to generate a modified input image 320. Thus, the system 300 can implement the function I B =I F ⊙M+I DE-FLARED ⊙(1-M), where I Brepresents the modified input image 320, I F represents the input image 302, M represents the light source mask 316, I DE-FLARED denotes the de-glare image 306, and ⊙ denotes a pixel multiplication. Thus, the modified input image 320 may represent the input image 302 with the lens flare representation removed from the input image 302 and including representations of one or more light sources causing the lens flare.

[0072] Alternatively, in some cases, the system 300 may instead be configured to apply the light source mask 316 (modified by the inverse operator 318) to the modified input image 312, rather than applying the light source mask 316 (modified by the inverse operator 318) to the de-glare image 306. Thus, the lens flare representation contained in the input image 302 may be modified according to the glare controller 310, while the light sources causing the lens flare representation may remain unmodified. Thus, the modified input image 320 may represent the input image 302 in which the lens flare representation is modified and includes one or more light source representations that cause the lens flare. Alternatively, the same effect may be achieved by applying the light source mask 316 (modified by the inverse operator 318) to the output of the glare controller 310, such that the light source representations in the restored lens flare image 308 are added back to the glare image 306 without modification.

[0073] Figure 3B and Figure 3C An example visual illustration includes an input image 302, a deglare image 306, a restored lens flare image 308, a light source mask 316, a modified input image 312, and a modified input image 320. Specifically, the input image 302 may include a representation 330 of a light source (e.g., the sun), a representation 332 of lens flare caused by the light source (indicated by a line projected from the light source), and a representation 334 of a mountain that forms part of the underlying scene represented by the input image 302. The deglare image 306 includes the representation 334 of the mountain, but lacks the lens flare representation 332. In some cases, the deglare image 306 may also lack the representation 330 of the light source, as shown by the shadow pattern thereon. The restored lens flare image 308 includes the lens flare representation 332 and, in some cases, includes the representation 330 of the light source.

[0074] Light source mask 316 includes mask region 336 corresponding to representation 330 of light source, and mask region 338 corresponding to the remainder of input image 302 (i.e., region not representing light source causing lens flare). Mask region 336 is shown in white, indicating that its pixels are assigned a first value (e.g., 255 in a grayscale image, or 1 in a binary image), and thus "pass" all corresponding pixels of input image 302, while mask region 338 is shown in black, indicating that its pixels are assigned a second value (e.g., 0 in a grayscale image and / or a binary image), and thus "block" all corresponding pixels of input image 302. Intermediate pixel values ​​are not shown as part of light source mask 316, but may be used to create a gradual transition between mask region 336 and mask region 338. The inverse (not shown) of light source mask 316 includes each pixel of mask region 336 being assigned a second value (e.g., 0, corresponding to black), and thus being configured to “block” all corresponding pixels of de-glare image 306, and each pixel of mask region 338 being assigned a first value (e.g., 255 or 1, corresponding to white), and thus being configured to “pass” all corresponding pixels of de-glare image 306.

[0075] Modified input image 312 includes representations 330, 332, and 334, where representation 332 (and in some cases, representation 330) has been modified to enhance the visual appearance of lens flare, as indicated by the corresponding lines being thicker than shown in input image 302. In other cases, representation 332 may alternatively and / or additionally be modified by changing its color and / or by reducing its intensity (e.g., reducing the brightness of corresponding pixels), among other possible ways. Modified input image 320 includes representations 330 and 334, and excludes representation 332. Thus, modified input image 320 represents the corresponding scene as it would appear without the lens flare.

[0076] IV. Example training operation

[0077] Figure 4An example system and process for training a machine learning model 304 to determine / identify a lens flare representation from an image is shown. Specifically, the machine learning model 304 can be trained based on a training image 420, which can be generated based on a reference image 400 and a lens flare image 406. The reference image 400 may include reference images 402 to 404 (i.e., reference images 402-404). The reference image 400 may represent a plurality of different scenes / environments. In some cases, the reference image 400 or a subset thereof may include an image without a lens flare representation, and thus may be considered to represent a "clean" scene / environment. Therefore, due to the lack of a lens flare representation, the reference image 400 may alternatively be referred to as a natural image, a clean image, a glare-free image, and / or a glare-free natural image. Each reference image of the reference image 400 may be represented using a variable I0.

[0078] The lens flare image 406 may include a simulated lens flare image 408 and / or an experimental lens flare image 414. The simulated lens flare image 408 may include lens flare images 410 to 412 (i.e., lens flare images 410-412), and the experimental lens flare image may include lens flare images 416 to 418 (i.e., lens flare images 416-418). The lens flare image 406 may include a plurality of different lens flare representations. In some cases, the lens flare image 406 may represent a plurality of different lens flare variations relative to a monochromatic (e.g., black) background rather than relative to a non-uniform background (e.g., an ambient representation), so that the portion of the lens flare image 406 representing the lens flare may be easily separated / distinguished from the portion that does not represent the lens flare. Therefore, the lens flare image 406 may be alternatively referred to as a glare-only image because it includes only the lens flare representation. Each lens flare image of the lens flare image 406 may be represented using a variable F.

[0079] The simulated lens flare image 408 may be generated based on a computer simulation of at least (i) a light source and (ii) one or more camera devices (including one or more corresponding optical components thereof). The simulated lens flare image 408 may represent lens flare caused by scattering and / or diffraction of light caused by imperfections (e.g., dust, scratches, etc.) present on the corresponding optical components of the camera device. Because such imperfections are random, it may be challenging, time-consuming, difficult, and / or infeasible to experimentally collect multiple images representing lens flare caused by various imperfections (which represent imperfections that may be encountered in practice). However, the simulation allows control of these various imperfections, thereby allowing the generation of images that cover and / or represent various possible imperfections that may be encountered in practice. In some cases, the simulated lens flare image 408 may additionally or alternatively represent lens flare caused by internal reflections of light from an optical component (e.g., a lens) of the camera device.

[0080] In one example, a computer simulation may include generating a point spread function for a camera device based on a Fourier transform of a pupil function of the camera device. Specifically, the pupil function can be based on (i) the aperture function A, (ii) the linear phase shift term φ S (x / z, y / z) and (iii) the defocus term φ DF The point spread function can be expressed as the product of (i) the aperture function and (ii) the exponential of the sum of the linear phase shift term and the defocus term, that is,

[0081] The aperture function A may alternatively be referred to as an amplitude, and may represent one or more simulated imperfections (e.g., the presence of dust or scratches) of an aperture and / or one or more lenses of a camera device. Thus, a plurality of different aperture functions may be used to represent a plurality of different imperfections. For example, to simulate a plurality of different imperfections, a plurality of different lines and dots of various sizes and / or various transparencies may be generated (e.g., randomly) on a disk representing an aperture and / or one or more lenses of a camera device.

[0082] Linear phase shift term φ S (x / z, y / z) can represent the phase shift caused by one or more lenses of the camera device. The linear phase shift term φ S (x / z, y / z) may be determined based on the angle at which the light is incident on one or more lenses of the camera device, and thus may be based on a simulated relative position between the camera device and the light source. DF (z) can represent the degree to which the simulated light source is out of focus based on the depth of the simulated light source relative to one or more lenses of the camera device. Therefore, the defocus term φ DF (z) may also be determined based on a simulated relative position between the camera device and the light source.

[0083] Point Spread Function may be specific to a single wavelength λ0. Thus, to simulate a light source comprising multiple wavelengths (e.g., corresponding to the visible portion of the electromagnetic spectrum), the point spread function may be plotted at multiple different wavelengths. For example, the point spread function The sample can be sampled from 380 nm to 740 nm in 5 nm increments to generate a multi-channel (e.g., 73-channel) point spread function (PSF) representing a light source emitting multiple different wavelengths. λ In some implementations, the point spread function Sampling may be performed along different wavelength ranges and / or at different increments.

[0084] Multi-channel point spread function PSF λ It can be used to generate simulated sensor measurements, thereby obtaining a corresponding simulated lens flare image in the simulated lens flare images 408. Specifically, the multi-channel point spread function PSF λ The analog sensor measurements are generated by multiplying the color matrix C corresponding to the image sensor of the analog camera device. The color matrix C can represent, for example, a transformation from the point spread function space to the image space (represented as, for example, red-green-blue (RGB) pixel values). Therefore, the PSF can be calculated according to the function RGB = C(PSF λ ) to determine the simulated lens flare image. Multiple different color matrices can be used to simulate different types of image sensors.

[0085] The simulated lens flare image 408 may include a plurality of different lens flare representations. These different representations may be obtained by varying the relative positions between the aperture function A, the light source, and the camera device (thereby varying the linear phase shift term φ S (x / z,y / z) and the defocus term φ DF (z)), multi-channel point spread function PSF λ and / or color matrix C. In some cases, generation of lens flare image 408 may also include application of one or more optical distortions (eg, barrel distortion, pincushion distortion, etc.) to make the lens flare representation appear more realistic.

[0086] Experimental lens flare images 414 may be generated by capturing images of physical light sources using one or more physical camera devices. Experimental lens flare images 414 may represent lens flare caused by internal reflections of light from an optical component (e.g., a lens) of a camera device. Simulating multiple images representing lens flare caused by internal reflections may be challenging, time consuming, difficult, and / or infeasible because such simulations may depend on accurate optical models of the optical component (e.g., a lens), which may not be publicly available.

[0087] However, representations of lens flare caused by internal reflections are generally similar across similar camera devices and / or similar optical components (e.g., lenses). For example, representations of lens flare may be similar across different instances of a particular camera model. Thus, images that have been physically captured using various (e.g., commonly used) camera devices and / or optical components may overlay and / or represent various possible representations of lens flare associated with the same or similar camera devices and / or optical components. In some cases, experimental lens flare images 414 may additionally or alternatively represent lens flare caused by scattering and / or diffraction of light caused by imperfections (e.g., dust, scratches, etc.) present on the corresponding optical components of the camera device.

[0088] Specifically, the experimental lens flare images 414 may include representations of the light source captured by one or more camera devices from a plurality of different relative positions (e.g., linear and / or angular positions) between the one or more cameras and the light source. The light source and the one or more camera devices may be arranged in an optically isolated environment such that the one or more camera devices capture substantially only light emitted by the light source. Additionally or alternatively, the light source may be photographed relative to a monochromatic (e.g., black) background such that the lens flare and / or representation of the light source may be separable from the background (e.g., based on a representation of the lens flare having a color other than black).

[0089] During the capture of the experimental lens flare image 414, the light source can be fixed, and the camera can be set on a rotating stage and / or a translation stage that allows its posture relative to the light source to be controlled and / or modified (or vice versa). In one example, the corresponding experimental lens flare image can be captured at each degree of an equidistant angle scan from negative fifty degrees to positive fifty degrees along a specific direction of the camera device. In another example, the corresponding experimental lens flare image can be captured at each of a plurality of linear positions of a linear scan along a specific direction of the camera device. In addition, in some cases, the experimental lens flare images captured at adjacent postures can be interpolated to generate additional images representing lens flare that may have been observed between adjacent postures. For example, two adjacent experimental images can be used to generate, for example, 1, 2, 4, 8 or other number of interpolated experimental images.

[0090] Capturing images of a light source from different relative positions can allow for the generation of multiple different representations of lens flare caused by internal reflections, as the visual pattern of the lens flare representation depends on the angle of incidence of the light. Similarly, capturing images of a light source using multiple different camera devices can further increase the number of different representations of lens flare caused by internal reflections, as the visual pattern of the lens flare representation depends on the optical properties of the optical components of each camera device.

[0091] The training images 420 may include training images 422 to 424 (i.e., training images 422-424). Each of the training images 420 may be formed by combining a corresponding reference image of the reference images 400 with a corresponding lens flare image of the lens flare images 406. For example, the training image 422 may be formed by combining the reference image 402 with the lens flare image 410, and the training image 424 may be formed by combining the reference image 404 with the lens flare image 418. In some cases, a specific reference image of the reference images 400 may be combined with a plurality of different lens flare images of the lens flare images 406 to form a plurality of different training images of the training images 420. Similarly, a specific lens flare image of the lens flare images 406 may be combined with a plurality of different reference images of the reference images 400 to form a plurality of different training images of the training images 420. Therefore, the number of the reference images 400 may be different from the number of the lens flare images 406. In some cases, training images 420 may be generated based on simulated lens flare images 408 but not based on experimental lens flare images 414 , or vice versa.

[0092] Each training image of training images 420 may use variable I F It can be expressed as, and can be expressed as, according to the function I F=I0+F+N, where N represents the added noise. Therefore, in one example, a specific reference image can be combined with a corresponding lens flare image by adding multiple pixel values ​​of the specific reference image to the corresponding pixel values ​​of the corresponding lens flare image. Specifically, the pixel values ​​representing lens flare (rather than the monotone background) in the corresponding lens flare image can be added to the values ​​of spatially corresponding pixels in the specific reference image. In some cases, noise can be added to each resulting pixel value to generate the corresponding training pixels of the training image. For example, a normal distribution N(0,σ) with zero mean and σ standard deviation (e.g., σ=0.01) can be generated. 2 ) Gaussian noise is selected in . Noise can also be added in other ways (such as by randomly sampling other types of distributions).

[0093] In another example, the training pixel values ​​may be based on a weighted sum of the reference pixel values ​​and the corresponding lens flare pixel values. In another example, a particular reference image may be combined with a corresponding lens flare image by replacing a plurality of pixel values ​​in the particular reference image with corresponding pixel values ​​from the corresponding lens flare image. Specifically, pixel values ​​of pixels in the corresponding lens flare image that represent lens flare (rather than a monotone background) may replace spatially corresponding pixels in the particular reference image. Thus, pixel values ​​associated with pixels of the particular reference image that did not originally represent lens flare may be replaced with new values ​​obtained from the corresponding lens flare image, such that the resulting training image represents the scene of the reference image with a lens flare representation added thereto.

[0094] By processing one or more corresponding training images by the machine learning model 304, a corresponding modified image 430 may be generated for each of the one or more corresponding training images of the training images 420. The modified image 430 may be used to generate a lens flare image 432 and / or may be generated based on the restored lens flare image 432. The restored lens flare image 432 may include a lens flare representation that was removed from the corresponding training image as a result of the processing by the machine learning model 304. The modified image 430 may be represented as I M , and the restored lens flare image 432 can be represented as I R .

[0095] In some implementations, the machine learning model 304 can be trained to generate the modified image 430 (e.g. Figure 4 ), and the restored lens flare image 432 can be determined by subtracting the modified image 430 from the corresponding training image. Therefore, the machine learning model 304 can be composed of the function f(I F,θ), where θ represents multiple parameter values ​​of the machine learning model 304 (adjustable during training), f(I F ,θ)=I M And I R =I F -I M In other implementations, the machine learning model 304 may instead be trained to generate a restored lens flare image 432 (not shown), and the modified image 430 may be determined by subtracting the restored lens flare image 432 from the corresponding training image. Thus, the machine learning model 304 may be determined by the function f(I F , α), where α represents multiple parameter values ​​of the machine learning model 304 (adjustable during training), f(I F ,α)=I R And I M =I F -I R .

[0096] Once training of machine learning model 304 is complete, modified image 430 may be a de-glare image that includes little or substantially no representation of lens flare found in the corresponding training image. As machine learning model 304 is trained, modified image 430 may represent an attempt to remove the representation of lens flare from the corresponding training image and may be used to adjust one or more parameters of machine learning model 304 to improve the accuracy of removing the representation.

[0097] Specifically, loss function 434 may be configured to compare modified image 430 with a corresponding reference image of a corresponding training image on which modified image 430 is based. For example, when modified image 430 is generated based on processing of training image 422 by machine learning model 304, loss function 434 may be configured to compare modified image 430 with reference image 402 to quantify how accurately reference image 402 has been reconstructed. In one example, loss function 434 may represent an L1 loss between modified image 430 and a corresponding reference image (e.g., reference image 402 in the above example), and may be represented by function L I (θ)=||I M In the case where the machine learning model 304 is not configured to in-paint saturated pixels and / or saturated pixel regions, such saturated pixels and / or saturated pixel regions may be omitted by the loss function 434.

[0098] Additionally or alternatively, loss function 436 may be configured to compare restored lens flare image 432 with a corresponding lens flare image of a corresponding training image that was used to form restored lens flare image 432. For example, when modified image 430 and restored lens flare image 432 are generated based on processing of training image 422 by machine learning model 304, loss function 436 may be configured to compare restored lens flare image 432 with lens flare image 410 to quantify how accurately lens flare image 410 has been removed. In one example, loss function 436 may represent an L1 loss between restored lens flare image 432 and a corresponding lens flare image (e.g., lens flare image 410 in the example above), and may be represented by L F (θ)=||I R In addition to loss function 434, loss function 436 may be used to improve the accuracy of trained machine learning model 304 in removing lens flare representation from an image.

[0099] In some cases, one or more additional loss functions (not shown) may be used for training of machine learning model 304. For example, a third loss function may be configured to compare the results of processing each of (i) modified image 430 and (ii) a corresponding reference image by a pre-trained image processing model (e.g., a pre-trained convolutional neural network). For example, when modified image 430 is generated based on processing of training image 422 by machine learning model 304, the third loss function may be configured to compare the results of processing modified image 430 by the pre-trained image processing model Φ l (I M ) and the result of processing the reference image 402 by the pre-trained image processing model Φ l (I0) for comparison.

[0100] In some cases, the result Φ l (I M ) and the result Φ l (I0) may represent a specific feature layer (indicated by the "l" subscript) of a pre-trained image processing model, and may therefore be considered as a semantic embedding of the corresponding image. Thus, the third loss function may quantify the accuracy with which the machine learning model 304 reconstructs the reference image 402 based on the corresponding semantic embeddings of the modified image 430 and the reference image 402. In one example, the third loss function may represent an L1 loss calculated across l feature layers between the embedding of the modified image 430 and the embedding of the corresponding reference image (e.g., the reference image 402 in the above example), and may be represented by L p (θ)=∑ l λ l ||Φl (I M )-Φ l (I0)||1, where λ l are feature layer specific weights.

[0101] Parameter adjuster 438 can be configured to adjust one or more parameters of machine learning model 304 based on loss function 434 and its output, loss function 436 and its output, and / or one or more additional loss functions and their outputs. For example, parameter adjuster 438 can be configured to calculate a total loss function based on a weighted sum of loss functions 434 and 436 and one or more additional loss functions. Therefore, the total loss function can be expressed as L(θ)=w1L I (θ)+w2L p (θ)+w3L F (θ). The parameter adjuster 438 may be configured to determine the gradient of the total loss function at a point corresponding to the loss value generated thereby. Based on the gradient, the parameter adjuster 438 may be configured to select one or more new values ​​for one or more parameters of the machine learning model 304. The parameter adjuster 438 may be configured to train the machine learning model 304 using, for example, stochastic gradient descent, batch gradient descent, and / or mini-batch gradient descent. Training iterations may be performed until the machine learning model 304 reaches at least a threshold level of accuracy as measured by the total cost function.

[0102] 5. Example Image Resampling

[0103] Figure 5 Aspects of the system 300 are shown being applied to input images of different resolutions. Specifically, the machine learning model 304 can be configured to accept image data having a specific resolution as input and generate image data having a specific resolution as output. For example, Figure 5 As shown, the specific resolution may be 512 pixels by 512 pixels (i.e., 512x512). Therefore, in order to operate on image data having different sizes, the system 300 may include a downsampler 502 and an upsampler 510. For example, an input image 500 having a (first) resolution of 2048 pixels by 2048 pixels (i.e., 2048x2048) may be downsampled by a factor of four by the downsampler 502 to generate an input image 504 having a (second) resolution of 512 pixels by 512 pixels.

[0104] The machine learning model 304 may be configured to process the input image 504 to generate a de-glare image 506 having a resolution of 512 pixels by 512 pixels. The system 300 may be configured to generate a restored lens flare image 508 having a resolution of 512 by 512 pixels based on the difference between the de-glare image 506 and the input image 504. In alternative implementations, the machine learning model 304 may instead be configured to process the input image 504 to generate the restored lens flare image 508, and the system 300 may be configured to generate the de-glare image 506 based on the difference between the restored lens flare image 508 and the input image 504 (or the generation of the de-glare image 506 may be omitted).

[0105] The upsampler 510 may be configured to upsample the restored lens flare image 508 by a factor of four to generate a restored lens flare image 512 having a resolution of 2048 pixels by 2048 pixels. The system 300 may be configured to generate a de-glare image 514 having a resolution of 2048 by 2048 pixels based on a difference between the restored lens flare image 512 and the input image 500.

[0106] Thus, the machine learning model 304 may be applied at a second, smaller resolution of 512 pixels by 512 pixels, while the input and output of the system 300 are provided at a first, higher resolution of 2048 by 2048 pixels. In other cases, other resolutions may be used. For example, an instance of the machine learning model 304 that is specific to a particular camera model may be trained to accept and generate images of a particular resolution, which may correspond to downsampling by a factor of N (e.g., two, four, eight, etc.) the full resolution image data that the particular camera model is configured to generate. The camera-specific instance of the machine learning model 304 may take into account the aspect ratio of the corresponding full resolution image data, and the aspect ratio of its input and output may be different from one to one.

[0107] In addition, the mask generator 314 can be configured to generate the light source mask 316 at a second resolution (e.g., 512x512) used by the machine learning model and / or at a first resolution (e.g., 2048x2048) of the input and output image data. Thus, in a first example, the light source mask 316 having the first resolution (2048x2048) can be applied to the input image 500 and the de-glare image 514. Alternatively, in a second example, the light source mask 316 having the second resolution (512x512) can be applied to the input image 504 and the de-glare image 506 and then upsampled to the second resolution (2048x2048).

[0108] 6. Additional sample operations

[0109] Figure 6A flow chart of operations associated with training a machine learning model for processing an image containing a representation of lens flare is shown. Figure 7 A flow chart of operations associated with processing an image using a machine learning model to remove a representation of lens flare from the image is shown. Figure 6 and / or Figure 7 The operations may be performed by computing device 100, computing system 200, system 300, and / or other types of computing devices. Figure 6 and / or Figure 7 The operations may be performed by a computing device equipped with one or more camera devices, and / or by a computing device without any camera device. Figure 6 and / or Figure 7 The embodiments of the present invention can be simplified by removing any one or more features shown therein. In addition, these embodiments can be combined with the features, aspects and / or implementations of any previous drawings, or described in other ways herein.

[0110] Go to Figure 6 , block 600 may include obtaining (i) a plurality of reference images and (ii) a plurality of lens flare images.

[0111] Block 602 may include generating a plurality of training images by combining each respective reference image in the plurality of reference images with a respective lens flare image in the plurality of lens flare images. For example, a value of one or more pixels in the respective reference image may be added to or replaced with a value of one or more pixels in the respective lens flare image that represent lens flare (rather than background).

[0112] Block 604 may include determining a modified image for each respective training image in the plurality of training images by processing the respective training images by the machine learning model. The modified image may include the respective training image from which at least a portion of the respective lens flare representation is removed. During training of the machine learning model, the degree and / or accuracy with which the machine learning model determines the respective lens flare representation in the training images may improve. Accordingly, the degree and / or accuracy with which the respective lens flare representation is removed from the respective training images may also improve.

[0113] Block 606 may include determining a loss value for each respective training image in the plurality of training images based on a loss function configured to compare the modified image to a respective reference image used to generate the respective training image.

[0114] Block 608 may include adjusting one or more parameters of the machine learning model based on (i) the loss value determined for each corresponding training image and (ii) the loss function. For example, one or more parameters may be adjusted based on a gradient value of the loss function at or near the loss value.

[0115] In some embodiments, the plurality of lens flare images may include one or more simulated lens flare images generated by computationally simulating an optical system of a camera device to generate one or more representations of a first lens flare pattern. For example, optical components of a camera device, such as a lens and an image sensor, and a light source for emitting light that the camera device is configured to capture may be simulated.

[0116] In some embodiments, at least a portion of the representation of one or more first lens flare patterns may represent scattering of light caused by one or more defects present on the lens of the camera device. For example, the defects may include dust present on the lens and / or scratches or other physical flaws present on the lens. Capturing experimental data including a wide variety of defects may be difficult, time consuming, and / or impractical because it may be difficult to accurately control the physical defects present on the physical components of the camera device. However, simulations may allow for easier control of the size, positioning, and / or other properties of the defects, and may therefore facilitate the collection of lens flare images representing a wide variety of lens flare representations.

[0117] In some embodiments, computationally simulating the optical system of a camera device may include determining an aperture function of the camera device. The aperture function may represent one or more defects present on the lens of the camera device. For example, each point along a simulated aperture of a simulated version of the camera device may be associated with a corresponding light transparency, where the defect has a different transparency than a non-defective portion of the aperture. The linear phase shift may be determined based on the angular position of the camera device relative to the simulated light source. The degree of defocus may be determined based on the depth of the camera device relative to the simulated light source. The point spread function of the camera device may be determined based on the Fourier transform of the product of (i) the aperture function and (ii) an exponential function of the sum of the linear phase shift and the degree of defocus. Thus, computational simulation may be used to model the physical response of one or more camera devices (with corresponding defects) to various simulated light sources.

[0118] In some embodiments, computationally simulating the optical system of a camera device may further include generating a multi-channel point spread function representing an incoherent light source by sampling the point spread function at a plurality of different wavelengths. For example, the point spread function may be sampled from 380 nanometers to 740 nanometers in increments of 5 nanometers, resulting in a 73-channel point spread function. A particular simulated lens flare image of one or more simulated lens flare images may be generated by multiplying the multi-channel point spread function by a color matrix configured to transform the multi-channel point spread function into an image space. Thus, an image formed on an image plane of the simulated camera device may be transformed into pixel values ​​(e.g., red-green-blue pixel values) that collectively define a corresponding image (which includes a corresponding lens flare representation caused by a corresponding defect represented by an aperture function).

[0119] In some embodiments, additional simulated lens flare images of the one or more simulated lens flare images may be generated by adjusting one or more of: (i) an aperture function, (ii) a position of a camera device relative to a simulated light source, or (iii) a color matrix. Thus, the simulated lens flare images may represent a variety of possible physical conditions that result in a variety of different lens flare representations.

[0120] In some embodiments, the plurality of lens flare images may include one or more experimental lens flare images captured using a physical camera device. The one or more experimental lens flare images may include representations of one or more second lens flare patterns caused by one or more light sources emitting light to the camera device. The one or more experimental lens flare images may be captured using a camera device that is set at one or more different postures relative to the one or more light sources.

[0121] In some embodiments, the one or more experimental lens flare images may include multiple experimental lens flare images captured with a camera device set at multiple different postures relative to one or more light sources. Therefore, the experimental lens flare images can represent lens flare captured by an actual camera device under real-world conditions. Variations in the position and / or orientation of the camera relative to the light source can generate a variety of different lens flare representations, thereby providing diversity in the training images.

[0122] In some embodiments, the camera device may include a plurality of different camera devices. Each camera device in the plurality of different camera devices may differ from other camera devices in the plurality of different camera devices in at least one imaging / optical parameter. Thus, the experimental lens flare images may include lens flare representations resulting from various possible camera designs / implementations, and may therefore allow the machine learning model to learn to extrapolate to lens flare representations captured by camera devices for which the machine learning model has not been specifically trained.

[0123] In some embodiments, at least a portion of the representation of one or more second lens flare patterns may represent reflections of light from one or more surfaces of a lens of a camera device. Lens flare caused by reflections of light from the lens may be difficult to simulate due to the difficulty in obtaining / determining an accurate model of the lens used by the camera device (which may not be publicly available / available to the lens designer and / or manufacturer). In addition, the representations of lens flare caused by reflections captured by a particular camera may be similar on similar camera devices and / or lens models. Therefore, experimental lens flare images captured using a particular camera may be representative of lens flare associated with similar camera devices. Using two or more different camera models to capture lens flare images can further increase the information diversity of the training images.

[0124] In some embodiments, a restored lens flare image may be determined for each respective training image in a plurality of training images (i) by a machine learning model or (ii) based on a difference between the modified image and the respective training image. For example, the restored lens flare image may include a lens flare representation contained in the respective training image and may represent other aspects of the scene that are little or not represented by the respective training image. In some cases, the trained machine learning model may be configured to produce a modified image that substantially does not contain any lens flare representation, thereby obtaining a restored lens flare image that substantially only includes the lens flare representation and substantially does not represent any other portion of the scene. In other cases, the trained machine learning model may be configured to produce a restored lens flare image that substantially only includes the lens flare representation and substantially does not represent any other portion of the scene. A loss value may be further determined for each respective training image based on a second loss function that is configured to compare the restored lens flare image with the respective lens flare image used to generate the respective training image.

[0125] Thus, the loss value can represent how accurately the machine learning model determines / recognizes the lens flare representations in the training images, and how accurately these lens flare representations match the original lens flare representations used to generate the training images. Using a second loss function can improve the accuracy of the trained machine learning model by providing an additional measure of the quality of the output of the machine learning model relative to real data.

[0126] In some embodiments, determining the loss value may include determining the sum of (i) the product of the first weighted value and the output of the loss function and (ii) the product of the second weighted value and the output of the second loss function. For example, each of the first weighted value and the second weighted value may be equal (e.g., each may be equal to 0.5) so that each of the loss function and the second loss function contributes equally to the calculation of the loss value. Other weighted values ​​are also possible and can be used to improve various aspects of the machine learning model, such as accuracy and / or training rate, etc.

[0127] In some embodiments, the machine learning model may include a convolutional neural network. For example, the machine learning model may be arranged according to a U-Net architecture or a VGG-19 architecture, as well as other possible architectures.

[0128] In some embodiments, the plurality of reference images may include a plurality of monoscopic images. Thus, the machine learning model may be trained to determine the lens flare representation without relying on clues and / or information present in, for example, stereo images and / or multiple sequentially captured images, etc.

[0129] In some embodiments, at least a subset of the plurality of reference images may include a non-glare image that does not contain a lens flare representation. Thus, when such a reference image is combined with a lens flare image, the lens flare representation in the resulting image may be accurately known because the lens flare image is the source of such lens flare representation and the reference image does not contain other lens flare representations. The reference images may represent a variety of different scenes / environments, such that the machine learning model may be trained to generalize to a variety of possible inputs.

[0130] In some embodiments, the plurality of lens flare images may include respective lens flare representations shown relative to a monochromatic background. The monochromatic background may be selected to allow the lens flare representations to be easily separated from the background. Thus, for example, the monochromatic background may be black to allow for a high contrast with the lens flare representations. Furthermore, the plurality of lens flare images may be captured in an optically isolated environment such that substantially only the light source being photographed causes the lens flare. When the lens flare images are combined with the respective baseline images, the lens flare representations may be added to the baseline image and the monochromatic background may be discarded.

[0131] In some embodiments, an input image containing a specific lens flare representation may be obtained. After adjusting one or more parameters of the machine learning model, a de-glare image may be generated based on the processing of the input image by the machine learning model. The de-glare image may be output (e.g., stored and / or displayed). The de-glare image may include the input image from which at least a portion of the specific lens flare representation is removed. In some cases (e.g., when the machine learning model is well trained), the processing of the machine learning model may cause substantially all of the specific lens flare representation to be removed.

[0132] In some embodiments, a specific restored lens flare image may be determined (i) by a machine learning model or (ii) based on a difference between a de-glare image (which may be generated by a machine learning model) and an input image. The specific restored lens flare image may constitute a portion of a specific lens flare representation that has been removed from the input image. A first modified version of the input image may be generated by adjusting a visual appearance of a specific lens flare representation in the input image based on the specific restored lens flare image. The first modified version of the input image may be output (e.g., stored and / or displayed).

[0133] In some embodiments, adjusting the visual appearance of a particular lens flare representation in the input image may include one or more of: (i) increasing the intensity of the visual appearance, (ii) decreasing the intensity of the visual appearance, or (iii) adjusting the color of the visual appearance. Thus, a particular restored lens flare image may be used to apply various visual effects to a particular lens flare representation in the input image.

[0134] In some embodiments, a downsampled version of an input image may be generated by downsampling the input image from a first resolution to a second resolution. A deglare image may be generated based on a downsampled version of the input image. The deglare image may have a second resolution. A specific restored lens flare image may be determined (i) by a machine learning model or (ii) by subtracting the deglare image from a downsampled version of the input image. A specific restored lens flare image may have a second resolution. An upsampled version of a specific restored lens flare image may be generated by upsampling the specific restored lens flare image from the second resolution to the first resolution. A first modified version of the input image may be generated by adjusting the visual appearance of a specific lens flare representation in the input image based on an upsampled version of the specific restored lens flare image. Therefore, computationally intensive operations may be performed at a second resolution, and the resulting visual effects may be applied at a first resolution, thereby allowing benefits to be achieved while reducing computational costs.

[0135] In some embodiments, a light source mask may be determined based on an input image. The light source mask may be configured to mask one or more light sources that are (i) represented in the input image and (ii) cause a particular lens flare representation. A second modified version of the input image may be generated by combining (i) a first product of a pixel-by-pixel multiplication between the input image and the light source mask and (ii) a second product of a pixel-by-pixel multiplication between the de-glare image and an inverse of the light source mask. The second modified version of the input image may be output (e.g., stored and / or displayed).

[0136] In some cases, the machine learning model can be configured to determine / identify the light source representation as well as the lens flare representation, thereby removing the light source representation. Thus, the light source mask can be used to reintroduce the light source representation into the de-glare image, thereby producing a more realistic image.

[0137] In some embodiments, the light source mask may include a plurality of pixels and may have the same resolution as the input image. Determining the light source mask may include identifying one or more pixels within the input image that are associated with corresponding brightness values ​​that exceed a threshold brightness, and determining the light source mask by (i) assigning a first value to one or more pixels of the plurality of pixels that spatially correspond to corresponding locations of the one or more pixels and (ii) assigning a second value to one or more other pixels of the plurality of pixels. For example, pixels corresponding to the light source may each be set to a value of 1 or 255, while other pixels not corresponding to the light source may each be set to a value of zero.

[0138] Go to Figure 7 , block 700 may include obtaining an input image containing a particular lens flare representation.

[0139] Block 702 may include generating a deglare image by processing an input image by a machine learning model, the deglare image comprising the input image with at least a portion of a particular lens flare representation removed. The machine learning model may have been trained to determine / recognize lens flare representations in an image.

[0140] Block 704 may include outputting the de-glare image.

[0141] In some embodiments, a specific restored lens flare image may be determined (i) by a machine learning model or (ii) based on a difference between a de-glare image and an input image. A first modified version of the input image may be generated by adjusting a visual appearance of a specific lens flare representation in the input image based on the specific restored lens flare image. The first modified version of the input image may be output (e.g., stored and / or displayed).

[0142] In some embodiments, adjusting the visual appearance of a particular lens flare representation in the input image may include one or more of: (i) increasing the intensity of the visual appearance, (ii) decreasing the intensity of the visual appearance, or (iii) adjusting the color of the visual appearance.

[0143] In some embodiments, a downsampled version of the input image may be generated by downsampling the input image from a first resolution to a second resolution. The deglare image may be generated based on the downsampled version of the input image. The deglare image may have a second resolution. A specific restored lens flare image may be determined (i) by a machine learning model or (ii) by subtracting the deglare image from the downsampled version of the input image. The specific restored lens flare image may have a second resolution. An upsampled version of the specific restored lens flare image may be generated by upsampling the specific restored lens flare image from the second resolution to the first resolution. A first modified version of the input image may be generated by adjusting the visual appearance of a specific lens flare representation in the input image based on the upsampled version of the specific restored lens flare image.

[0144] In some embodiments, a light source mask may be determined based on an input image. The light source mask may be configured to mask one or more light sources that are (i) represented in the input image and (ii) cause a particular lens flare representation. A second modified version of the input image may be generated by combining (i) a first product of a pixel-by-pixel multiplication between the input image and the light source mask and (ii) a second product of a pixel-by-pixel multiplication between the de-glare image and an inverse of the light source mask. The second modified version of the input image may be output (e.g., stored and / or displayed).

[0145] In some embodiments, the light source mask may include a plurality of pixels and may have the same resolution as the input image. Determining the light source mask may include identifying one or more pixels within the input image that are associated with corresponding brightness values ​​that exceed a threshold brightness, and determining the light source mask by (i) assigning a first value to one or more pixels of the plurality of pixels that spatially correspond to the corresponding positions of the one or more pixels and (ii) assigning a second value to one or more other pixels of the plurality of pixels.

[0146] In some embodiments, a machine learning model may be trained based on a plurality of training images generated by combining each respective reference image of a plurality of reference images with a respective lens flare image of a plurality of lens flare images.

[0147] In some embodiments, a machine learning model may have been trained to determine / identify a lens flare representation in an image by obtaining (i) a plurality of reference images and (ii) a plurality of lens flare images. A plurality of training images may be generated by combining each respective reference image in the plurality of reference images with a respective lens flare image in the plurality of lens flare images. By processing the respective training images by the machine learning model, a modified image may be determined for each respective training image in the plurality of training images. The modified image may include a respective training image in which at least a portion of the respective lens flare representation is removed. A loss value may be determined for each respective training image in the plurality of training images based on a loss function, the loss function being configured to compare the modified image with a respective reference image used to generate the respective training image. One or more parameters of the machine learning model may be adjusted based on (i) the loss value determined for each respective training image and (ii) the loss function.

[0148] In some embodiments, a restored lens flare image may be determined for each respective training image in a plurality of training images (i) by a machine learning model or (ii) based on a difference between the modified image and the respective training image. A loss value may be further determined for each respective training image in the plurality of training images based on a second loss function configured to compare the restored lens flare image with the respective lens flare image used to generate the respective training image. One or more parameters may be further adjusted based on the second loss function.

[0149] In some embodiments, determining the loss value may include determining the sum of (i) a product of the first weighted value and an output of the loss function and (ii) a product of the second weighted value and an output of the second loss function.

[0150] In some embodiments, the plurality of lens flare images may include one or more simulated lens flare images generated by computationally simulating an optical system of a camera device to generate representations of the one or more first lens flare patterns.

[0151] In some embodiments, at least a portion of the representation of the one or more first lens flare patterns may represent scattering of light caused by one or more imperfections present on the lens of the camera device.

[0152] In some embodiments, computationally simulating the optical system of a camera device may include determining an aperture function of the camera device. The aperture function may represent one or more defects present on the lens of the camera device. The linear phase shift may be determined based on the angular position of the camera device relative to the simulated light source. The degree of defocus may be determined based on the depth of the camera device relative to the simulated light source. The point spread function of the camera device may be determined based on the Fourier transform of the product of (i) the aperture function and (ii) the exponential function of the sum of the linear phase shift and the degree of defocus. A multi-channel point spread function representing an incoherent light source may be generated by sampling the point spread function at a plurality of different wavelengths. A specific simulated lens flare image of one or more simulated lens flare images may be generated by multiplying the multi-channel point spread function by a color matrix configured to transform the multi-channel point spread function to an image space. Additional simulated lens flare images of one or more simulated lens flare images may be generated by adjusting one or more of the following: (i) the aperture function, (ii) the position of the camera device relative to the simulated light source, or (iii) the color matrix.

[0153] In some embodiments, the plurality of lens flare images may include one or more experimental lens flare images captured using a camera device. The one or more experimental lens flare images may include representations of one or more second lens flare patterns caused by one or more light sources emitting light to the camera device. The one or more experimental lens flare images may be captured using a camera device that is set at one or more different attitudes relative to the one or more light sources.

[0154] In some embodiments, the one or more experimental lens flare images may include a plurality of experimental lens flare images captured with a camera device disposed at a plurality of different poses relative to the one or more light sources.

[0155] In some embodiments, at least a portion of the representation of the one or more second lens flare patterns may represent reflections of light from one or more surfaces of the lens of the camera device.

[0156] In some embodiments, the machine learning model may include a convolutional neural network.

[0157] In some embodiments, the plurality of reference images may include a plurality of monoscopic images.

[0158] In some embodiments, at least a subset of the plurality of reference images may include glare-free images that contain no representation of lens flare.

[0159] In some embodiments, the plurality of lens flare images may include respective lens flare representations shown against a monochromatic background.

[0160] In some embodiments, outputting the de-glare image may include storing the de-glare image and / or displaying the de-glare image.

[0161] VII. Conclusion

[0162] The present disclosure is not limited to the specific embodiments described in this application, which are intended to be illustrations of various aspects. Various modifications and variations can be made without departing from the scope of the present disclosure, which will be clear to those skilled in the art. In addition to those methods and devices described herein, according to the foregoing description, functionally equivalent methods and devices within the scope of the present disclosure will be clear to those skilled in the art. Such modifications and variations are intended to fall within the scope of the appended claims.

[0163] The above detailed description describes various features and operations of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the accompanying drawings, similar symbols generally identify similar components unless the context otherwise indicates. The example embodiments described herein and in the accompanying drawings are not meant to be limiting. Other embodiments may be utilized and other changes may be made without departing from the scope of the subject matter presented herein. It will be readily understood that aspects of the present disclosure, as generally described herein and shown in the accompanying drawings, may be arranged, replaced, combined, separated, and designed in a variety of different configurations.

[0164] With respect to any or all message flow diagrams, scenarios, and process diagrams in the figures and as discussed herein, each step, box, and / or communication may represent information processing and / or information transmission according to the example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, depending on the functionality involved, the operations described as steps, boxes, transmissions, communications, requests, responses, and / or messages may not be performed in the order shown or discussed, including substantially simultaneously or in reverse order. In addition, more or fewer boxes and / or operations may be used with any message flow diagrams, scenarios, and process diagrams discussed herein. Figure 1 The message flow diagrams, scenarios and flowcharts can be used together, and these message flow diagrams, scenarios and flowcharts can be combined with each other in part or in whole.

[0165] The steps or boxes representing information processing may correspond to circuits that may be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, the boxes representing information processing may correspond to modules, program code segments, or program code portions (including associated data). The program code may include one or more instructions that may be executed by a processor to implement specific logical operations or actions in a method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including a random access memory (RAM), a disk drive, a solid-state drive, or other storage medium.

[0166] Computer readable media may also include non-transitory computer readable media, such as computer readable media that store data for a short time, such as register memory, processor cache, and RAM. Computer readable media may also include non-transitory computer readable media that store program code and / or data for a long time. Therefore, computer readable media may include auxiliary or permanent long-term storage, such as read-only memory (ROM), optical or magnetic disk, solid-state drive, compact disk read-only memory (CD-ROM). Computer readable media may also be any other volatile or non-volatile storage system. Computer readable media may be considered to be, for example, a computer readable storage medium or a tangible storage device.

[0167] In addition, the steps or boxes representing one or more information transfers may correspond to information transfers between software and / or hardware modules in the same physical device. However, other information transfers may be performed between software modules and / or hardware modules in different physical devices.

[0168] The specific arrangement shown in the figures should not be considered as limiting. It should be understood that other embodiments may include more or less of each element shown in a given figure. In addition, some of the elements shown may be combined or omitted. In addition, the example embodiments may include elements not shown in the drawings.

[0169] Although various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and not limitation, with the true scope being indicated by the appended claims.

Claims

1. A computer-implemented method comprising: obtaining (i) a plurality of reference images and (ii) a plurality of lens flare images; generating a plurality of training images by combining each corresponding one of the plurality of reference images with a corresponding one of the plurality of lens flare images; for each respective training image in the plurality of training images, determining a modified image by processing the respective training image by the machine learning model, wherein the modified image comprises the respective training image with at least a portion of the respective representation of the lens flare removed; For each corresponding training image in the plurality of training images: determining a restored lens flare image (i) by the machine learning model or (ii) based on a difference between the modified image and the corresponding training image; determining a loss value based on (i) a first loss function configured to compare the modified image with a corresponding reference image used to generate the corresponding training image and (ii) a second loss function configured to compare the restored lens flare image with a corresponding lens flare image used to generate the corresponding training image; and One or more parameters of the machine learning model are adjusted based on (i) the loss value determined for each corresponding training image, (ii) the first loss function, and (iii) the second loss function.

2. The computer-implemented method of claim 1 , further comprising: obtaining an input image comprising a specific representation of lens flare; processing the input image by a machine learning model to generate a deglare image, the deglare image comprising the input image with at least a portion of a particular representation of lens flare removed; as well as The de-glare image is output.

3. The computer-implemented method of claim 1 , wherein: Determining the modified image by processing the corresponding training image by the machine learning model includes: determining the modified image by the machine learning model, wherein the restored lens flare image is based on a difference between the modified image and a corresponding training image when the modified image is determined by the machine learning model; or A modified image is determined based on a difference between the restored lens flare image and a corresponding training image, wherein when the modified image is determined based on the difference between the restored lens flare image and the corresponding training image, the restored lens flare image is determined by a machine learning model.

4. The computer-implemented method of claim 1 , wherein: Determining the loss value includes: A sum of (i) a product of a first weight value and an output of a first loss function and (ii) a product of a second weight value and an output of a second loss function is determined.

5. The computer-implemented method of claim 1 , wherein: The plurality of lens flare images include one or more simulated lens flare images generated by computationally simulating an optical system of a camera device to generate representations of one or more first lens flare patterns.

6. The computer-implemented method of claim 5, wherein: At least a portion of the representation of the one or more first lens flare patterns represents scattering of light by one or more defects present on a lens of the camera device.

7. The computer-implemented method of claim 5, wherein: Computationally simulating the optical system of a camera device involves: Determining an aperture function of the camera device, wherein the aperture function indicates one or more defects present in a lens of the camera device; determining a linear phase shift based on an angular position of the camera device relative to the simulated light source; determining a degree of defocus based on a depth of the camera device relative to the simulated light source; determining a point spread function of the camera device based on a Fourier transform of the product of (i) an aperture function and (ii) an exponential function of the sum of a linear phase shift and a degree of defocus; generating a multi-channel point spread function representing an incoherent light source by sampling the point spread function at a plurality of different wavelengths; and A specific simulated lens flare image among one or more simulated lens flare images is generated by multiplying a multi-channel point spread function by a color matrix configured to transform the multi-channel point spread function into an image space, wherein an additional simulated lens flare image of the one or more simulated lens flare images is generated by adjusting one or more of the following: (i) an aperture function, (ii) a position of a camera device relative to a simulated light source, or (iii) a color matrix.

8. The computer-implemented method of claim 1 , wherein: The multiple lens flare images include one or more experimental lens flare images captured using a camera device, wherein the one or more experimental lens flare images contain representations of one or more second lens flare patterns caused by one or more light sources emitting light toward the camera device, and wherein the one or more experimental lens flare images are captured using a camera device set at one or more different postures relative to the one or more light sources.

9. The computer-implemented method of claim 8, wherein: At least a portion of the representation of the one or more second lens flare patterns represents reflections of light from one or more surfaces of a lens of the camera device.

10. The computer-implemented method of claim 1, wherein: The plurality of reference images include a plurality of monoscopic images.

11. The computer-implemented method of claim 1 , wherein: At least a subset of the plurality of reference images includes glare-free images that contain no representation of lens flare.

12. The computer-implemented method of claim 1, wherein: The plurality of lens flare images include respective lens flare representations shown relative to a monochromatic background.

13. The computer-implemented method of claim 2, further comprising: (i) determining a specific restored lens flare image by a machine learning model or (ii) based on a difference between the de-glare image and the input image; generating a first modified version of the input image by adjusting the visual appearance of a particular representation of lens flare in the input image based on a particular restored lens flare image; and Outputs the first modified version of the input image.

14. The computer-implemented method of claim 13, wherein: Adjusting the visual appearance of the particular representation of lens flare in the input image includes one or more of: (i) increasing the intensity of the visual appearance, (ii) decreasing the intensity of the visual appearance, or (iii) adjusting a color of the visual appearance.

15. The computer-implemented method of claim 13, further comprising: generating a downsampled version of an input image by downsampling the input image from a first resolution to a second resolution, wherein the de-glare image is generated based on the downsampled version of the input image, wherein the de-glare image has the second resolution, wherein the particular restored lens flare image is (i) determined by a machine learning model or (ii) determined by subtracting the de-glare image from the downsampled version of the input image, and wherein the particular restored lens flare image has the second resolution; and An upsampled version of the specific restored lens flare image is generated by upsampling the specific restored lens flare image from a second resolution to a first resolution, wherein the first modified version of the input image is generated by adjusting the visual appearance of a specific representation of lens flare in the input image based on the upsampled version of the specific restored lens flare image.

16. The computer-implemented method of claim 2, further comprising: determining, based on the input image, a light source mask configured to mask one or more light sources that are (i) represented in the input image and (ii) cause a particular representation of lens flare; generating a second modified version of the input image by combining (i) a first product of pixel-wise multiplications between the input image and the illuminant mask and (ii) a second product of pixel-wise multiplications between the de-glare image and an inverse of the illuminant mask; and Outputs a second modified version of the input image.

17. The computer-implemented method of claim 16, wherein: The light source mask comprises a plurality of pixels and has the same resolution as the input image, and wherein determining the light source mask comprises: identifying, within the input image, one or more pixels associated with respective brightness values ​​exceeding a threshold brightness; and A light source mask is determined by (i) assigning first values ​​to one or more pixels of a plurality of pixels that spatially correspond to respective locations of the one or more pixels, and (ii) assigning second values ​​to one or more other pixels of the plurality of pixels.

18. The computer-implemented method of claim 1, wherein: The machine learning model includes a convolutional neural network.

19. A computer-implemented system comprising: processor; as well as A non-transitory computer-readable medium having instructions stored thereon, which, when executed by the processor, causes the processor to perform operations, the operations comprising: obtaining (i) a plurality of reference images and (ii) a plurality of lens flare images; generating a plurality of training images by combining each corresponding one of the plurality of reference images with a corresponding one of the plurality of lens flare images; for each respective training image in the plurality of training images, determining a modified image by processing the respective training image by the machine learning model, wherein the modified image comprises the respective training image with at least a portion of the respective representation of the lens flare removed; For each corresponding training image in the plurality of training images: determining a restored lens flare image (i) by the machine learning model or (ii) based on a difference between the modified image and the corresponding training image; determining a loss value based on (i) a first loss function configured to compare the modified image with a corresponding reference image used to generate the corresponding training image and (ii) a second loss function configured to compare the restored lens flare image with a corresponding lens flare image used to generate the corresponding training image; and One or more parameters of the machine learning model are adjusted based on (i) the loss value determined for each corresponding training image, (ii) the first loss function, and (iii) the second loss function.

20. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by a computing device, cause the computing device to perform operations, the operations comprising: obtaining (i) a plurality of reference images and (ii) a plurality of lens flare images; generating a plurality of training images by combining each corresponding one of the plurality of reference images with a corresponding one of the plurality of lens flare images; for each respective training image in the plurality of training images, determining a modified image by processing the respective training image by the machine learning model, wherein the modified image comprises the respective training image with at least a portion of the respective representation of the lens flare removed; For each corresponding training image in the plurality of training images: determining a restored lens flare image (i) by the machine learning model or (ii) based on a difference between the modified image and the corresponding training image; determining a loss value based on (i) a first loss function configured to compare the modified image with a corresponding reference image used to generate the corresponding training image and (ii) a second loss function configured to compare the restored lens flare image with a corresponding lens flare image used to generate the corresponding training image; and One or more parameters of the machine learning model are adjusted based on (i) the loss value determined for each corresponding training image, (ii) the first loss function, and (iii) the second loss function.

Images