Systems and methods for underwater imagery enhancement

Abstract

A computer-implemented method for training a generative adversarial network (GAN) for enhancing underwater images. An adversarial loss is computed for updating a discriminator model and a combined loss is calculated for updating a generator model. The combined loss is calculated based on loss components including the adversarial loss and at least one further loss component. Additionally disclosed herein is a generator network for processing underwater images that includes a novel encoder-decoder model architecture. Unlocking insights from Geo-Data, the present invention further relates to improvements in sustainability and environmental developments: together we create a safe and liveable world.

Description

SYSTEMS AND METHODS FOR UNDERWATER IMAGERY ENHANCEMENTFIELD OF THE INVENTION

[0001] The present disclosure generally relates to the field of underwater image and video (imagery) processing. More particularly, the invention relates to methods and systems for enhancing underwater imagery. Unlocking insights from Geo-Data, the present invention further relates to improvements in sustainability and environmental developments: together we create a safe and liveable world.BACKGROUND OF THE INVENTION

[0002] Underwater image enhancement has become an increasingly important area of research in computer vision, driven by the expanding interest in exploring the underwater world. Remotely operated vehicles (ROVs) and autonomous underwater vehicles (AUVs) are extensively utilized in underwater research, where clear visual conditions are essential for the visual systems onboard these vehicles. However, achieving clarity in underwater environments is challenging due to factors such as color degradation, where different light wavelengths are absorbed unevenly (with red wavelengths diminishing most at greater depths), water turbidity causing haziness from scattered light by numerous small particles, and poor illumination at deeper levels due to light absorption. These conditions can make the maneuvering of ROVs and AUVs under water risky in some situations. Image and video enhancement methods for underwater conditions aim to address these problems by improving the quality of the footage, leading to enhanced visibility.

[0003] Deep learning and artificial intelligence (Al) systems have been proposed for enhancing visual data from underwater imaging. Al-based techniques have emerged as powerful tools to address the challenges posed by the underwater environment, including low light conditions, varying water clarity, and color distortion due to light absorption and scattering. These technologies may employ complex neural network architectures to learn from large datasets ofunderwater images, enabling them to automatically apply corrections that enhance visual clarity, restore natural colors, and improve overall image quality without manual intervention.

[0004] Some deep learning algorithms for underwater image enhancement may analyze input imagery to identify and mitigate issues such as haze, blur, and unnatural color casts. By adjusting parameters like contrast, brightness, and hue based on learned models, these systems can produce images that more closely resemble how the scene would appear in optimal lighting conditions above water. The training process may involve exposing the neural network to a wide range of underwater images and their enhanced counterparts, allowing the model to learn the transformations needed to correct various underwater imaging artifacts.

[0005] Initial attempts at adopting deep learning methods to enhance underwater images grappled with a lack of authentic training data, prompting the use of Generative Adversarial Networks (GANs) to synthetically create underwater images. The synthesized datasets allowed for the training of neural networks that could estimate and correct color distortions caused by light attenuation and scattering in an underwater environment. With a workable dataset of paired raw and high-quality enhanced images, CNN networks could be trained to deliver a model for underwater image enhancement.

[0006] Despite the progress of Al image enhancement, the computational intensity of deep learning models can be a problem in some applications. Models with intricate designs, composed of multiple modules for distinct tasks such as color correction and dehazing, demand substantial processing power, limiting their deployability in real-time applications and on devices with constrained computational resources. Moreover, the increasing complexity of network architectures, while improving output quality, escalate the computational demands

[0007] Further, while deep learning models have progressively become more adept at enhancing underwater imagery, the methodologies for training these networks have their own set of challenges. Traditional loss functions used in these models often struggle to adequately capture the diverse range of underwater imaging conditions. Consequently, models trained with single-objective loss functions may excel in one aspect of image quality, such as color fidelity or sharpness, but fail to holistically enhance the image in a manner that balances all attributes. This imbalance can result in outputs that, while improved, do not consistently reflect the true undersea environment across various scenes and conditions.

[0008] Moreover, the training of these networks using such loss functions can lead to convergences on suboptimal solutions, where the network's output may appear artificially processed rather than naturally enhanced. Models that rely on simplistic loss functions are prone to artifacts and may not generalize well to the unpredictability of real-world underwaterscenarios, which are characterized by non-uniform lighting, varying visibility conditions, and diverse aquatic flora and fauna.

[0009] There is thus a need for systems and methods to further enhance underwater image quality using a suitably trained Al model, while maintaining practicality of deployment in diverse and resource-limited applications. There is also a need for systems and methods for enhancing underwater imagery using Al that provides high quality underwater imagery enhancement whilst also meeting computational constraints of resource limited processing environments.BRIEF SUMMARY OF THE INVENTION

[0010] The above mentioned and other features and advantages of the disclosure will be best understood from the following description referring to the attached drawings. In the drawings, like reference numerals denote identical parts or parts performing an identical or comparable function or operation.

[0011] In a first aspect, a generator network is provided for processing underwater images. The generator network includes an encoder-decoder model, comprising: an encoder including a series of downsampling units configured to process input underwater images, wherein each downsampling unit comprises a downsampling block that includes a combination of pointwise and depthwise convolutional layers for reducing spatial dimensions of the input underwater images, thereby providing encoded feature maps; and a decoder including a series of upsampling units configured to generate enhanced images from the encoded feature maps provided by the encoder, wherein each upsampling unit comprises an upsampling block that includes a combination of pointwise and depthwise convolutional layers for increasing spatial dimensions and enhancing features representation of the enhanced images; wherein the generator network is adapted to receive the input underwater images and output corresponding enhanced images.

[0012] In embodiments, the encoder includes an instance normalization layer disposed prior to the series of downsampling units, the instance normalization layer being operative to normalize feature maps on a per-image basis as derived from the input underwater images.

[0013] In embodiments, each downsampling unit includes a basic block comprising a channel shuffle layer disposed upstream of the combination of pointwise and depthwise convolutional layers.

[0014] In embodiments, the downsampling block does not include a channel shuffle layer.

[0015] In embodiments, the upsampling block includes a pixel shuffle layer configured to increase a resolution of feature maps by rearranging tensor elements from a lower resolution to a higher resolution.

[0016] In embodiments, skip connections are included connecting the downsampling units of the encoder with corresponding upsampling units of the decoder.

[0017] In a second aspect, a computer-implemented method for training a generative adversarial network (GAN) for enhancing underwater images is provided. The method is performed by at least one processor executing programming instructions stored on a non- transitory computer readable medium. The method comprises providing, to the at least one processor, a training dataset comprising pairs of raw underwater images and corresponding reference images; processing, by the at least one processor, the raw underwater images through a generator model of a generator to produce enhanced images; comparing, by the at least one processor using a discriminator model of a discriminator, the enhanced images and the corresponding reference images to generate a discriminator output representing a classification indicating a likelihood of the enhanced images being synthetically generated by the generator model; computing, by the at least one processor, an adversarial loss based on the discriminator model performance in correctly classifying the enhanced images according to the discriminator output; calculating, by the at least one processor, a combined loss for the generator model, the combined loss including a weighted sum of a plurality of loss components, each loss component corresponding to a different aspect of image quality enhancement. The plurality of loss components includes the adversarial loss and at least one of: a first loss component based on a mean absolute difference between corresponding pixels of the enhanced images and the corresponding reference images; a second loss component based on a deep learning model providing perceptual similarity assessment of texture and structural characteristics between the enhanced images and the corresponding reference images and which has been trained to include image datasets annotated based on human perception; a third loss component based on a luminance, contrast, and structure comparison of the enhanced images and the corresponding reference images; and a fourth loss component based on comparing reference feature maps with corresponding enhanced image feature maps obtained from processing the enhanced images and the corresponding reference images through an image classification convolutional neural network, wherein the reference feature maps and the enhanced image features maps are extracted from an intermediate layer of the image classification convolutional neural network. The method includes adjusting, by the at least one processor, the model of the generator basedon the combined loss; and adjusting, by the at least one processor, the model of the discriminator based on the adversarial loss.

[0018] In embodiments, the generator comprises an encoder-decoder model.

[0019] In embodiments, the generator comprises an encoder-decoder model as described above. In one embodiment, the encoder-decoder model comprises attention-modulated skip connections.

[0020] In embodiments, a plurality of bottleneck residual layers between encoder and decoder parts of the encoder-decoder model.

[0021] In embodiments, the discriminator comprises local and global discriminator models including the discriminator model, the local and global discriminator models operating at different image patch sizes.

[0022] In embodiments, the discriminator comprises the local and global discriminator models and an intermediate discriminator model, each operating at different image patch sizes.

[0023] In embodiments, the adversarial loss is computed according to a negative log likelihood approach.

[0024] In embodiments, the adversarial loss is computed according to a least-squares objective.

[0025] In embodiments, the method includes iterating the computing-implemented method, outputting a trained generator and deploying the trained generator on a system for enhancing underwater imagery to enhance raw image frames.

[0026] In another aspect, a system for enhancing underwater imagery. The system includes at least one processor; data storage comprising a trained generator as described above. The at least one processor executes programming instructions configured to: receive raw image frames of underwater imagery; process the raw image frames using the trained generator to provide enhanced image frames; and use the enhanced image frames for at least one of: generating a visual display; controlling an underwater vehicle; and performing visual recognition.

[0027] In another aspect, computer readable media is provided comprising the trained generator as described above.BRIEF DESCRIPTION OF THE DRAWINGSIn order to describe the manner in which the above-recited and other advantages and features of the disclosure can be obtained, a more particular description of the principles brieflydescribed above will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. Understanding that these drawings depict only exemplary embodiments of the disclosure and are therefore not to be considered to be limiting of its scope, the principles herein are described and explained with additional specificity and detail through the use of the accompanying drawings in which:

[0028] FIG. 1 is a schematic diagram of a system for enhanced consistency in underwater imagery, according to embodiments of the present disclosure;

[0029] FIG. 2 is a schematic diagram of the system of FIG. 1 further detailing data flow within underwater imagery enhancement software, according to embodiments of the present disclosure;

[0030] FIG. 3 is a flowchart of a method for enhanced consistency in underwater imagery, according to embodiments of the present disclosure;

[0031] FIG. 4 illustrates a Generative Adversarial Network (GAN) training system according to embodiments of the present disclosure;

[0032] FIG. 5 illustrates a small generator network as an exemplary embodiment of an Al based image enhancement module as shown in FIG.2;

[0033] FIG. 6 illustrates a basic block of the small generator network of FIG. 5 according to embodiments of the present disclosure;

[0034] FIG. 7 illustrates a downsampling block of the small generator network of FIG. 5 according to embodiments of the present disclosure;

[0035] FIG. 8 illustrates an upampling block of the small generator network of FIG. 5 according to embodiments of the present disclosure; and

[0036] FIG. 9 illustrates a GAN training method according to embodiments of the present disclosure.DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

[0037] Embodiments contemplated by the present disclosure will now be described in more detail with reference to the accompanying drawings. The disclosed subject matter should not be construed as limited to only the embodiments set forth herein. Rather, the illustrated embodiments are provided by way of example to covey the scope of the subject matter to those skilled in the art.

[0038] FIG. 1 is a schematic diagram of a system for enhanced consistency in underwater imagery 10 including an underwater camera 12, a processing unit 14, a user interface 32, a datatransmission system 34, a vehicle control application 28, a display device 26, data storage 30 and a visual recognition application 36. The processing unit 14 includes a processor 16, volatile memory 18 and underwater imagery enhancement software 20. The underwater imagery enhancement software includes a Dynamic Underwater Color Transfer (DUCT) algorithm 22 and an Al based image enhancement module 24. Whilst the processing unit 14 is shown to be functionally separate from the underwater camera 12, they may be included in a common housing, e.g. implemented in an edge computing manner. Alternatively, the processing unit 14 may be separately housed or cloud based.

[0039] The system for enhanced consistency in underwater imagery 10 is designed to improve the quality and consistency of underwater imagery, utilizing advanced processing techniques and Al to enhance visual content captured by the underwater camera 12.

[0040] The underwater camera 12 serves as an image capture device, designed to operate in subaquatic environments. The underwater camera 12 captures raw image frames that are often subject to underwater-specific distortions like color casting, reduced contrast, and blurriness due to light absorption and scattering by water particles. The raw images captured by the underwater camera 12 are subsequently processed and enhanced for clarity, color accuracy, and overall visual quality by the processing unit 14. In one example, the underwater camera 12 is mounted on a Remotely Operated Vehicle (ROV). An ROV may be tethered to a control unit on the surface and employ the underwater camera 12 for real-time visual feedback. An ROV may be utilized for underwater exploration, infrastructure inspection, and scientific research. The underwater camera 12 may be equipped on an Autonomous Underwater Vehicle (AUV) that operate autonomously, following pre-programmed missions. The integration of underwater cameras on an AUV supports navigation, obstacle avoidance, and data collection. The underwater camera 12 may be attached to dive equipment or handheld by a diver for targeted image capture such as coral reef monitoring or underwater cinematography. The underwater camera 12 may be connected to fixed installations requiring continuous observation of a specific site, such as an artificial reef. The underwater camera 12 may be incorporated in unmanned surface vehicles (USVs) for applications involving both surface and underwater observation. USVs can launch camera systems (including the underwater camera 12) for underwater investigations while maintaining surface mobility, which may be useful in search and rescue operations or water quality assessments.

[0041] Regardless of the deployment scenario — whether mounted on ROVs, equipped on AUVs, handheld by divers, or utilized in fixed or wearable configurations — the provision of high-quality images is important. Underwater environments introduce unique distortions andimage artifacts, including color casting, reduced visibility due to particulate matter, and light attenuation, which can significantly degrade the quality of captured imagery. The processing unit 14, equipped with underwater imagery enhancement software 20, is specifically designed to address these challenges, utilizing advanced algorithms and Al techniques tailored to correct such underwater-specific distortions, ensuring the output of clear, color-accurate, and visually compelling images.

[0042] The output of the enhanced frames from the processing unit 14, achieved through the system's Dynamic Underwater Color Transfer (DUCT) algorithm and Al-based enhancement processes, can be applied in a variety of use cases. In operations involving ROVs and AUVs, enhanced real-time video feeds can significantly improve operator or autonomous decisionmaking processes. Enhanced frames serve as a superior input for visual recognition algorithms, facilitating automated identification of marine life, anomaly detection in underwater structures, and machine learning models trained on underwater data. In every scenario, the system's ability to provide enhanced underwater imagery is not just about visual improvement but about unlocking new possibilities and enhancing the efficacy of underwater operations and observations. The enhanced frames boost human perceptibility when displayed across a variety of output devices, such as high-definition monitors, virtual reality (VR) headsets, mobile devices, and immersive projection systems. The enhanced clarity and color accuracy of the images facilitate educational outreach, heritage preservation, and improve operational efficiency in public safety efforts (e.g. rescue missions) by providing greater visual precision.

[0043] The processing unit 14 includes the processor 16, volatile memory 18, and underwater imagery enhancement software 20. The processing unit 14 is configured to process, analyze, and enhance raw images captured by the underwater camera 12 using the DUCT algorithm 22 and the Al-based image enhancement module 24.

[0044] The processor 16 executes the software algorithms described herein and manages data flow within the system 10. The processor 16 can encompass various types of processors, such as Central Processing Units (CPUs) for general-purpose computing, Graphics Processing Units (GPUs) for intensive parallel processing tasks, and specialized processors like Tensor Processing Units (TPUs) and Field-Programmable Gate Arrays (FPGAs) designed to accelerate machine learning and image processing workflows. Examples of such processors where the underwater imagery enhancement software 20 has been deployed and tested for performance include low-power, compact processors like those found in Raspberry Pi 4 and Nvidia Jetson series, which are ideal for real-time applications and embedded systems, as well as high-end GPUs like the Nvidia RTX4090, known for their high computational throughput and efficiencyin handling intensive image processing tasks. This varied deployment highlights that the underwater imagery enhancement software 20 described herein is particularly processing efficient and facilitates a wide range of hardware platforms.

[0045] Volatile memory 18 temporarily stores data and instructions for the processor 16 during execution of the underwater imagery software. Types of volatile memory utilized within the system can include Random Access Memory (RAM), which provides the high-speed read / write capabilities essential for real-time data processing. During the execution of the DUCT algorithm 22, volatile memory 18 is tasked with storing a variety of data. Raw image frames received from the underwater camera 12 may be stored, awaiting processing by the Al-based image enhancement module 24. Enhanced image frames before and after application of color characteristics adjustments may be stored, enabling comparison and blending with an enhanced image template. Image quality scores may be stored for each enhanced frame, used in determining a selection of an enhanced image template and guiding dynamic adjustment of blending steps. Volatile memory 18 may store color statistics (mean and standard deviation across color channels) for both a current enhanced image frame and an enhanced image template for executing the color transfer function. Dynamically calculated blending parameters may be stored such as the number of blending steps (N) and the current blending factor ( >) for managing gradual application of color characteristics across consecutive frames.

[0046] When a generator model of the Al based image enhancement module 24 is deployed, it may be loaded from nonvolatile storage (data storage 30) into volatile memory 18. This includes loading weights, biases, and architecture configuration of neural networks included in the generator model. Volatile memory 18 may be used to store input data temporarily as it is being processed by the generator such as raw image frames to be enhanced, where they are accessed by the Al based image enhancement module 24 for processing. During an image enhancement process, the generator performs computations involving multiple layers within the neural network, each potentially adding, modifying, or refining features in the data. Volatile memory 18 stores the intermediate results of these computations between layers.

[0047] The underwater imagery enhancement software 20 comprises specialized algorithms (computer program instructions) designed to correct and enhance underwater images. The underwater image enhancement software 20 includes the DUCT algorithm 22, which adjusts the color characteristics of underwater imagery to ensure consistency across a series of images, addressing issues of color inconsistency and enhancing visual appeal. The Al-based image enhancement module 24 utilizes Al techniques, based on a Generative Adversarial Network (GAN). The generator of the GAN has been found to be particularly effective for the presentimage enhancement tasks when trained specifically for underwater imagery to correct distortions and enhance image quality.

[0048] The Al-based image enhancement module 24 employs artificial intelligence techniques to systematically correct common underwater image distortions. The neural network architecture in the form of a generator of a GAN, identifies and rectifies issues such as color distortion, loss of contrast, and blurriness inherent in underwater imagery. The Al-based image enhancement module 24 is trained on a dataset that encompasses a diverse array of underwater images, each paired with target images that represent the desired output quality. These target images are typically generated through expert-led enhancement processes or through advanced simulation techniques that mimic optimal underwater visibility conditions. This dual-image approach — comprising raw, distorted underwater scenes alongside their enhanced counterparts — enables the GAN to accurately learn the specific characteristics of underwater image degradation, such as color fading due to the water’s selective absorption of light wavelengths, blurriness from suspended particulate matter, and the loss of contrast caused by scattering The training process is designed to familiarize the GAN with the gamut of underwater conditions, from the murkiness of coastal waters to the clearer but deeper oceanic realms. By effectively modelling the transformation from degraded to idealized visibility, the Al based image enhancement module 24 is adept at autonomously correcting these distortions, enhancing details, and adjusting image properties to yield outputs that closely resemble the clarity, color balance, and detail definition found in the target images. This precise calibration against a well-curated dataset ensures that the Al-based image enhancement module 24 significantly improves the quality of underwater imagery. The GAN implementation of the Al based image enhancement module 24 and the training of the GAN is described below with respect to FIGS. 4 and 5. An example generator network according to one embodiment is described below with respect to FIGS. 5 to 8.

[0049] Following the initial enhancement by the Al based image enhancement module 24, the DUCT algorithm 22 further refines color characteristics of the enhanced image frames to ensure visual consistency and fidelity across a series of images. This algorithm addresses the challenge of color inconsistency across image frames by employing a color transfer function that aligns the color palette of each enhanced frame with that of a selected enhanced image template, chosen for its high image quality score. The color transfer function operates by matching mean color values and standard deviations for each color channel within the selected template to those of the enhanced image frames. In scenarios where the template is updated due to higher quality frames being processed, the algorithm adeptly applies the colorcharacteristics in a blended mode to maintain continuity; otherwise, the algorithm defaults to a direct mode of color transfer for efficiency. In the blending mode, the algorithm incrementally applies the color characteristics of the template to subsequent frames. This blending is governed by a dynamically determined number of blending steps and a blending factor, <|>, which progressively increases to control the degree of color characteristic application. The process ensures a gradual and controlled transition of colors.

[0050] The user interface 32 allows users to interact with the system, configure settings, and view processed images. It provides accessibility to system functionalities and facilitates user control over the image enhancement process.

[0051] Data transmission system 34 enables the transfer of data between the system and external devices or networks. The data transmission system 34 can be used to upload raw images for processing and download enhanced images for various applications. The data transmission system 34 may alternatively allow remote upload of locally produced enhanced images.

[0052] The vehicle control application 28 may be part of autonomous or remotely operated underwater vehicles, and use the enhanced imagery provided by the processing unit 14 for navigation, mission planning, and / or environment interaction. The vehicle control application 28 benefits from clearer, more consistent images for improved decision-making and operational efficiency.

[0053] The display device 26 acts as the interface through which users may view the enhanced underwater images or videos produced by the system 10. The display device 26 may include monitors, immersive VR headsets, high-resolution projectors, interactive touch screens, augmented reality (AR) glasses, portable tablets and smartphones.

[0054] Data storage 30 (non-volatile) may store raw and enhanced images, along with other relevant data. Data storage 30 ensures that data is securely kept for future reference, analysis, or sharing. Data storage 30 permanently stores a trained generator model utilized by the Al based image enhancement module 24, including its weights, biases, and configuration settings.

[0055] The visual recognition application 36 may leverage the enhanced imagery for visual recognition tasks, such as identifying underwater features, marine life, or man-made objects. The visual recognition application 36 may utilize advanced algorithms to interpret images, benefitting from the increased clarity and consistency provided by the system 10.

[0056] The data transmission system 34, the vehicle control application 28, the display device 26, the visual recognition application and data storage 30 are examples of potential uses of theenhanced imagery provided by the system 10. It should be understood that only one or any combination of such devices may be included in the system 10.

[0057] FIG. 2 illustrates a schematic diagram of the system for enhanced consistency in underwater imagery 10 focusing on data flows. The underwater camera 12 is shown to provide a raw image frame 40 for processing by the Al based image enhancement module 24. The Al based image enhancement module 24 provides an enhanced image frame 42 to the DUCT algorithm 22. The DUCT algorithm 22 includes various software modules including a frame quality evaluation module 46 (providing an image quality score 48), a blending steps determination module 58 (providing a number of steps N 60), a best template management module 50 (providing template data 52), a blending module 62 (providing a blended color adjusted frame 64), a color transfer module 54 (providing a color adjusted image frame 56) and a scene change detection module 66 (providing a similarity score 68).

[0058] The underwater camera 12 captures the raw image frames from underwater environments and outputs each raw image frame 40. These raw frames often contain distortions specific to underwater settings, such as color degradation and blurriness due to light refraction. The raw image frame 40 may be the initial data captured by the underwater camera 12, representing the unprocessed visual information directly from the underwater scene. Alternatively, the raw image frame 40 may be the result of some initial pre-processing such as noise reduction, white balance correction, contrast adjustment and image cropping and alignment.

[0059] A raw image frame 40 captured by the underwater camera 12 is composed of pixels, each representing the smallest unit of an image. These pixels carry information on light intensity and color, structured into color channels that typically correspond to the primary colors of light: red, green, and blue (RGB). The color channels for image processing and enhancement in the present disclosure are not limited to the RGB (Red, Green, Blue) model; alternatively, the YUV color space (or another color space), which separates luminance (Y) from chrominance (U and V components), can also be utilized for manipulating and enhancing underwater imagery. The aggregation of these pixels and their respective color channel values forms the complete image, capturing the visual essence of the underwater scene. The pixels in a raw image frame 40 embody the light intensity captured by the camera's sensor. Each pixel is represented by a combination of values across the RGB color channels, dictating the color and brightness of that specific point in the image. Raw image frames are composed of multiple color channels, usually RGB, each channel storing the intensity information for its respective color across the image. The intensity values in these channels may range from 0 to 255 in an8-bit image, where 0 represents the absence of that color (black) and 255 represents the full intensity of the color. The resolution of a raw image frame, typically measured in pixels (e.g., 1920x1080), indicates the total number of pixels in the image. Underwater cameras vary widely in their capabilities, and the resolution of the raw image frames they generate can range from standard definition to ultra-high definition, depending on the camera's specifications and the intended use of the imagery.

[0060] The underwater camera 12 captures raw image frames at a specific frame rate, which denotes the number of frames recorded or displayed per second. This frame rate can vary widely depending on the camera's design and the requirements of the particular underwater operation, ranging from lower frame rates (e.g., 24 fps for cinematic look) to higher frame rates (e.g., 60 fps or more for smooth motion capture). The Al-based image enhancement module 24 and the DUCT algorithm 22 are configured to process the raw image frames sequentially, accommodating real-time data flow directly from the camera 12 as well as batch processing from stored data. In real-time processing, such as for navigational aid for ROVs or live monitoring, the system 10 processes incoming frames on-the-fly. The Al module 24 and the DUCT algorithm 22 provide enhanced images without significant delay, maintaining the frame rate as closely as possible to the camera's output. Alternative embodiments perform batch processing from memory whereby raw image frames stored in data storage 30 (for example) are processed at a predetermined frame or clock rate. This batch processing mode allows the system 10 to access and enhance images sequentially at a rate optimized for processing power and quality outcomes, independent of the original capture frame rate. In both processing modes, the system 10 applies consistent quality improvements through the Al based image enhancement module 24 and color corrections through the DUCT algorithm 22 frame by frame.

[0061] The Al-based image enhancement module 24 utilizes advanced machine learning algorithms, specifically trained on underwater imagery, to correct distortions observed in raw image frame 40. This module enhances overall image quality by improving clarity, adjusting colors to more natural tones, and increasing contrast to mitigate the effects of underwater scattering and absorption. The Al based image enhancement module 24 outputs an enhanced image frame 42, which represents the visually improved version of the raw image frame 40. This frame has corrected distortions and enhanced features, ready for further processing by the DUCT algorithm 22.

[0062] The Al based image enhancement module 24 includes an encoder-decoder architecture for image enhancement as described with reference to FIGS. 5 to 8. The encoder-decoderarchitecture learns to enhance underwater imagery by encoding the input image into a compact representation and then decoding it back to an enhanced version.

[0063] The Al based image enhancement module 24, embodied as a GAN having a generator including an encoder-decoder network, is trained on a diverse underwater imagery dataset. This dataset includes a wide range of conditions such as different levels of turbidity, varying lighting conditions, and diverse underwater landscapes. The training process involves adjusting the generator model parameters of the GAN to minimize the difference between the model's output and the target image quality. This is achieved through various loss functions, which can include pixel-wise loss for direct similarity, perceptual loss for feature similarity, and adversarial loss for realism.

[0064] The Al-based image enhancement module 24 leverages a tailored GAN architecture optimized for underwater imagery enhancement. Three scales of model configurations have been developed to meet diverse requirements across computational resources and image enhancement objectives. The models range from small to large, each designed for specific application scenarios.

[0065] The small model configuration is engineered for environments with limited computational resources, such as onboard systems in remotely operated vehicles (ROVs) or autonomous underwater vehicles (AUVs). It features a lightweight generator architecture that employs depthwise and pointwise convolutions, significantly reducing the model's computational footprint while still providing quality image enhancement. This configuration is suited for real-time applications where processing speed is required, employing streamlined encoder-decoder blocks and instance normalization to enhance efficiency and image output quality.

[0066] The medium model configuration offers a balanced solution, suited for systems with moderate processing capabilities. This setup is based on a GAN architecture but includes advancements such as gated attention mechanisms within skip connections and instance normalization. These enhancements allow for more focused feature refinement and efficient normalization, making it suitable for applications that demand a compromise between image quality and processing speed, such as onboard analysis on research vessels.

[0067] The large model configuration is designed for maximum image quality enhancement, targeting platforms with substantial computational resources, like research labs or cloud-based processing centers. It utilizes a deep network architecture capable of processing larger-sized training images and incorporates bottleneck residual layers and a multi-scale discriminator network. At least five residual layers (e.g. 9) may be included between the encoder and decoderparts of the generator model. This configuration is aimed at applications where the highest quality image output is paramount and computational time is not a limiting factor, such as detailed scientific analysis or documentary filmmaking.

[0068] These configurations can be deployed independently, chosen based on the specific hardware capabilities and the real-time requirements of the deployment environment, e.g. based on the processing power of the processor 16. For instance, underwater vehicles with limited onboard computing power might employ the small configuration for real-time navigation and basic imaging tasks. In contrast, the large configuration might be reserved for post-mission analysis where higher image quality can significantly enhance the value of the collected data. Furthermore, systems that support dynamic switching between the various generator model configurations (small, medium or large) can adjust in real-time to changing computational resources or task requirements, optimizing the balance between enhancement quality and processing speed.

[0069] The training regime for the Al-based image enhancement module 24 involves a paired image training approach, wherein each raw underwater image is associated with a corresponding reference image for objective assessment. This approach combines multiple loss functions to guide the enhancement model toward generating images that closely match the reference images.

[0070] The reference images may be provided by professional enhancement by which images taken underwater are professionally edited by experts using photo editing software or algorithms to correct color, enhance clarity, and adjust brightness and contrast to approximate the appearance of the scene as if viewed under natural, non-water conditions. The reference images may additionally or alternatively be provided by controlled conditions photography by taking photographs in controlled underwater environments with optimal lighting and visibility conditions can also provide high-quality reference images. These images serve as a benchmark for enhancing more degraded images taken in less ideal conditions. The reference images may additionally or alternatively be provided by synthetic image generation by which reference images are synthetically generated using computer graphics techniques. This method may include creating realistic 3D models of underwater scenes and rendering these models using computer graphics software that simulates the optical properties of water. For training purposes, these reference images are paired with their corresponding raw underwater images. The paired dataset supports training the Al based image enhancement module 24 as it allows the model to learn the transformation needed to correct the distortions and enhance the overall quality of underwater images, mimicking the quality seen in the reference images.

[0071] The Al based image enhancement module 24 according to embodiments of the present disclosure may be trained based on at least one loss as described in the following. In some embodiments, the Al based image enhancement module 24 is trained based on any combination of some or all of the following losses. The loss function may employ an adversarial loss component so that the model learns a transformation function mapping raw underwater images to their visually enhanced versions. This process involves a generative network striving to produce images indistinguishable from real, enhanced images, while a discriminative network attempts to differentiate between the real and generated images. The adversarial loss optimizes this competition, refining the generative model's ability to replicate the quality of enhanced underwater imagery accurately. The loss function may incorporate an LI loss (mean absolute error), which targets the minimization of artifacts and blurriness by calculating the mean absolute difference between the reference images and the images enhanced by the model. This ensures a closer match in visual content between the generated images and their corresponding high-quality targets. The loss function may include a further loss that employs a deep learningbased image quality assessment metric that considers both the structural and textural aspects of images, providing a detailed and perceptually relevant evaluation of image quality. The loss function may incorporate an SSIM (Structural Similarity Index Measure) loss component to quantify the perceptual difference between the enhanced images and their corresponding reference images. SSIM evaluates the visual impact of three characteristics of an image: luminance, contrast, and structure, comparing local patterns of pixel intensities that have been normalized for luminance and contrast. This approach allows for a more detailed and human- perceptible measure of image similarity. The loss function can include a perceptual loss component that utilizes intermediate representations extracted from a pretrained Visual Geometry Group Network (VGG-19) (or other image classification CNN) to emphasize the significance of high-level features in the image enhancement process. This method compares the enhanced images with their reference counterparts by measuring differences in the feature maps at various layers of the VGG-19 network. This approach of combined loss functions ensures that the enhanced images not only match the reference images in terms of pixel accuracy but also in textures, patterns, and structural elements that contribute to the overall perceptual quality of the images.

[0072] The loss function may employ a linear combination of the aforementioned loss components to establish a comprehensive training objective. The balance among these components is maintained through weight scaling hyperparameters, which are empirically determined.

[0073] According to the present disclosure, the Al-based image enhancement module 24 is a deployed generator of a GAN. FIG. 4 illustrates a GAN training system 400 according to embodiments of the present disclosure, which also allows a method of training the GAN to be understood. The GAN training system 400 may be executed on a training Central Processing Unit (CPU) or a training Graphics Processing Unit (GPU) executing computer program instructions embodying the various components of the GAN training system 400 described below (including a generator 402, a discriminator 408 and the various modules).

[0074] The GAN training system 400 includes a pair of a training raw image 404 and a training reference image 410, a generator 402 outputting a training enhanced image 406, and a discriminator 408 providing a discriminator output 412. An adversarial loss module 416 outputs an adversarial loss 422, an individual loss module 418 outputs loss components 414 based on both the training reference image 410 and the training enhanced image 406, and a combined loss module 420 outputs a combined loss 424. The adversarial loss module 416 receives the discriminator output 412. The discriminator 408 receives the training reference image 410 and the training enhanced image 406 along with the adversarial loss 422. The combined loss module 420 receives both the adversarial loss 422 and the loss components 414. The generator 402 receives the training raw image 404 and the combined loss 424, utilizing feedback to iteratively refine the image enhancement capabilities. This training process repeats across multiple iterations to optimize the performance of the generator 402 using optimization algorithms.

[0075] The GAN training system 400 utilizes a dataset (stored on data storage of the GAN training system 400) comprising pairs of training raw images 404 and training reference images 410. The training raw image 404 is an unprocessed or minimally processed image captured from an underwater environment. These images typically exhibit common underwater imaging challenges such as poor light penetration, color distortion (mainly a blue or green tint due to selective absorption and scattering of light), blurring, and low contrast. The training reference image 410 is a high-quality image that either has been manually enhanced by experts or generated using advanced processing techniques to correct the distortions observed in the raw image. The reference images serve as the target output for the GAN, embodying the desired attributes in terms of clarity, color balance, and visibility. The training raw images 404 may be captured by underwater cameras equipped on divers, submarines, or remotely operated vehicles (ROVs). The training raw images 404 and training reference images 410 may be obtained through synthetic generation using software to simulate underwater conditions on above-water images. The training raw images 404 and training reference images 410 may beobtained from existing datasets. The training raw images 404 and the training reference images 410 may be still images or extracted frames from video footage.

[0076] In the GAN training system 400, the training raw images 404 are fed directly into the generator 402, which attempts to transform them into images that resemble the training reference images 410 in terms of enhanced quality. Reference Images 410 are used by the discriminator 408 in conjunction with the output from the generator 402 (in the form of the training enhanced image 406). The discriminator 408 assesses the similarity of the training enhanced image 406 from the training reference image 410, providing feedback (in the form of discriminator output 412) that influences the adversarial training process.

[0077] The generator 402 generates enhanced images (in the form of training enhanced image 406) from raw underwater images (in the form of training raw image 404). The generator 402 aims to transform raw images so that they are indistinguishable from high-quality reference images. The generator receives feedback through the combined loss 424 during the training process. This feedback guides learning of the generator 402. During training, the generator 402 dynamically adjusts based on feedback from the combined loss 424, fine-tuning weights and architectural behaviors to minimize discrepancies between raw and reference images. These adjustments may include optimizing layer- specific feature processing and employing adaptive mechanisms such as attention modules to enhance critical areas of the image, ensuring the generator 402 evolves to produce high-quality, realistic enhancements tailored to diverse underwater environments.

[0078] The generator 402 includes convolutional, normalization, and activation layers to transform the training raw image 404 into the training enhanced image 406. The generator 402 leverages deep learning techniques, specifically convolutional neural networks (CNNs), to process and progressively refine the image data through its network layers. The architecture of the generator 402 may be according to the small, medium, and large models described above. The small model according to embodiments of the present disclosure is described further with reference to FIG. 5 below. Each model variant is tailored for specific deployment scenarios, from embedded systems with limited computational resources to more robust systems capable of handling complex computations. Each of these models employ a form of encoder-decoder architecture, where the encoder part compresses the input image into a feature-rich latent space, and the decoder reconstructs an enhanced image from this latent representation. Skip connections may be used to preserve information from the earlier layers to the later layers, helping to maintain image details throughout the network.

[0079] The discriminator 408 distinguishes between real images and those generated by the generator 402. During training, the discriminator 408 evaluates authenticity of the images from the generator and provides feedback that shapes the training of the generator 402. The discriminator 408 functions similarly to a binary classifier within the GAN framework. The discriminator 408 receives both the training enhanced images 406 and the training reference images 410 during the training process. The discriminator 408 processes these inputs through a neural network, which typically includes layers of convolutional filters that extract features from the images. As the images pass through the layers, the discriminator 408 evaluates and compares their features to learn to identify the subtle differences between generated and real images. This process may involve assessing details like texture, color consistency, edge clarity, and overall image composition that contribute to the perceived authenticity of the images. The discriminator 408 and the generator 402 have an adversarial relationship in that the generator create training enhanced images 406 aimed to be indistinguishable from real images, effectively "fooling" the discriminator, whilst the discriminator 408 aims to accurately classify images as real or generated.

[0080] As described hereinbelow, the Al-based image enhancement module 24 may employ different GAN models defined as small, medium, and large, depending on processing capabilities. The discriminator 408 may vary with the GAN model type. In particular, the discriminator 408 of the small GAN model is designed without constraints to feature a global- local discrimination approach. The discriminator 408 model is not constrained by the need to be lightweight, as only the trained generator 402 is utilized during model deployment. The discriminator 408 uses two discriminators working at different scales: one local discriminator assessing smaller, e.g. 16x16, patches, and one global discriminator evaluating larger, e.g. 70x70 patches, facilitating fine-grained and consistent image quality across varying scales.

[0081] The medium-sized GAN model mirrors the discriminator 408 of the small GAN model maintaining the global-local setup.

[0082] The discriminators for the small and medium GANs operate at two different scales: a local discriminator that focuses on smaller image patches and a global discriminator that operates on larger pixel image patches. The local discriminator examines fine-grained details within small image patches and evaluates the similarity between small patches of the enhanced images and the corresponding patches of the reference image. The global discriminator operates at a larger scale, considering broader regions of the image and evaluates the overall structure, composition, and color consistency of the enhanced images compared to the reference images.

[0083] For the largest model GAN, the discriminator 408 is tailored for higher-resolution inputs and employs a multi-scale strategy with three discriminators, each operating on 70x70 patches. Images are downsampled progressively to enable a detailed and scale-sensitive evaluation, enhancing the network's ability to produce consistent results across different image resolutions. Varying scales are achieved by bilinearly downsampling the input image by a factor of two before feeding it into the subsequent discriminator so that each discriminator receives a different scale of input image.

[0084] The discriminator output 412 of the discriminator 408 may be a probability score or a classification decision indicating whether each image it reviews is more likely to be a real reference image or an image generated by the generator 402. The discriminator output 412 is used for calculating the adversarial loss 422 measuring how well the generator 402 is performing in its task of creating realistic images and is used to adjust the generator 402.

[0085] The discriminator output 408, denoted as y discriminator, represent a classification decision made by the discriminator model regarding the authenticity of the enhanced images compared to the reference images. The discriminator 408 takes pairs of images as input: the training enhanced images 406 generated by the generator (G(x)) and the corresponding reference images (y).

[0086] The discriminator model comprises convolutional layers, activation functions (e.g., Leaky ReLU), and possibly pooling layers, depending on the architecture. These layers extract features from the input images to make a classification decision. The discriminator model of the discriminator 408 provides a single scalar value (discriminator score) for each input image, indicating a likelihood that the input image belongs to the real (reference) class. The discriminator output 412 y may be obtained by applying a sigmoid activation function to the discriminator score.

[0087] Mathematically, the discriminator output can be represented as ydiscriminator=c (Discriminator_Score) (equation 1) where o is the sigmoid activation function.

[0088] The discriminator output 412 may lie in the range [0, 1], A value close to 1 indicates that the discriminator is confident that the input image is real (i.e., from the reference distribution), while a value close to 0 indicates that the discriminator believes the input image is fake (i.e., generated by the generator 402). The discriminator output 412 represents the probability that the input image (the training enhanced image 406) is real (not generated by the generator 402).

[0089] During the training process, the discriminator 408 is refined based on the adversarial loss 422, which quantifies how well the discriminator 408 distinguishes between real and generated images. This adversarial loss 422 is derived from the ability of the discriminator 408 to correctly classify training enhanced images 406 from the generator 402 as fake (generated). The adversarial loss 422 guides the training by prompting adjustments in the neural network parameters of the discriminator 408, specifically its weights and biases. These adjustments may be achieved through backpropagation, where the gradients of the adversarial loss 422 are used to update the internal settings of the discriminator 408 to improve its classification accuracy.

[0090] The adversarial loss module 418 quantifies the effectiveness of the generator 402 (G) in fooling the discriminator 408 (D) into classifying generated images as real. Although shown as a separate module herein, the adversarial loss 422 may be provided directly by the discriminator 408. The adversarial loss 422 is a metric used during the training phase to improve the generative model of the generator to define a mapping function from a source domain of raw underwater images (X) to a target domain of enhanced underwater images (Y), using the generator network (G) that maps X to Y. The loss function for the adversarial training, denoted as LOAN is computed as follows:Lr,zw (G,D) = Exy[log£) (x,y)] + Ex [log(l - Z)(x,G(x)))] (equation 2)

[0091] G is the generator 402 aiming to create images that closely resemble the authentic images from the desired domain Y. D is the discriminator 408 attempting to correctly classify real images y and generated images G(x). x represents the raw underwater images, (training raw image 404). y represents the reference underwater images (training reference image 410) that serve as the ground truth during training. Ex,y and Ex represent the expected values over the distributions of real and raw images, respectively. In this equation, the first term, Ex,y[logD(x,y)]Ex,y[logD(x,y)], calculates the log probability that the discriminator 408 assigns to real images being classified correctly as real. The second term, Ex[log(l-D(x,G(x)))], represents the log probability that the discriminator 408 assigns to the generated images being classified correctly as fake.

[0092] The generator 402 is trained to minimize at least the adversarial loss 422, indicating better performance in deceiving the discriminator 408 into determining that the generated images are real. Conversely, the discriminator 412 aims to maximize the adversarial loss 422, striving to become better at distinguishing between real and generated images.

[0093] For the small and medium-sized GAN models, which utilize a global-local discriminator architecture, the adversarial training objective is modified to account for themulti-scale assessment conducted by the discriminators. This modification incorporates separate loss terms for the global and local aspects of the images, with the global discriminator assessing the overall image content for authenticity and the local discriminator focusing on the finer details and textures. The combined adversarial loss thus becomes a weighted sum of the global and local losses, ensuring that the generated images are convincing both in overall composition and in high-resolution detail. The adjustment of the weighting between the global and local loss components is tuned during training to optimize the synthesis of enhanced underwater images that are realistic across different scales of perception. This multi-scale approach is more pronounced in the larger GAN model, which involves three separate discriminator networks, each contributing to the final adversarial training objective. The discriminator of the larger GAN uses a least-squares version of the adversarial loss (as opposed to the negative loss likelihood approach for the small and medium GANs) to stabilize training with high-resolution images.

[0094] The individual loss module 418 evaluates a variety of training losses as described in the following The individual loss module 418 evaluates how closely the generated images (training enhanced images 406) match the target training reference images 410. The individual loss module 418 employs several loss functions to guide the generator 402 towards producing images that visually resemble the desired outcome in terms of dehazing, color restoration, and overall quality. Each loss function is designed to focus on specific aspects of the generated image and plays a role in the learning process. Below is a description of different loss functions that have been found to support high quality outcomes.

[0095] LI loss, or mean absolute error, measures a pixel-wise absolute difference between the training enhanced images 406 from the generator 402 and the training references images 410. The LI loss may defined according to equation 3.Ln (G) = Ex,y [|| - G(x)||i] (equation 3)

[0096] The LI loss measures the discrepancy between the output generated by a model of the generator 402 G and the target or ground truth. G represents the generator neural network which takes an input image x and produces an enhanced image G(x). y is the target or ground truth image (training reference image 410), which represents the desired outcome after enhancement. Ex,y denotes the expectation over all pairs of input images x (training raw image 404) and target images (training reference image 410). [Ily - G( ) Hi] is the LI norm of the difference between the target image y and the generated image G(x), which calculates the absolute differences pixel-wise between the two imagesand sums these differences over the entire image. The equation for LI loss means that for each pair of images (x, y) in the dataset, where x is a raw underwater image and y is its corresponding enhanced image, the generator G attempts to create an image G(x) that is as close as possible to y. The LI loss calculates the total absolute difference between G(x) and y across all pixels, providing a measure of error. The goal during training is to minimize this loss, which would mean that the output, G(x), of the generator 402 is getting closer to the ground truth image y, therefore improving the quality of the image enhancement performed by G.

[0097] Perceptual quality assessment loss is generated based on a deep learning-based image quality assessment algorithm that evaluates perceptual similarity, considering both structural and textural components of the images. The perceptual quality assessment loss is computed using a convolutional neural network to ensure that the enhanced images produced by the generator 402 are of high perceptual quality. The perceptual quality assessment loss is generated by a quality assessment model that is designed to capture both the structural and textural information that are important for human visual perception. The quality assessment tool aims to reflect the way humans perceive image quality by leveraging deep learning, quality assessment mode uses a neural network assessing the perceptual quality of an image by comparing the deep features extracted from the reference and distorted (or processed) images. The training of the neural network component of quality assessment model is trained to minimize the perceptual difference between a pair of images. During this training process, the neural network learns to align its perception of image quality with human ratings from a given dataset. The quality assessment model optimizes the weights in a way that the extracted features from the enhanced image are as close as possible to those from the reference image, from a human perceptual standpoint. The quality assessment model evaluates the likeness in structure and texture, which are elements of how humans perceive and interpret visual data.

[0098] The individual loss module 418 further determines a structural similarity index loss to assess image quality. The structural similarity index loss is effective at preventing the creation of visual artifacts that can be induced by the perceptual quality assessment loss. The Structural Similarity Index (SSIM) is an advanced metric used to measure the similarity between two images. SSIM is particularly useful in the context of image processing as it provides a more perceptually relevant evaluation of image quality compared to traditional metrics like mean squared error (MSE). The SSIM index considers changes in texture, contrast, and luminance,which are crucial components of human visual perception. The SSIM loss may be defined according to the following equation 4.LSSIM ( 6) = Ey, [l — SS1M(J^6(^))] (equation 4)

[0099] LSSIM(G) represents the SSIM loss function for the generator G. G is the generator function that processes an input image x (training raw image 404) to produce a training enhanced image 406 G(x) y is the reference image (training reference image 410), which is the target outcome the enhanced image should aim to replicate in terms of quality and detail. Ex,y represents the expectation (or average) taken over all pairs of input and target images in the dataset. SSIM(y,G(x)) is the structural similarity index computed between the target image y and the generated image G(x). SSIM may range from -1 to 1, where a higher value indicates greater similarity. The SSIM loss function works by evaluating the discrepancy between the generated image and the reference image in terms of their structural similarity. The closer the SSIM value is to 1, the more structurally similar the two images are. The Structural Similarity Index (SSIM) is a metric that assesses the quality of images by comparing three key components: luminance, contrast, and structure. Luminance measures the matching brightness levels between the reference and target images, ensuring that the overall illumination remains consistent. Contrast evaluates the clarity and depth by comparing the spread or variability of pixel intensities, which highlights details within the image. Lastly, the structure component examines the correlation of pixel values that are spatially close, focusing on the preservation of visible patterns and textures. Together, these components provide a nuanced approach to image quality assessment that mirrors human visual perception, making SSIM a preferred method in various image processing applications.

[0100] Perceptual loss measures the similarity of high-level features between the generated and reference images (training enhanced image 406 and training reference image 410) by comparing the feature maps obtained from a pretrained convolutional neural network model. Perceptual loss allows for a deeper assessment of image quality by considering factors beyond pixel-level differences. Output channels of various layers in the convolutional neural network are extracted. The loss is then calculated as a weighted distance between these channels, derived from both the reference samples y and the enhanced samples generated by G(x). This approach allows for a more nuanced assessment of image quality, closely aligning with human perception by considering higher-level features beyond pixel-level differences.

[0101] The perceptual loss is calculated based on intermediate feature maps obtained from a pretrained VGG-19 model or another image classification model. The image classificationmodel has already been trained on a large dataset for image classification tasks. In the perceptual loss calculation, the feature maps from this pretrained model serve as a way to capture high-level image features. To compute the perceptual loss, feature maps from intermediate layers of the image classification model are extracted. The feature maps represent activations at different abstraction levels of the input images. The extracted feature maps from the image classification model are used to compute the perceptual loss between the training reference images 410 and the training enhanced images 406 generated by the generator 402 by comparing the feature representations of both the reference and enhanced images at multiple layers of the image classification model (where the comparison is between same level layers). The perceptual loss may be calculated as a weighted LI distance or cosine similarity between the feature maps derived from the reference and enhanced images.

[0102] The combined loss module 420 combines a plurality of loss components 414 described above as provided by the individual loss module 418. The combined loss module 420 synthesizes a holistic training objective by amalgamating multiple loss functions, each weighted to optimize the performance of the generative model of the generator 402 across various aspects of image quality. The combined loss module integrates the adversarial loss, LI loss, SSIM loss, perceptual loss, and perceptual quality assessment loss into a single objective function, as specified in equation 5 below.LCOMBINED = LAdversarial + AllLll + ASSIMLSSIM + AperceptualLperceptual + ADISTSLDISTS (equation 5)

[0103] LI, ASSIM, Aperceptuai, and ADISTS are hyperparameters that determine the relative weight of each loss component in the training process. These weights may be empirically set to values that balance the contribution of each loss to the overall training goal.

[0104] Adversarial Loss (LAdversarial) encourages the generator 402 to produce realistic images that the discriminator 408 cannot distinguish from actual enhanced images. LI Loss (LLI) measures the absolute differences between the generated images and the reference images, focusing on pixel -level accuracy. SSIM Loss (LSSIM) evaluates structural similarity between the generated and reference images, preserving visual structures crucial to human perception. Perceptual Loss (Lperceptuai) assesses higher-level discrepancies in content and style by comparing feature maps from a deep network, capturing nuances that simple pixel-based losses might miss. The perceptual quality assessment loss (LDISTS) provides a deep learning-based assessment of texture and structural similarities between the generated images and the target or reference images, offering a perceptually relevant measure of image quality.

[0105] Referring to FIG. 9, a flow chart representing a GAN training method 900 is provided. The GAN training method 900 is implemented through computer programming instructions stored on a non-transitory computer-readable medium that, when executed by a training processor, perform the steps of the GAN training method 900. The training process is executed on specialized computer hardware, typically involving high-performance CPUs and GPUs, to manage the computationally intensive tasks efficiently. These hardware components are orchestrated to execute complex neural network calculations, driven by software instructions designed specifically for deep learning.

[0106] In step 902, training images are input, which includes feeding training raw images 404 into the GAN training system 40 and feeding paired training reference images 410, which serve as the inputs for the training process.

[0107] In step 904, training enhanced images 406 are generated by the generator 402 processing the input training raw images 404 to produce enhanced versions.

[0108] In step 906, discriminator processing is performed in which both the training reference images 410 and the training enhanced images 406 are evaluated by the discriminator 408. The discriminator 408 aims to classify the enhanced images 406 as real or synthesized.

[0109] In step 908, the adversarial loss 422 is calculated based on the performance of the discriminator in distinguishing between real and generated images. This metric is used to improve the classification of the discriminator 408 and to push the generator 402 towards producing increasingly realistic images.

[0110] In step 910, the combined loss 424 is calculated that compares the various loss components 414 including the adversarial loss, LI, SSIM, perceptual, and performance quality assessment losses described above. The combined loss is used to train the generator 402.

[0111] In step 912, the model of the generator 402 is updated based on the feedback from the combined loss to minimize the combined loss 424, refining its ability to produce high-quality enhanced images. This iterative refinement advances the generator's capabilities.

[0112] In step 914, the model of the discriminator 408 is updated based on the adversarial loss 422, enhancing its ability to correctly identify and differentiate real from generated images.

[0113] In step 916, the method includes iterating or ending training. The training process assesses whether the models of the generator 402 and the discriminator 408 have converged to a satisfactory level of performance defined by a convergence criteria. If not, the process iterates with new batches of images. The training concludes when the desired accuracy and image quality are achieved as defined by the convergence criteria.

[0114] FIG. 5 provides a block diagram of a small generator network 500 in accordance with embodiments of the present disclosure. The small generator network 500 includes an encoder 502 and a decoder 504. The encoder 502 includes a 2D convolution layer 506, an instance normalization layer 508 and an encoder activation layer 510 at an image input end and a following series of downsampling units 512. Each downsampling unit 512 includes a downsampling block 514 and a basic block 516. The decoder 504 includes a series of upsampling units 518 followed by a transposed convolutional layer 522 and a decoder activation layer 524. The direction of input to output (upstream to downstream) through the small generator network 500 follows a data flow direction from the encoder 502 to the decoder 504, starts with the 2D convolutional layer 506 and ends with the decoder activation layer 524.

[0115] The small generator network 500 has a compact structure to suit environments with limited computational resources. The encoder 502 serves as an initial segment of the small generator network 500 where raw image frames 40 (see FIG. 2) are processed and transformed into a compact feature representation. The encoder 502 initiates with a 2D convolutional layer 506, which applies filters to the raw image frames 40 to extract various features. The 2D convolution layer 506 processes two spatial dimensions of height and width of input images. The 2D convolution layer 506 is followed by an instance normalization layer 508, which standardizes features within each image independently, enhancing the model's robustness to contrast variations and improving generalization across different underwater scenarios.

[0116] Instance normalization operates on each instance in a batch of input raw image frames 40 (see FIG. 2) independently, normalizing the data within each individual image rather than across the batch as in batch normalization. The process involves scaling and shifting the data so that it has a mean of zero and a variance of one, which in the context of image processing can equate to normalizing contrast within an image. Instance normalization can contribute to faster model training and better performance in generating high-quality images, as it simplifies the generation process by removing instance-specific contrast information.

[0117] Regarding instance normalization layer 508, this should be understood in the context of the GAN training system 400 of FIG. 4 and the GAN training method of FIG. 9 operating on a batch of training data, which refers to a subset of the training dataset that is processed by the generator and discrimator models at one time. Rather than updating weights (parameters) of the models after each single example (e.g. stochastic gradient descent), the GAN training system 400 may update the models after an entire training dataset (stored on data storage of the GAN training system 400) or mini batches of the training dataset is processed by the GAN training system 400 (e.g. batch gradient descent). In batch gradient descent, the weights of thegenerator 402 and the discriminator 408 are updated after running through all the training examples. In mini-batch gradient descent, the training dataset is divided into smaller parts and the weights are updated after processing each batch. The size of these batches is a hyperparameter for the training process that is adjusted during training.

[0118] Unlike batch normalization, which normalizes the data across an entire batch (taking into account the mean and variance of the batch), instance normalization normalizes the data within each individual image independently (by adjusting the data to have a mean of zero and a variance of one for each image, effectively standardizing the contrast on a per-image basis).

[0119] An encoder activation layer 510, which may employing a ReLU activation function, introduces non-linearity into the system, allowing the small generator network 500 to learn complex mappings between input raw image frames 40 (see FIG. 2) and the compressed feature space. Each downsampling unit 512 is a composite structure comprising a downsampling block 514 paired with a basic block 516. The downsampling block 514 reduces the spatial dimensions of the provided feature maps, condensing the information and reducing computational requirements. The downsampling block 514 utilizes efficient convolutional strategies such as depthwise separable convolutions to minimize parameter count as described with reference to FIG. 7. The basic block 516, incorporating residual connections to facilitate training deeper networks, maintains the feature map dimensions while refining the features through additional convolutional layers. The basic block 516 is described further with reference to FIG. 6. The basic block 516 enables the encoder 502 to effectively capture and encode pertinent features at various scales and resolutions.

[0120] The decoder 504 reconstructs enhanced image frames 42 (see FIG. 2) from the compressed feature representation provided by the encoder 502. The decoder 504 includes a sequence of upsampling units 518 that work inversely to the downsampling units, incrementally increasing the resolution of the feature maps. The upsampling units 518 each include the basic block 516 and an upsampling block 520. The upsampling block 520 leverage techniques including a pixel shuffle operation to upscale the feature maps efficiently, enhancing the resolution without heavy computational demands, as described further with reference to FIG. 8. The upsampled features are then passed through a transposed convolutional layer 522, which further refines the spatial details, and a decoder activation layer 524, typically utilizing the Tanh activation function, to ensure the final image output (enhanced image frame 42) has pixel values normalized between a specific range, suitable for visual display or further processing.

[0121] The small generator network 500 employs skip connections 530 to create direct pathways between the encoder 502 and decoder 504 layers. The skip connections 530 facilitate the preservation of high-frequency details by allowing gradients to flow unimpeded across the small generator network 500, which supports maintaining the sharpness and texture of the enhanced underwater images, while also aiding in the efficient training of the small generator network 500 by mitigating issues related to vanishing gradients.

[0122] The skip connections 530 connect corresponding layers of the encoder 502 with those of the decoder 504. That is, the skip connections 530 provide a direct link between layers at the same level or at symmetrical points in the encoder 502 and decoder 504 parts of the network. The skip connections 530 between the corresponding stages of the encoder 502 and the decoder 504 create shortcuts that skip over intermediate layers. This means that the outputs (feature maps) of a specific downsampling unit 512 in the encoder 502 are directly fed to a matching upsampling unit 518 in the decoder 504 with the aim to reintegrate spatial and detailed information that might be lost in the downsampling process, which can help the small generator network 500 to better reconstruct the original image. The first layer of the encoder 502 corresponds to the last layer of the decoder 504, the second layer of the encoder 502 corresponds to the second-last layer of the decoder 504, and so on. This kind of direct connection facilitates the combination of high-level abstract features (from deeper layers) and low-level detailed features (from earlier layers).

[0123] The small generator network 500 features a lean architecture containing only 911.5k parameters in one embodiment, making it well-suited for deployment on edge computing platforms, such as Nvidia Jetson boards and similar embedded systems, where processing efficiency and rapid inference are crucial. These platforms typically balance computational power with energy efficiency, providing real-time processing capabilities in compact and portable form factors.

[0124] Referring to FIG. 6, the basic block 516 is shown in further detail. The basic block 516 includes a channel shuffle 602 and a channel split 604. The basic block 516 includes first and second paths 614 and 616 after the channel split 604. The second path 616 includes a first pointwise convolution 606, a depthwise convolution 608 and a second pointwise convolution 610. The first path 614 provides a route direct to a concatenation 612 to merge the feature maps from the first path 614 and the second path 616.

[0125] The channel shuffle 602 rearranges the order of the channels in the input feature map to ensure cross-channel information flow, enhancing the learning capability of the network without additional computational cost. Following this, the channel split 604 divides the featuremap into two separate parts along the channel dimension, which results in a first path 614 and second path 616.

[0126] In the second channel path 616, a sequence of convolutions refine the features: a first pointwise convolution 606 applies a 1x1 convolution to mix the features within the channels, a depthwise convolution 608, which is a 3x3 convolution performed independently across each input channel, deeply processes spatial features with reduced computational overhead, and a second pointwise convolution 610 fuses the features across the channels once more.

[0127] The first path 614 provides a direct shortcut from the channel split to the concatenation 612. This direct path preserves the original features from the channel split in basic block 516, ensuring that the subsequent layers have access to both the original and processed feature maps.

[0128] The concatenation 612 serves as a reunion point for the dual pathways, merging the feature maps from the first path 614 and the second path 616. This operation combines the depthwise processed features with the original features, retaining high-frequency details while expanding the network's representational capacity.

[0129] Referring to FIG. 7, the downsampling block 514 is illustrated in further detail. The downsampling block 514 includes a first path 702 and a second path 704. The first path 702 includes a first depthwise convolution 712 and a first pointwise convolution 714. The second path 704 includes a second pointwise convolution 706, a second depthwise convolution 708 and a third pointwise convolution 710. The outputs from the first path 702 and the second path 701 are merged at a concatenation 716.

[0130] The downsampling block 514, as detailed in FIG. 7, reduces the spatial dimensions of the feature maps within the small generator network 500. The first path 702 is responsible for directly downsampling the spatial dimensions of the input feature maps without altering the depth. The first path 702 includes a first depthwise convolution 712 that applies filtering across each input channel independently, maintaining the depth of the feature maps while capturing spatial features. Following the first depthwise convolution 712, the first pointwise convolution 714 is employed. This layer uses 1x1 convolutions to combine the filtered channels for integrating the spatial features captured by the depthwise convolution 712.

[0131] Simultaneously, the second path 704 operates on the input feature maps to transform and refine them. The second path 704 starts with a second pointwise convolution 706, which alters the channel-wise representation of the input features, possibly increasing or decreasing the number of channels depending on the network's needs. This is followed by a second depthwise convolution 708 that spatially processes the features within each channel. The thirdpointwise convolution 710 amalgamates these features, preparing them to be combined with the first path 702.

[0132] The outputs of the first and second paths 702, 704, now processed and downsized, are then combined at the concatenation 716. This merging step (feature map concatenation) allows the network to fuse the distinct feature representations processed by the first and second paths 702, 704, enriching the feature set available for the subsequent layers of the network.

[0133] In the example embodiment, the first depthwise convolution 712 and the second depthwise convolution 818 are implemented as a 3x3 depthwise separable convolution with a stride of 2. This configuration allows the convolutional layer to filter input feature maps channel-wise, applying a distinct 3x3 kernel to each channel, while the stride of 2 effectively halves the spatial dimensions of the output. A subsequent pointwise convolution effectively combines these features across the channels, preserving information while reducing dimensionality.

[0134] The architecture of downsampling block 514 is crafted to enhance processing efficiency through the direct handling of feature maps. Upon entry into the first path 702, feature maps are subjected to the first depthwise convolution 712, which is immediately succeeded by the first pointwise convolution 714. This sequence is designed to capture spatial relationships and to facilitate the integration of these relationships across the feature channels without any intermediate mixing or reordering of channels. In parallel, the second path 704 commences with the second pointwise convolution 706 to alter the feature map depth, followed by the second depthwise convolution 708. The design of each path 702, 704, by directly transitioning through convolutional layers without a channel shuffling mechanism, allows for the refined processing of feature maps. The paths culminate in a concatenation point 716, where the harmonized feature maps combine to form a composite output, rich in spatial and channel detail, purely derived from the input feature maps' intrinsic properties.

[0135] FIG. 8 illustrates an upsampling block 526 in accordance with embodiments of the present disclosure. The upsampling block 526 includes a channel shuffle 802, a channel split 804, a first path 806 comprising a first depthwise convolutional layer 810, a first pointwise convolutional layer 812 and a first pixel shuffle 814. The second path 808 includes a second pointwise convolutional layer 816, a second depthwise convolutional layer 818, a third pointwise convolutional layer 820 and a second pixel shuffle 822. The first and second paths 806, 808 come together at a concatenation 824.

[0136] The channel shuffle 802 layer rearranges the data channels of the input feature maps to promote cross-channel information flow. The channel split 804 operation divides the featuremaps into two separate groups along the channel dimension to create two processing paths (the first path 806 and the second path 808) for different subsets of features, which can then be individually operated on before being merged back together at the concatenation 824.

[0137] The first depthwise convolutional layer 810 applies a depthwise spatial convolution on each input channel separately, which is a computationally efficient way to process spatial information. The first pointwise convolutional layer 812 may be a 1x1 convolution to combine the features learned by the first depthwise convolution 810, effectively allowing the network to leam more complex features.

[0138] The first pixel shuffle 814 operation reorganizes the output of the previous convolutional layer, upscaling the spatial resolution while downsampling the depth (number of channels).

[0139] The second pointwise convolutional layer 816 is similar to the first pointwise layer 812 and processes the second set of features from the channel split 804 to potentially capture a different aspect of the input data. The second depthwise convolutional layer 818 applies another set of depthwise convolutions, again keeping computational costs low The third pointwise convolutional layer 820 combines the depthwise processed features before they are upscaled. The second pixel shuffle 822 completes the upsampling process for the second path 808 before feature maps from the first path 806 and the second path 808 are combined along the channel dimension at concatenation 824.

[0140] The first and second pixel shuffle 814, 822 operations may be understood in greater detail from the paper "Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network" (M. J. Islam, P. Luo, and J. Sattar, 2020. arXiv: 2002.01155) and is a technique that rearranges the elements of a tensor with shape (H, W, C * rA2) into a tensor of shape (H * r, W * r, C), where H and W are the height and width of the original low-resolution image, C is the number of channels, and r is the upscaling factor. This is accomplished through a sub-pixel convolution layer that aggregates the feature maps from low-resolution space and builds the super-resolution image in a single step.

[0141] Specifically, the pixel shuffle operation effectively implements a convolution with a fractional stride of 1 / r, thus increasing the image resolution while avoiding the use of any learnable parameters or complex operations. This significantly reduces the computational complexity and execution times.

[0142] Pixel shuffle speeds up processing by eliminating the need for high-resolution space operations until the very end of the upsampling block 526 and provides a more complex and potentially better-performing low resolution to high resolution mapping as the small generatornetwork 500 learns the processing necessary for super-resolution. Additionally, this method helps to reduce checkerboard artifacts commonly seen in GAN-generated images, which can be introduced by transposed convolutions.

[0143] In the small generator network 500, the basic block 516 and the upsampling block 526 employ a channel splitting operation at their inputs. Specifically, channel splitting divides input feature maps into separate groups along the depth dimension — where each 'channel' corresponds to a feature detector or a set of activations. For instance, the basic block 516 receives receive input feature maps and divides them into two subsets, where the first path 614 processes one group of channels, and the second path 616 processes another. Contrastingly, the downsampling block 514 utilizes a different approach where the input feature maps are duplicated rather than split. This means the entire set of channels is fed into both the first path 702 and the second path 704.

[0144] Enhancing underwater imagery using the Al based image enhancement module 24 introduces improvements in image clarity, contrast, and color accuracy. The Al-based image enhancement module 24 excels in correcting distortions specific to underwater conditions, thereby elevating the quality of each processed frame. However, while individual frame enhancement yields sharper and more vivid images, flickering across frames may remain. This flickering effect, which has been found to result, at least in part, from frame-to-frame color inconsistencies, can detract from video quality. Significant scene changes may introduce additional complexity to the enhancement process. As the underwater environment or lighting conditions shift, the color characteristics that were suitable for one scene may not perfectly suit the next, causing abrupt visual transitions.

[0145] The DUCT algorithm 22 ensures color consistency across consecutive frames, mitigating flickering and facilitating smoother transitions between scenes. By dynamically adjusting color characteristics, the DUCT algorithm 22 maintains visual continuity across the video sequence. This sophisticated method enhances individual frames processed by the AI- based image enhancement module 24 and the overall cohesiveness of the video through the DUCT algorithm 22.

[0146] The frame quality evaluation module 46 of the DUCT algorithm 22 analyzes the enhanced image frame 42 to assess its quality based on predefined criteria, generating an image quality score 48. This score reflects the frame's visual fidelity and suitability to serve as a reference in the color consistency process. The frame quality evaluation module 46 quantitatively assesses the quality of enhanced underwater images or video frames. In one embodiment, the frame quality evaluation module 46 employs a deep learning-based approachto evaluate aspects of image quality. In such an embodiment, the frame quality evaluation module 46 utilizes a pretrained deep learning model that has been exposed to a vast array of image-text pairings. This pretrained model captures a comprehensive understanding of various quality dimensions across a wide range of images, enabling it to assess both the tangible (e.g., sharpness, exposure) and intangible (e.g., aesthetics, mood) aspects of image quality. The evaluation process involves the generation of textual prompts that describe desired image qualities or attributes. These prompts can be specifically crafted to assess different aspects of image quality, such as brightness, contrast, or even more abstract qualities like the "feel" of the image. The frame quality evaluation module 46 uses these prompts to guide the assessment process, allowing for a flexible evaluation framework that can adapt to various quality dimensions. For each enhanced image or frame, the frame quality evaluation module 46 performs inference by presenting it alongside a set of predefined prompts to the pretrained model. The model then assesses the compatibility of the image with the qualities described in the prompts, generating scores that reflect the image's adherence to these quality attributes. The output from the model is a set of quantitative scores corresponding to each assessed quality dimension. These scores provide a comprehensive evaluation of the image's quality, encompassing both its physical characteristics and its ability to convey desired abstract qualities. The individual quality dimension scores are aggregated or selectively combined to form a comprehensive quality score for each image or frame, which corresponds to the image quality score 48.

[0147] In another embodiment, the frame quality evaluation module 46 includes a combination of deep learning and traditional image quality assessment techniques. A convolutional neural network (CNN) may extract pertinent features from underwater images, focusing on aspects like texture, color distribution, and edge sharpness. Pretrained networks such as VGG or ResNet, may be adapted to underwater imagery through transfer learning, to capture a range of features reflective of image quality. Algorithmic processes may capture image quality metrics such as the Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Colorfulness metric. These metrics evaluate various dimensions of image quality, including brightness, contrast, and color fidelity. A machine learning model, e g. a neural network, may be provided and trained on a dataset of underwater images annotated with quality scores. This model learns to predict image quality scores based on the extracted features and traditional metrics, offering insights into the perceptual quality of the image. A final image quality score is a composite measure that reflects objective quality aspects and subjective perceptual qualities and output as the image quality score 48.

[0148] The image quality score 48 is a numerical value representing the assessed quality of the enhanced image frame 42 and is used to identify the most suitable frames for template selection. The best template management module 50 manages the selection and storage of enhanced image frames as templates based on their quality scores. The best template management module 50 manages multiple template types, including current, next, previous, and raw image templates. The current enhanced image template serves as the primary reference for color transfer and blending processes. It represents the highest quality frame encountered up to the current moment in the video sequence, determined by its image quality score. When a new enhanced image frame receives a higher score 48 from the frame quality evaluation module 46, the current enhanced image template is updated to this new frame. This ensures that color transfer operations use the most visually appealing frame as a reference. Upon detecting a frame with a higher quality score 48 than the current best, the algorithm may encounter two scenarios. A first scenario occurs when blending (discussed further below) is in progress. If the system is already blending between templates, the newly identified superior frame is saved as the next image template. This template will be utilized in future blending steps once the current blending process concludes. A second scenario occurs when no current blending is currently taking place. In the second scenario, the newly observed frame is set as the new image template. This initiates a new blending process to gradually transition from the previous template to this new template, ensuring a smooth change in visual appearance across frames.

[0149] The best template management module 50 also manages a raw image template, which represents the raw counterpart of the current template and is used for scene change detection (described further below) and resetting the best score under significant environmental shifts. When a new enhanced image template is detected, the raw template is updated to the raw version of enhanced image frame that has been set to the new image enhanced image template.

[0150] The template data 52 includes information on the current enhanced image template, which is a reference for color transfer, a newly observed image template that becomes the next template for color transfer, and a temporary next template stored (queued) during blending processes. Additionally, the template data includes a raw image template, which is a raw image counterpart of the current enhanced image template used in scene change detection.

[0151] The blending steps determination module 58 calculates the number of steps (N) 60 to blend the color characteristics between the current enhanced image template and a newly observed template, ensuring a smooth transition in visual appearance across frames. That is, the number of blending steps is determined based on a difference in color characteristics between the current enhanced image template and a newly observed image template so that agreater difference results in a greater number of steps. Referring to equation 6 below, this calculation is based on comparing mean pixel values (Ma and Mb) for each color channel of the current and new templates (Ta and Tb, respectively), across all pixels (X, Y) and color channels (C). A smoothness factor (r|), which may be empirically set, influences the sensitivity of blending to changes in color characteristics, with a lower value resulting in more blending steps. Nmax is a maximum allowable number of blending steps, and may also be empirically determined, to balance transition smoothness with processing efficiency. The outcome, N, is used to adjust a blending factor (<])) in subsequent processing, dictating the proportion of color characteristics from each template applied during each blending step. (equation 6)

[0152] The number of steps N 60 is output by the frame quality evaluation module 46 as the number of steps required for blending color characteristics between frames.Color transfer module 54 applies color adjustment to enhanced image frames, resulting in a color-adjusted image frame 56. The color transfer module 54 operates on the principle of adjusting the color characteristics of the enhanced image frame 42 to align with the color characteristics of a selected template (e g., current, or next enhanced image templates). This alignment is critical for maintaining visual consistency across frames, especially in sequences where lighting conditions or environmental factors may cause abrupt color shifts. The color transfer module 54 converts the color space of the enhanced image frame and the template from RGB to a decorrelated color space. This conversion reduces correlations between the color channels, allowing for independent manipulation of color characteristics. The chosen color space is one where the correlation between the color channels is minimized (such as lap or another perceptually uniform color space), facilitating adjustments that closely mimic human visual perception. For both the enhanced image frame 42 and the template (which could be a current or next template depending on the operational context within the DUCT algorithm 22), the color transfer module 54 calculates statistical measures representing color characteristics, e.g. specifically, the mean and / or standard deviation for each color channel in the convertedcolor space. These statistical measures capture the core color characteristics of both the enhanced image frame 42 and the enhanced image template. The color transfer module 54 adjusts the color characteristics of the enhanced image frame 42 to match those of the template by: subtracting the mean of each color channel of the enhanced image frame 42 to center its distribution, scaling the centered distribution by the ratio of the template's standard deviation to the frame's standard deviation for each color channel to adjust the spread of the color distribution of the enhanced image frame 42 to match that of the template, and adding the mean of the template's color channels to the scaled distribution to shift the color distribution enhanced image frame 42 to align with the template's central tendency. After adjusting the color characteristics, the color-corrected frame is converted back from the decorrelated color space to the standard RGB color space to prepare the color adjusted image frame 42 for display or further processing.

[0153] In other words, the color transfer module 54 adapts the color characteristics of enhanced image frames to align with those of an input enhanced image template, which includes, for each color channel, computing the mean and standard deviation for both the frame and the template and adjusting the colors of the frame by aligning mean of the frame with that of the template and / or scaling the color values in each channel of the enhanced image frame 42 to have the same standard deviation as the corresponding color channel in the template. The color transfer module 54 is applied to each enhanced image frame in relation to the chosen template.

[0154] A colour adjusted image frame 56 is output from the color transfer module 54 representing an enhanced image frame with adjusted color characteristics aligned with color characteristics of the template.

[0155] When there is a new image template that has been set based on the enhanced image frame 42 having an image quality score 48 that exceeds the best score, gradual color adjustments may be applied across N frames by the blending module 62. The blending module 62 in the Dynamic Underwater Color Transfer (DUCT) algorithm addresses the issue of abrupt color changes across video frames, which was a limitation observed in prior algorithms. The blending module 62 uses a previous template frame and a new template frame. The previous template frame corresponds to the previous best enhanced image frame prior to it being updated by the new best enhanced image frame. During the blending process color characteristics from both templates are applied to the current enhanced image frame, ensuring a gradual transition of color characteristics between frames.

[0156] The blending may be conducted according to the equation 7 below: lab = la • (1 - ( / >) + lb - <p (equation 7)la and lb are the frames to be blended, 4> is the blending factor that ranges from 0 to 1, and lab is the resulting blended frame. This calculation is performed on a pixel-by-pixel basis across all color channels.

[0157] The algorithm of the blending module dynamically adjusts the blending factor (<]>) based on the current blending step (step) and the total number of blending steps N, calculated as <|> = step / (N + 1). With each successive frame, the influence of the new template gradually increases while the influence of the previous template correspondingly decreases, leading to a smooth transition in color characteristics across frames. When the blending module reaches the final step, the algorithm directly applies the color characteristics from the new template to the current frame, marking the completion of the blending process for that set of templates. If a next template is waiting, the algorithm proceeds to blend with this new template, updating the total number of blending steps (N) for the new blending process.The blending module 62 utilizes the color transfer function of the color transfer module as a precursor to the blending operation by first applying the color transfer function to adjust the color characteristics of the current enhanced image frame to match those of the previous and new enhanced image templates. This process results in two color-adjusted frames: one where the current frame's colors are adjusted to match the previous template and another adjusted to match the new template. The final blending operation is performed pixel-by-pixel across all three color channels, combining the two color-adjusted frames based on the blending factor. The blending module 62 outputs a blended color adjusted image frame 56.

[0158] The scene change detection module 66 monitors for significant changes in the scene that may affect the relevance of the current template, using image analysis to output a similarity score 68. The score is used to determine when to update the template to better match new scene characteristics by resetting the best score. The scene change detection module 66evaluates the similarity between the raw image of the current template and the raw image of the current enhanced image frame 42. The raw images are compared to determine significant changes in the scene. The scene change detection module 66 determines a similarity score 68, which can be compared to a threshold. When the threshold is exceeded, a scene change is detected and the best score is reset to zero. In one embodiment, scene detection module 66 may leverage deep neural network features to assess similarities, focusing on both the structural integrity and textural details of the compared images. Other embodiments for comparing the raw images can be used either alone or in combination. Structural Similarity Index Measure (SSIM) is one example, which is a perception-based model that considers changes in luminance, contrast, and structure between two images. In another alternative, feature matching using descriptors couldbe employed including Scale-Invariant Feature Transform (SIFT) or Speeded-Up Robust Features (SURF), which extract and compare feature points between images. The number of matched features and the quality of matches can serve as indicators of scene continuity or change. A histogram comparison could also be included wherein the color or intensity histogram of an image provides a global description that can be useful for scene change detection. Histogram intersection, Chi-square distance, or Bhattacharyya distance between histograms of the raw frames can quantify scene changes. Deep learning-based change detection can be deployed, especially Convolutional Neural Networks (CNNs) or Transformerbased models, which can be trained to directly classify pairs of images as "same scene" or "scene change".

[0159] Similarity score 68 provides a value indicating the degree of visual similarity between a new enhanced image frame and a current template, used to detect scene changes.FIG3 provides a flowchart of a method 300 for enhanced consistency in underwater imagery. The method 300 is performed by the processing unit 14 executing the underwater imagery enhancement software 20.

[0160] It will be understood by those skilled in the art that all the steps as illustrated in FIG. 3 do not have to be performed in the order as shown in FIG 3 Instead, some steps can be performed in parallel or earlier or later than shown in FIG. 3.

[0161] Method 300 starts at 302. The method 300 initializes variables at step 304 to prepare for processing enhanced image frames provided by the Al based image enhancement module 24. A blending flag is set to false, indicating that template blending is not initially occurring. Various template variables are initialized to none, including the current enhanced image template (Tprev), the raw image used to create the current template (TR), a newly observed image template (TEnew), and a next template if currently blending (TEnext). Initial values for the total number of blending steps (N and Nnext) and the current blending step (step) are set to 1, the best score is initialized to zero and a threshold for scene change detection (dists thresh) is set to an empirically determined value.

[0162] At step 306, the algorithm receives the enhanced image frame 42 and the counterpart raw image frame 40. The processing pipeline outlined by method 300 (after initialize variables step 304) is performed on each newly received enhanced image frame 42 and generates a color adjusted version for each newly received image frame 42.

[0163] At step 308, frame quality of the enhanced image frame 42 (FE) is evaluated using the frame quality evaluation module 46 to provide a score (image quality score 48) representing a determined quality of the enhanced image frame 42. The frame quality evaluation of step 308may evaluate image quality by considering perceptual features, such as visual clarity, contrast, and texture detail, comparing these attributes against learned models of high-quality imagery to generate the score.Step 310 compares the score to a best score that is maintained based on scores achieved by prior enhanced image frames (or a best score of zero for a first processed enhanced image frame or after a reset of the best score). If a new best score is detected, the algorithm checks if there is a current blending process (at step 312). If there is current blending, the enhanced image frame (FR) is set as a next template (TEnext) and the template is queued (step 320) for later application. In step 322, the raw image frame (FR) corresponding to the enhanced image frame is set as the raw template (TR). Further, and in step 324, a next number of blending steps (Nnext) is determined. The number of blending steps is determined by the blending steps determination module 58 based on color characteristic differences between the new template (TEnew) and the next template (TEnext).

[0164] If it has been determined that there is no current blending in step 312, the algorithm sets the current enhanced image (FE) as the new template (TEnew) (at step 314) and sets the counterpart raw image frame (TR) as the best raw template (TR) (at step 316). Further, the next template (TEnext) is set to none and the blending flag is set to true. The blending flag is used in steps 312 and 326 to determine whether a blending process is being performed. In step 318, the number of blending steps (N) is determined in step 318 based on a difference in color characteristics between a previous template (TEprev) and the new template (TEnew), which is set in step 314. The number of blending steps N 60 is determined by the blending steps determination module 58 as described previously.In step 326, there is a second step of determining if a current blending process is being performed based on the blending flag. If there is a current blending process, the algorithm determines if the number of blending steps is complete in step 308. If the number of blending steps is complete, a final one of the color transfer processes of the blending process is applied in step 346. That is, the enhanced image frame (FE) is set to a result of the color transfer function (CT(FE, TEnew)) by which the enhanced image frame has its color characteristics adjusted according to the new template (TEnew). The color transfer function is executed by the color transfer module 54. The color transfer function adjusts the color characteristics of the enhanced image frame to match those of the new template by first aligning their color spaces, then adjusting the mean and standard deviation of the color distributions channel-wise, ensuring the enhanced image adopts the color ambiance of the template. Further, the currentcount of steps (step) is reset to 1 and the stored previous template (TEprev) is set as the new template (TEnew)In step 348, it is determined whether there is a queued template (see discussion of step 320 above). If not, the color adjusted enhanced image frame (from step 346) is returned. Further, the new template (TEnew) is set to none and blending is set to false.

[0165] If there is a queued template, the new template (TEnew) is set, at step 352, to the queued template (TEnext) from step (320) and the next template (TEnext) is set to none. At step 354, the number of blending steps (N) is set to the next number of blending steps (Nnext) from step 324 such that when the next enhanced image frame is received at step 306, the blending process begins based on that enhanced image frame and the queued template. The color adjusted enhanced frame from step 346 is returned in step 356.

[0166] Returning to the outcome of decision 338 as to whether the number of blending steps are complete, if the blending process is not yet complete, the algorithm determines a blending factor (<|>) in step 340. The blending factor is calculated as a fraction of the current blending step over the total number of blending steps plus one, determining the proportionate influence of the new template on the enhanced image frame during the blending process. In step 342, a blended color transfer is performed in step 342 using the color transfer function. That is, the color transfer function (CT(FE, TEprev)) is applied to the enhanced image frame and the previous template so that the color characteristics of the previous template are applied to the enhanced image frame and the color transfer function (CT(FE, TEnew)) is applied to the enanced image frame and the new template (set in step 314) so that the color characteristics of the new template are applied to the enhanced image frame. The color characteristics of the enhanced image frame is adjusted based on a blend ( blend(CT(FE, TEprev), CT(FE, TEnew), 4>) of the color transfers from the pervious and new templates according to the dynamically adjusted blending factor from step 340. The blending step 342 is performed by the blending module 62, which operates by linearly interpolating between the two color-adjusted enhanced image frames on a per-pixel basis across all color channels, using the blending factor <|) that varies from 0 to 1 (as discussed with reference to equation 7 above). The resulting frame is a blend of the enhanced frame adjusted to the previous template's color characteristics, and the enhanced image frame adjusted to the new template's color characteristics, ensuring a smooth transition between frames. The step count is increased by 1. The color adjusted enhanced image frame from step 342 is returned.

[0167] Returning to decision step 326, if there is no current blending and no new best score is detected in step 310, the color transfer function (CT(FE, TEprev)) is directly applied to theenhanced image frame using the previous template. The algorithm then checks for significant scene changes by comparing the current raw frame (FR) against the best raw frame (TR) using the scene change detection module 66. In one embodiment of step 330, the structural and textural similarities between the raw image frame of the current template (TR) and the raw image frame of the enhanced image frame (FR) are evaluated to produce the similarity score. This algorithm may be the same as that used to determine perceptual quality assessment loss as described above with reference to FIG. 4. Significant changes in scene composition or quality degradation, triggering a reset in in the best score if the discrepancy (similarity score) surpasses a predefined threshold (dists thresh). That is, if a major scene change is detected in step 330, indicating a divergence greater than the predefined threshold, the best score is reset to zero to accommodate for the new scene dynamics by facilitating the next received enhanced image frame being set as the new template in step 314. The enhanced image frame with the color characteristics adjusted as a result of step 328 is returned in steps 334 or 336.

[0168] Accordingly, method 300 provides a number of routes for determining and outputting a color adjusted version of the enhanced image frame. A new template may be used for applying the color transfer or a previous template may be used depending on whether the enhanced image frame has a sufficiently high image quality score. Further, the color transfer may be performed in a blended way according to a dynamic number of blending steps determined depending on a difference in mean pixel values between the next template to be used and the previous template used. Each enhanced image frame that is received is converted to a single color adjusted enhanced image frame by performance of method 300, thereby facilitating real-time, frame-by-frame, color adjustment to reduce flickering and enhancing consistency in a video output.

[0169] The invention has been described by reference to certain embodiments discussed above. It will be recognized that these embodiments are susceptible to various modifications and alternative forms well known to those of skill in the art.

[0170] Further modifications in addition to those described above may be made to the structures and techniques described herein without departing from the spirit and scope of the invention. Accordingly, although specific embodiments have been described, these are examples only and are not limiting upon the scope of the invention.

Claims

CLAIMS1. A generator network for processing underwater images, comprising: an encoder-decoder model, comprising: an encoder including a series of downsampling units configured to process input underwater images, wherein each downsampling unit comprises a downsampling block that includes a combination of pointwise and depthwise convolutional layers for reducing spatial dimensions of the input underwater images, thereby providing encoded feature maps; and a decoder including a series of upsampling units configured to generate enhanced images from the encoded feature maps provided by the encoder, wherein each upsampling unit comprises an upsampling block that includes a combination of pointwise and depthwise convolutional layers for increasing spatial dimensions and enhancing features representation of the enhanced images; wherein the generator network is adapted to receive the input underwater images and output corresponding enhanced images.

2. The generator network of Claim 1, wherein the encoder includes an instance normalization layer disposed prior to the series of downsampling units, the instance normalization layer being operative to normalize feature maps on a per-image basis as derived from the input underwater images.

3. The generator network of Claim 1 or 2, wherein each downsampling unit includes a basic block comprising a channel shuffle layer disposed upstream of the combination of pointwise and depthwise convolutional layers4. The generator network of Claim 3, wherein the downsampling block does not include a channel shuffle layer.

5. The generator network of any preceding Claim, wherein the upsampling block includes a pixel shuffle layer configured to increase a resolution of feature maps by rearranging tensor elements from a lower resolution to a higher resolution.

6. The generator network of any preceding Claim, comprising skip connections connecting the downsampling units of the encoder with corresponding upsampling units of the decoder.

7. A computer-implemented method for training a generative adversarial network (GAN) for enhancing underwater images, the method performed by at least one processor executing programming instructions stored on a non-transitory computer readable medium, the method comprising: providing, to the at least one processor, a training dataset comprising pairs of raw underwater images and corresponding reference images; processing, by the at least one processor, the raw underwater images through a generator model of a generator to produce enhanced images; comparing, by the at least one processor using a discriminator model of a discriminator, the enhanced images and the corresponding reference images to generate a discriminator output representing a classification indicating a likelihood of the enhanced images being synthetically generated by the generator model; computing, by the at least one processor, an adversarial loss based on the discriminator model performance in correctly classifying the enhanced images according to the discriminator output; calculating, by the at least one processor, a combined loss for the generator model, the combined loss including a weighted sum of a plurality of loss components, each loss component corresponding to a different aspect of image quality enhancement, wherein the plurality of loss components includes the adversarial loss and at least one of a first loss component based on a mean absolute difference between corresponding pixels of the enhanced images and the corresponding reference images; a second loss component based on a deep learning model providing perceptual similarity assessment of texture and structural characteristics between the enhanced images and the corresponding reference images and which has been trained to include image datasets annotated based on human perception; a third loss component based on a luminance, contrast, and structure comparison of the enhanced images and the corresponding reference images; and a fourth loss component based on comparing reference feature maps with corresponding enhanced image feature maps obtained from processing the enhancedimages and the corresponding reference images through an image classification convolutional neural network, wherein the reference feature maps and the enhanced image features maps are extracted from an intermediate layer of the image classification convolutional neural network; and adjusting, by the at least one processor, the model of the generator based on the combined loss; and adjusting, by the at least one processor, the model of the discriminator based on the adversarial loss.

8. The computer-implemented method of Claim 7, wherein the generator comprises an encoder-decoder model.

9. The computer-implemented method of Claim 7, wherein the generator comprises an encoder-decoder model according to Claims 1 to 6.

10. The computer-implemented method of Claim 8, wherein the encoder-decoder model comprises attention-modulated skip connections.

11. The computer-implemented method of Claim 8, comprising a plurality of bottleneck residual layers between encoder and decoder parts of the encoder-decoder model.

12. The computer-implemented method of any of Claims 7 to 11, wherein the discriminator comprises local and global discriminator models including the discriminator model, the local and global discriminator models operating at different image patch sizes.

13. The computer-implemented method of Claim 12, wherein the discriminator comprises the local and global discriminator models and an intermediate discriminator model, each operating at different image patch sizes.

14. The computer-implemented method of any one of Claims 7 to 13, wherein the adversarial loss is computed according to a negative log likelihood approach.

15. The computer-implemented method of any one of Claims 7 to 13, wherein the adversarial loss is computed according to a least-squares objective.

16. The computer-implemented method of any one of Claims 7 to 15, comprising iterating the computing-implemented method, outputting a trained generator and deploying the trained generator on a system for enhancing underwater imagery to enhance raw image frames.

17. A system for enhancing underwater imagery, comprising: at least one processor; data storage comprising a trained generator according to Claim 16; the at least one processor executing programming instructions configured to: receive raw image frames of underwater imagery; process the raw image frames using the trained generator to provide enhanced image frames; and using the enhanced image frames for at least one of: generating a visual display; controlling an underwater vehicle; and performing visual recognition.

18. Computer readable media comprising the trained generator according to Claim 16.

Images