A lightweight multi-scale image enhancement method and system for embedded real-time underwater video

Abstract

The application discloses a lightweight multi-scale image enhancement method and system for embedded real-time underwater video, which comprises the following steps: constructing and dividing an underwater image dataset; constructing a down-sampling module and an up-sampling module without a batch normalization layer; building a backbone network of a U-shaped encoding-decoding architecture based on the modules, introducing multi-scale skip connections and improved residual blocks into the network; constructing a loss function combining L2 loss and SSIM loss and training the backbone network; and deploying the trained network to an embedded device, sequentially performing preprocessing, backbone network enhancement and self-adaptive brightness adjustment post-processing on input underwater images, and outputting a final enhanced image. By removing the batch normalization layer and optimizing the network structure, the application greatly reduces the computational complexity, realizes real-time processing on the embedded platform while ensuring excellent enhancement quality, and effectively solves the problems of color deviation, detail blur and brightness attenuation of underwater images.

Description

Technical Field

[0001] This application relates to the field of computer vision and image processing technology, specifically to a lightweight multi-scale image enhancement method and system for embedded real-time underwater video. Background Technology

[0002] Underwater image enhancement technology is primarily used to improve image detail loss, insufficient brightness, color shift, and reduced contrast caused by light absorption, scattering, and suspended particles in underwater environments. This technology is widely used in underwater robots, unmanned underwater vehicles, underwater exploration and monitoring, and is a crucial foundation for achieving underwater visual perception.

[0003] Existing underwater image enhancement technologies mainly include physical model-based methods, image processing-based methods, and deep learning-based methods. Physical model-based methods typically construct underwater imaging models to invert the light propagation process; image processing-based methods often employ histogram equalization and color correction; and deep learning-based methods use neural networks to learn the mapping relationship between underwater images and enhanced images. With the development of embedded computing platforms and underwater video applications, deep learning-based enhancement methods have gradually become mainstream in recent years. However, existing technologies still face the following core technical bottlenecks:

[0004] (1) High computational complexity and difficult to deploy in real time: Existing high-performance deep learning models (such as UColor and Transformer-based methods) usually have complex network structures, large number of parameters, and high computational overhead, making it difficult to implement real-time processing on resource-constrained embedded devices (such as AUV or ROV).

[0005] (2) Artifacts and artificial traces: Some networks use batch normalization (BN) layers. When the statistical distribution of the training set and the test set differs greatly, artifacts are easily generated, as well as color fluctuations and inconsistent brightness.

[0006] (3) Imbalance between detail and color restoration: Traditional methods are difficult to balance between dehazing and color correction, which can easily lead to color distortion in the background area or oversaturation of the red channel.

[0007] (4) There is an overall brightness decay problem: Due to the superposition effect of nonlinear mapping in deep networks, existing methods are prone to cause an overall brightness decrease during the enhancement process, which affects the visibility and practicality of underwater videos in engineering applications.

[0008] Therefore, how to provide an underwater image enhancement solution that can guarantee enhancement quality, meet the real-time requirements of embedded platforms, and effectively avoid artifacts and brightness decay is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0009] The main objective of this application is to provide a lightweight multi-scale image enhancement method and system for embedded real-time underwater video, comprising the following steps:

[0010] Step S1: Construct an underwater image dataset and divide the dataset into a training set, a validation set, and a test set;

[0011] Step S2: Construct a downsampling module and an upsampling module based on convolution operations and nonlinear activation functions. The downsampling module and the upsampling module do not contain a batch normalization layer. Based on the downsampling module and the upsampling module, build an underwater image enhancement backbone network with a U-shaped encoder-decoder architecture.

[0012] Step S3: Construct a comprehensive loss function that combines the L2 loss function and the structural similarity loss function;

[0013] Step S4: Train the underwater image enhancement backbone network using the training set and the comprehensive loss function to obtain the trained underwater image enhancement backbone network, and deploy the trained backbone network into the target embedded device;

[0014] Step S5: Acquire real-time underwater video images, perform size padding and normalization on the underwater video images through preprocessing; input the preprocessed underwater video images into the trained backbone network for feature extraction and enhancement, and output a preliminary enhanced image;

[0015] Step S6: Input the preliminary enhanced image into the adaptive brightness adjustment post-processing module, and output the final enhanced underwater video image by compensating for brightness attenuation through brightness statistics and dynamic stretching.

[0016] In one embodiment, the step of constructing a downsampling module and an upsampling module based on convolution operations and nonlinear activation functions, wherein the downsampling module and the upsampling module do not contain a batch normalization layer; and building an underwater image enhancement backbone network with a U-shaped encoder-decoder architecture based on the downsampling module and the upsampling module, includes:

[0017] A downsampling module for spatial downsampling and an upsampling module for spatial upsampling are constructed by stacking convolutional layers and Leaky ReLU activation functions, and neither the downsampling module nor the upsampling module contains a batch normalization layer;

[0018] Multiple downsampling modules are cascaded to form an encoder, which is used to extract features from the input image layer by layer and reduce the spatial resolution; multiple upsampling modules are cascaded to form a decoder, which is used to reconstruct the deep features output by the encoder layer by layer and restore the image spatial resolution; and multi-scale skip connections are introduced between corresponding levels of the encoder and the decoder to fuse the shallow features extracted by the encoder with the deep features in the decoder.

[0019] An improved residual block is connected after each downsampling module of the encoder and after each upsampling module of the decoder; the improved residual block consists of two sets of cascaded convolutional layers and Leaky ReLU activation functions, and the input is directly added to the output through skip connections.

[0020] In one embodiment, the backbone network in step S2 further includes:

[0021] A lightweight feature calibration unit is interspersed among the transposed convolutional modules of the decoder. The calibration unit first compresses the channel dimension through 1×1 convolution, then extracts spatial features through 3×3 depth convolution, and finally restores the channel dimension through 1×1 convolution, thereby achieving lightweight calibration of the decoder features.

[0022] In one embodiment, the backbone network in step S2 further includes:

[0023] By initializing a scaling vector that matches the number of input feature channels and broadcasting it to the spatial dimension of the input features, and then multiplying it element-wise with the input features, adaptive control of the activation intensity of each channel can be achieved.

[0024] In one embodiment, the comprehensive loss function in step S4 consists of two parts: the L2 loss function and the structural similarity loss function.

[0025] The L2 loss function is used to calculate the Euclidean distance between the enhanced image and the reference image, constraining low-level visual features; the specific formula is as follows:

[0026]

[0027] U is the enhanced image, I is the real reference image, where H and W represent the height and width of the image, respectively, and (m, n) represent the spatial coordinates of the pixel;

[0028] The structural similarity loss function is used to calculate the structural similarity within a 13×13 local neighborhood, preserving the texture and structural features of the image. The formula is as follows:

[0029]

[0030] in and This represents the mean of a local area. and Represents local variance. Describing covariance, , It is the stability constant;

[0031] The final comprehensive loss function L is given by the following formula:

[0032]

[0033] Where ω1 and ω2 are weighting coefficients.

[0034] In one embodiment, the underwater image enhancement backbone network is trained using the training set and the comprehensive loss function to obtain the trained underwater image enhancement backbone network. The step of deploying the trained backbone network to a target embedded device further includes:

[0035] Configure the initial learning rate, batch size, and number of training rounds for network training, and use the Adam optimizer as the parameter update algorithm to iteratively optimize the weights of the underwater image enhancement backbone network during training.

[0036] The underwater images in the training set are input in pairs into the underwater image enhancement backbone network for forward propagation. The combined loss function value between the enhanced output image and the real reference image is calculated. Then, the gradient is calculated through the backpropagation algorithm, and the parameters of the backbone network are updated using the optimizer until the loss function converges or reaches the preset number of training rounds, so as to obtain the trained backbone network model.

[0037] The trained backbone network model parameter file is exported in an inference format suitable for the target embedded platform, loaded into the target embedded device, and the input / output interface is configured to enable the embedded device to acquire underwater video images in real time.

[0038] In one embodiment, the steps of acquiring real-time underwater video images, performing size padding and normalization on the underwater video images through preprocessing, inputting the preprocessed underwater video images into the trained backbone network for feature extraction and enhancement, and outputting a preliminary enhanced image further include:

[0039] The video acquisition unit deployed on the embedded device acquires continuous frame images of underwater video in real time, and inputs each frame of raw image data into the preprocessing module.

[0040] The preprocessing module obtains the original size information of the input image, performs edge-padding and expansion processing on the width and height of the image according to the preset network input size alignment rules, so that the expanded image size meets the specific multiple of the network input requirements, and uses a constant filling method to fill the expanded area with pixels, while normalizing the pixel values ​​to a preset numerical range, generating a preprocessed image that meets the backbone network input requirements.

[0041] The preprocessed image is input into the trained backbone network deployed on the embedded device. The encoder of the backbone network performs multi-scale feature extraction, the decoder performs resolution restoration, and the multi-scale skip connections perform feature fusion. Finally, the output layer of the backbone network generates and outputs the preliminarily enhanced underwater image.

[0042] In one embodiment, the step of inputting the preliminary enhanced image to the adaptive brightness adjustment post-processing module, compensating for brightness attenuation through brightness statistics and dynamic stretching, and outputting the final enhanced underwater video image further includes:

[0043] The system receives the initial enhanced image output from the backbone network and extracts the global brightness statistics of the image through a global average pooling operation, which serves as the basis for subsequent brightness adjustments.

[0044] The extracted global brightness statistics are input into a micro-network consisting of two 1×1 convolutional layers. After nonlinear mapping, brightness adjustment coefficients corresponding to the number of channels in the input image are generated. A learnable scaling factor is introduced to adaptively adjust the amplitude of the brightness adjustment coefficients in order to control the intensity of brightness adjustment.

[0045] The generated brightness adjustment coefficients are extended to the same spatial dimension as the input image through a broadcast mechanism, and then multiplied or weighted with the preliminary enhanced image element-wise to achieve dynamic stretching compensation of the image brightness, correct the brightness attenuation caused by the nonlinear mapping of the network, and finally output an underwater video image with enhanced visual comfort.

[0046] A lightweight multi-scale image enhancement system for embedded real-time underwater video, the system comprising:

[0047] The dataset building module is used to build and segment underwater image datasets;

[0048] The network construction module is used to build the downsampling module, upsampling module, and underwater image enhancement backbone network without a batch normalization layer.

[0049] The training module is used to train the backbone network using the comprehensive loss function;

[0050] The preprocessing module, deployed on the embedded device, is used to perform size filling and normalization processing on the real-time acquired underwater images.

[0051] The lightweight enhanced backbone network module is deployed on an embedded device to load the trained backbone network model and enhance the preprocessed image to output a preliminary enhanced image.

[0052] The adaptive brightness adjustment post-processing module, deployed on the embedded device, is used to perform brightness compensation on the initially enhanced image and output the final enhanced image.

[0053] Therefore, this application has the following beneficial effects:

[0054] First, lightweight and high efficiency. This application significantly reduces model complexity by removing the computationally expensive batch normalization layer and using an improved residual block to construct the backbone network. Experiments show that at a resolution of 640×480, the method achieves a processing speed of 36.23 FPS, with an average processing time of only 28.52 ms per frame, meeting the real-time processing requirements of embedded platforms such as underwater robots and solving the bottleneck of existing high-performance models being difficult to deploy on resource-constrained devices.

[0055] Second, superior enhancement quality. This application fuses shallow details and deep semantic features through multi-scale skip connections, introduces a lightweight feature calibration unit to enhance detail perception, and employs a weighted fusion of L2 loss and SSIM loss for training, achieving a balance between pixel accuracy and structure preservation. Quantitative evaluation on the UIEB dataset shows a PSNR of 18.57 dB, an SSIM of 0.91, and a UIQM of 0.79, significantly outperforming existing mainstream methods in overall performance.

[0056] Third, improved visual comfort. This application addresses the brightness attenuation problem caused by nonlinear mapping in deep networks by designing an adaptive brightness adjustment post-processing module. This module dynamically compensates for brightness loss through global brightness statistics and a learnable scaling factor. Experimental data shows that the average brightness of the processed image increases from 109.45 to 122.88, more closely resembling the brightness level of a real scene. The output image exhibits more natural colors, resulting in a significantly improved visual effect.

[0057] Fourth, it exhibits strong robustness. This application uses a composite dataset covering multiple degradation types for training and is tested in real underwater environments with different turbidity and lighting conditions. It demonstrates good generalization ability and enhanced stability, making it suitable for complex and varied real-world underwater operation scenarios. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart of a lightweight multi-scale image enhancement method and system for embedded real-time underwater video.

[0060] Figure 2 This is a lightweight multi-scale image enhancement method and system for embedded real-time underwater video, and its overall network architecture diagram.

[0061] Figure 3 This is a schematic diagram of an improved residual block structure for a lightweight multi-scale image enhancement method and system for embedded real-time underwater video. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0064] To address the core shortcomings of existing underwater image enhancement technologies in embedded real-time applications, such as high computational complexity, susceptibility to artifacts, imbalance between detail and color restoration, and overall brightness degradation, this application provides a lightweight multi-scale image enhancement method and system for embedded real-time underwater video, aiming to achieve synergistic optimization of enhancement quality and processing efficiency. Existing high-performance deep learning models (such as UColor and Transformer-based methods) typically have complex network structures, large numbers of parameters, and widely employ batch normalization layers, resulting in high computational overhead and slow inference speed, making it difficult to achieve real-time processing on resource-constrained embedded platforms. Furthermore, BN layers are prone to introducing artifacts and color fluctuations when there are significant differences in the distribution of the training and test sets. In addition, existing methods struggle to achieve a balance between dehazing and color correction, easily leading to color distortion in background areas or oversaturation of the red channel. Moreover, the nonlinear mapping superposition effect of deep networks often causes an overall decrease in brightness, affecting engineering practicality.

[0065] To address the aforementioned technical issues, this application adopts the following technical solution: First, an underwater image dataset containing various degradation types is constructed and divided into training, validation, and test sets. Second, a downsampling and upsampling module without batch normalization layers is constructed. Based on this, a lightweight backbone network with a U-shaped encoder-decoder architecture is built. An improved residual block is introduced after each layer of the encoder and decoder. This residual block consists of two sets of convolutional layers and a Leaky ReLU activation function. Skip connections are used to preserve the gradient propagation advantage, while BN layers are removed to reduce computational complexity. Lightweight feature calibration units are interspersed between the transposed convolutional modules of the decoder. Lightweight feature calibration is achieved by compressing channels with 1×1 convolutions, extracting spatial features with 3×3 depthwise convolutions, and then restoring channels with 1×1 convolutions. Multi-scale skip connections are also introduced to fuse shallow detail features of the encoder with deep semantic features of the decoder, reducing information loss. Then, a comprehensive loss function fusing L2 loss and SSIM loss is constructed to train the backbone network, balancing pixel accuracy and structure preservation. After training, the model is deployed to an embedded device. Finally, during the inference phase, the real-time acquired underwater images are preprocessed, the backbone network is enhanced, and then input into an adaptive brightness adjustment post-processing module. The brightness attenuation is dynamically compensated through global brightness statistics and a learnable scaling factor, and the final enhanced image is output.

[0066] Through the above solution, this application achieves a real-time processing speed of 36.23 FPS at a resolution of 640×480, with a PSNR of 18.57 dB and an SSIM of 0.91. It effectively solves the problems of high computational complexity, artifacts, color distortion, and brightness attenuation in existing technologies, and provides an underwater image enhancement solution that combines high performance and high quality for embedded vision systems such as underwater robots.

[0067] This application provides a lightweight multi-scale image enhancement method and system for embedded real-time underwater video, including steps S1-S6, as described above. Figure 1 , Figure 1 This is a flowchart of a lightweight, multi-scale image enhancement method for embedded real-time underwater video.

[0068] Step S1: Construct an underwater image dataset and divide the dataset into a training set, a validation set, and a test set;

[0069] Step S2: Construct a downsampling module and an upsampling module based on convolution operations and nonlinear activation functions. The downsampling module and the upsampling module do not contain a batch normalization layer. Based on the downsampling module and the upsampling module, build an underwater image enhancement backbone network with a U-shaped encoder-decoder architecture.

[0070] Step S3: Construct a comprehensive loss function that combines the L2 loss function and the structural similarity loss function;

[0071] Step S4: Train the underwater image enhancement backbone network using the training set and the comprehensive loss function to obtain the trained underwater image enhancement backbone network, and deploy the trained backbone network into the target embedded device;

[0072] Step S5: Acquire real-time underwater video images, perform size padding and normalization on the underwater video images through preprocessing; input the preprocessed underwater video images into the trained backbone network for feature extraction and enhancement, and output a preliminary enhanced image;

[0073] Step S6: Input the preliminary enhanced image into the adaptive brightness adjustment post-processing module, and output the final enhanced underwater video image by compensating for brightness attenuation through brightness statistics and dynamic stretching.

[0074] Specifically, in this embodiment, this application provides a lightweight multi-scale image enhancement method for embedded real-time underwater video, including steps S1 to S6, referring to... Figure 1 .

[0075] Step S1: Construct an underwater image dataset and divide it into training, validation, and test sets. To ensure the model's generalization ability in different underwater environments, this embodiment selects multiple public datasets and a self-built dataset to form a complementary sample library. Public datasets UIEB, EUVP, RUIE, and SUID, as well as the self-built O200 dataset, are selected to form a complementary and diverse sample library. The UIEB (Underwater Image Enhancement Benchmark Dataset) dataset contains 950 real underwater scene images, of which 890 have expert-annotated reference images, covering different water depths, turbidity, and lighting combinations. The remaining 60 images serve as a challenge set for extreme scenario testing. The EUVP (Enhancing Underwater VisualPerception Dataset) dataset provides 3,700 pairs of real underwater images (degraded-clear) captured by 7 types of underwater cameras. The RUIE (Real-world Underwater Image Enhancement) dataset contains diverse marine biological samples, covering the degradation characteristics of typical underwater scenes. The SUID (Synthetic Underwater Image Dataset) provides 900 physically based images, supplemented with samples of various degradation types and turbidity levels.

[0076] The training set consists of 90% of the samples from the UIEB dataset (855 images), 1000 randomly selected reference images from the EUVP dataset, and the complete SUID dataset (900 images), totaling approximately 2755 images, ensuring a balanced distribution of degradation types. The validation and test sets are stratified samples from the remaining data—the validation set includes the remaining 10% of the UIEB samples (95 images), 100 images from the RUIE dataset, and 100 images from the EUVP dataset; the test set includes the remaining 10% of the UIEB samples (95 images), 200 images from the RUIE dataset, 200 images from the self-built O200 dataset, and 300 images from the EUVP dataset (without overlap with the training set), totaling approximately 795 images. This stratified sampling strategy maintains a balanced number of samples across different water depths while ensuring consistent distribution of various degradation types, effectively preventing the model from overfitting to specific scenarios and thus improving the network's generalization ability and robustness in unknown underwater environments. This stratified sampling strategy ensures a balanced distribution of samples across different water depths and degradation types, preventing the model from overfitting to specific scenarios.

[0077] Step S2: Construct downsampling and upsampling modules based on convolutional operations and nonlinear activation functions. These modules do not include batch normalization layers. Based on these modules, an underwater image enhancement backbone network with a U-shaped encoder-decoder architecture is built. This step further includes: First, constructing a downsampling module for spatial downsampling and an upsampling module for spatial upsampling using stacked convolutional layers and Leaky ReLU activation functions, neither of which contains a batch normalization layer. Second, cascading multiple downsampling modules sequentially to form an encoder, and cascading multiple upsampling modules sequentially to form a decoder, introducing multi-scale skip connections between corresponding layers of the encoder and decoder. Finally, connecting an improved residual block after each downsampling module of the encoder and after each upsampling module of the decoder. This residual block consists of two sets of cascaded convolutional layers and Leaky ReLU activation functions, directly adding the input to the output through skip connections, enhancing feature representation while maintaining lightweight design. In addition, lightweight feature calibration units are interspersed between the transposed convolution modules of the decoder. Feature calibration is achieved by using a 1×1 convolution compression channel, a 3×3 depth convolution to extract spatial features, and then a 1×1 convolution to restore the channel.

[0078] Step S3: Construct a comprehensive loss function that integrates the L2 loss function and the structural similarity loss function. The pixel-level L2 loss function calculates the Euclidean distance between the enhanced image and the real reference image, which is used to constrain low-level visual features. The structural similarity loss function calculates the similarity in three dimensions—brightness, contrast, and structure—within the local neighborhood to preserve the texture and structural features of the image. In this embodiment, the preferred weighting coefficients are ω1=0.4 and ω2=0.6 to emphasize the preservation of structural details and balance pixel reconstruction accuracy with structural consistency.

[0079] Step S4: Train the underwater image enhancement backbone network using the training set and the comprehensive loss function to obtain the trained underwater image enhancement backbone network, and deploy the trained backbone network to the target embedded device. This step further includes: First, configuring the training hyperparameters. In this embodiment, the Adam optimizer (β1=0.5, β2=0.999) is used, and the initial learning rate is set to 1×10⁻⁶. -4 The batch size is 8, and the training rounds are 200. Next, underwater images in the training set are cropped to 128×128 pixels and normalized to the [0,1] range. These images are then input into the backbone network for forward propagation to calculate the comprehensive loss value. The network parameters are updated through backpropagation, and training is iterated until the loss converges. Finally, the trained model parameter file is exported to an inference format suitable for the target embedded platform. This is then loaded into the inference engine of embedded computing platforms such as the NVIDIA Jetson series, and the input / output interfaces are configured to enable the device to have real-time forward inference enhancement capabilities.

[0080] Step S5: Acquire real-time underwater video images, and perform size padding and normalization on the underwater video images through preprocessing; input the preprocessed underwater video images into the trained backbone network for feature extraction and enhancement, and output a preliminary enhanced image. This step further includes: First, acquiring continuous frame images of underwater video in real time through a video acquisition unit deployed on an embedded device; second, the preprocessing module acquires the original size information of the input image, and performs edge padding and expansion processing on the width and height of the image according to the preset network input size alignment rules (such as multiples of 16), using constant padding to fill the expanded area with pixels, and normalizing the pixel values ​​to the range of [0,1]; finally, inputting the preprocessed image into the deployed backbone network, performing multi-scale feature extraction through the encoder, resolution restoration through the decoder, and feature fusion through multi-scale skip connections, and generating a preliminary enhanced image from the network output layer. Experiments show that at a resolution of 640×480, the processing speed of this method reaches 36.23 FPS, and the average processing time per frame is only 28.52 ms.

[0081] Step S6: The initially enhanced image is input into the adaptive brightness adjustment post-processing module. Brightness attenuation is compensated through brightness statistics and dynamic stretching, resulting in the final enhanced underwater video image. This step further includes: First, residual map generation. At the decoder end, the network uses the Tanh activation function to constrain the output features, generating a feature residual map reflecting the local brightness gain of the image. The Tanh function maps the output values ​​to the [-1,1] interval, allowing the residual map to represent positive (brightening) or negative (darkening) brightness adjustments. Through this design, the network possesses the ability to statistically evaluate complex illumination distributions and can generate differentiated compensation coefficients for the degree of brightness attenuation in different regions.

[0082] Secondly, global residual fusion achieves adaptive brightness compensation. The generated feature residual map is globally element-wise superimposed with the original input image, i.e., enhanced image = original input + feature residual map. This operation directly compensates for the energy attenuation caused by multi-layer nonlinear mapping during deep processing by utilizing the brightness offset learned by the deep network. Unlike traditional post-processing methods, this method couples the brightness compensation process to the end of the network, forming an end-to-end learning path, enabling the compensation coefficients to adaptively adjust according to the brightness distribution characteristics of the input image.

[0083] Finally, end-to-end brightness-balanced output is achieved. Through the aforementioned global residual path, the enhanced image retains its original structural features while dynamically adapting and correcting the brightness distribution characteristics of the input image. Experiments show that this mechanism effectively solves the inherent brightness attenuation problem of deep networks, restoring the average brightness of the output image from 109.45 before processing to 122.88, which is closer to the 121.27 of the real scene, achieving an enhancement effect with natural brightness, clear details, and realistic colors.

[0084] like Figure 2 The diagram shows the overall network architecture of the lightweight multi-scale image enhancement system of this application, detailing the connections between the encoder, decoder, skip connections, and post-processing module. First, the raw underwater image is received through preprocessing, its size adjusted and normalized to a standard 3×256×256 input tensor, providing a unified input format for subsequent network processing. The preprocessed image is then input into the backbone network, which employs a U-shaped encoder-decoder architecture.

[0085] The encoder consists of multiple cascaded downsampling modules that extract multi-scale features layer by layer: the input image sequentially passes through downsampling modules with 64 channels at 128×128 resolution, 128 channels at 64×64 resolution, and 256 channels at 32×32 resolution, ultimately yielding a 512-channel, 16×16 resolution deep feature map. Each downsampling module is followed by an improved residual block to enhance feature representation and maintain gradient flow.

[0086] The decoder consists of multiple cascaded upsampling modules that progressively restore image resolution: starting with 512-channel 16×16 features, it upsamples layer by layer to restore resolutions of 256-channel 32×32, 128-channel 64×64, and 64-channel 128×128. Each upsampling module is also followed by an improved residual block, and lightweight feature calibration units are interspersed between each transposed convolution module to enhance detail perception.

[0087] Multi-scale skip connections directly transmit the shallow features (128×128, 64×64, 32×32 resolution) output from each level of the encoder to the corresponding level of the decoder. Through feature fusion operations, they are fused with the upsampled deep features in the channel dimension, effectively preserving details such as edges and textures.

[0088] At the end of the decoder, the network generates a feature residual map with the same resolution as the input image. This residual map is constrained to the [-1,1] interval by the Tanh activation function, reflecting the local brightness gain.

[0089] Finally, the global residual adaptive brightness adjustment module element-wise overlays the feature residual map with the original input image to achieve adaptive brightness compensation, outputting a final enhanced 3×256×256 image. The entire architecture, through end-to-end design, achieves an organic unity of feature extraction, detail preservation, and brightness correction.

[0090] In one embodiment, the step of constructing a downsampling module and an upsampling module based on convolution operations and nonlinear activation functions, wherein the downsampling module and the upsampling module do not contain a batch normalization layer; and building an underwater image enhancement backbone network with a U-shaped encoder-decoder architecture based on the downsampling module and the upsampling module, includes:

[0091] A downsampling module for spatial downsampling and an upsampling module for spatial upsampling are constructed by stacking convolutional layers and Leaky ReLU activation functions, and neither the downsampling module nor the upsampling module contains a batch normalization layer;

[0092] Multiple downsampling modules are cascaded to form an encoder, which is used to extract features from the input image layer by layer and reduce the spatial resolution; multiple upsampling modules are cascaded to form a decoder, which is used to reconstruct the deep features output by the encoder layer by layer and restore the image spatial resolution; and multi-scale skip connections are introduced between corresponding levels of the encoder and the decoder to fuse the shallow features extracted by the encoder with the deep features in the decoder.

[0093] An improved residual block is connected after each downsampling module of the encoder and after each upsampling module of the decoder; the improved residual block consists of two sets of cascaded convolutional layers and Leaky ReLU activation functions, and the input is directly added to the output through skip connections.

[0094] Specifically, in this embodiment, step S2 includes the following:

[0095] First, this embodiment employs a stacked approach of convolutional layers and Leaky ReLU activation functions to construct a downsampling module for spatial downsampling and an upsampling module for spatial upsampling. The downsampling module uses a 3×3 convolution with a stride of 2 to halve the spatial size, while the upsampling module uses a 3×3 transposed convolution with a stride of 2 to double the spatial size. Unlike conventional network designs, neither the downsampling nor upsampling modules in this embodiment contain batch normalization (BN) layers. This design choice aims to reduce the computational overhead during forward inference and avoid artifacts and color fluctuations introduced by BN layers when there are significant differences in the distributions of the training and test sets, thereby improving the model's efficiency and enhancing its stability on embedded platforms.

[0096] Secondly, an encoder-decoder backbone architecture is constructed. Multiple downsampling modules are cascaded to form an encoder, used for layer-by-layer feature extraction from the input image and progressively reducing spatial resolution. In this embodiment, the encoder includes three levels of downsampling, with the number of channels increasing progressively from 64 to 128, 256, and 512. Multiple upsampling modules are cascaded to form a decoder, used for layer-by-layer reconstruction of the deep features output by the encoder and restoring the image's spatial resolution. In this embodiment, the decoder includes three levels of upsampling, with the number of channels increasing progressively from 512 to 256, 128, and 64. Multi-scale skip connections are introduced between corresponding layers of the encoder and decoder to fuse the shallow detail features (such as edges and textures) extracted by the encoder with the deep semantic features in the decoder, effectively reducing information loss during feature transfer and improving the network's ability to restore details in underwater images.

[0097] Finally, an improved residual block is introduced. An improved residual block is connected after each downsampling module of the encoder and after each upsampling module of the decoder. For example... Figure 3As shown, this improved residual block consists of two sets of cascaded convolutional layers and a Leaky ReLU activation function, with the input directly added to the output via skip connections. Compared to traditional residual blocks, this embodiment removes the batch normalization layer, retaining only the stacking of convolutional and activation functions. By using skip connections, it preserves the gradient propagation advantage of the residual structure, ensuring feature representation capability while significantly reducing computational complexity, thus balancing network lightweightness and information flow efficiency.

[0098] In one embodiment, the backbone network in step S2 further includes:

[0099] A lightweight feature calibration unit is interspersed among the transposed convolutional modules of the decoder. The calibration unit first compresses the channel dimension through 1×1 convolution, then extracts spatial features through 3×3 depth convolution, and finally restores the channel dimension through 1×1 convolution, thereby achieving lightweight calibration of the decoder features.

[0100] Specifically, in this embodiment, the backbone network in step S2 further includes the design and deployment of a lightweight feature calibration unit, as detailed below:

[0101] To improve the decoder's ability to recover detailed features without excessively increasing computational burden, this embodiment introduces a lightweight feature calibration unit interspersed between the transposed convolutional modules of the decoder. The design concept of this unit is to adaptively calibrate the feature map through a compact network structure, enhancing the network's ability to perceive details such as edges and textures in underwater images, while maintaining the overall lightweight nature of the network. The lightweight feature calibration unit adopts a three-stage structure of "compression-convolution-recovery". First, the input feature map passes through a 1×1 convolutional layer, which compresses the channel dimension of the feature map, typically by 1 / 4 or 1 / 2. This operation aims to reduce the computational cost of subsequent depthwise convolutions, achieving lightweight feature processing. Second, the compressed feature map is input into a 3×3 depthwise convolutional layer. Depthwise convolutions are characterized by each kernel processing only a single input channel, significantly reducing the number of parameters and computational overhead compared to standard convolutions, while effectively extracting spatial feature information and capturing local structural details of the image. Finally, the feature map output from the depthwise convolution is input into another 1×1 convolutional layer. The function of this layer is to restore the channel dimension to the original input size, ensuring that the feature map can be fused or processed with other features in the main path.

[0102] Through the above structural design, the lightweight feature calibration unit can achieve fine-grained feature calibration with extremely low computational cost. To further enhance detail perception capabilities, this embodiment uses 3×3 depthwise convolutions with dilation in some network layers instead of ordinary depthwise convolutions. By expanding the receptive field, it captures contextual information over a wider range, thereby enhancing the ability to recover details in blurred areas of underwater images. This unit is interspersed among the transposed convolutional modules at each stage of the decoder, effectively improving the stability of feature flow and the response strength of detailed features.

[0103] In one embodiment, the backbone network in step S2 further includes:

[0104] By initializing a scaling vector that matches the number of input feature channels and broadcasting it to the spatial dimension of the input features, and then multiplying it element-wise with the input features, adaptive control of the activation intensity of each channel can be achieved.

[0105] Specifically, in this embodiment, the backbone network in step S2 further includes a channel scaling design, the specific design of which is as follows:

[0106] To enhance the network's adaptive adjustment capability for feature channels, this embodiment introduces a channel scaling design in the backbone network. The core idea of ​​this module is to dynamically adjust the activation intensity of each channel in the feature map using a learnable channel-level scaling factor, thereby enhancing the network's feature representation flexibility. Specifically, firstly, for an input feature map of size C×H×W (where C is the number of channels, H and W are the spatial height and width), a learnable scaling vector is initialized. The dimension of this vector is consistent with the number of channels C of the input feature map, denoted as V = [v1, v2, ..., v c Each element in this scaling vector corresponds to a scaling factor for a feature channel. The initial value is usually set to 1, and it is automatically optimized and updated through backpropagation during network training.

[0107] Secondly, in order to apply the channel-level scaling vector to each pixel in the spatial dimension, the module adopts a broadcast mechanism to expand the scaling vector V of dimension C×1×1 in the spatial dimension so that it is consistent with the spatial size H×W of the input feature map, thus obtaining a scaling parameter matrix P of dimension C×H×W.

[0108] Finally, the expanded scaling parameter matrix P is multiplied element-wise with the input feature map X, resulting in the output feature map Y = P⊙X, where ⊙ represents the element-wise multiplication operation. Through this design, the network can adaptively enhance or suppress the feature response intensity of specific channels based on the learned data distribution, giving higher activation weights to important feature channels while suppressing redundant or noisy channels.

[0109] In one embodiment, the comprehensive loss function in step S4 consists of two parts: the L2 loss function and the structural similarity loss function.

[0110] The L2 loss function is used to calculate the Euclidean distance between the enhanced image and the reference image, constraining low-level visual features; the specific formula is as follows:

[0111]

[0112] U is the enhanced image, I is the real reference image, where H and W represent the height and width of the image, respectively, and (m, n) represent the spatial coordinates of the pixel;

[0113] The structural similarity loss function is used to calculate the structural similarity within a 13×13 local neighborhood, preserving the texture and structural features of the image. The formula is as follows:

[0114]

[0115] in and This represents the mean of a local area. and Represents local variance. Describing covariance, , It is the stability constant;

[0116] The final comprehensive loss function L is given by the following formula:

[0117]

[0118] Where ω1 and ω2 are weighting coefficients.

[0119] Specifically, in this embodiment, the comprehensive loss function in step S4 consists of two parts: the L2 loss function and the structural similarity loss function, as detailed below:

[0120] The L2 loss function is used to calculate the pixel-level Euclidean distance between the enhanced image and the reference image. By constraining low-level visual features, it ensures that the enhanced image approximates the true reference image in pixel values. Its mathematical expression is:

[0121]

[0122] Where U represents the enhanced image, I represents the ground truth reference image, H and W represent the height and width of the image, respectively, and (m, n) represent the spatial coordinates of the pixel. The L2 loss, by summing the squared errors at each pixel location, imposes a stronger penalty on larger pixel differences, thereby guiding the network to generate enhancement results with high pixel accuracy.

[0123] SSIM is a structural similarity loss function used to evaluate the similarity between the enhanced image and the reference image in three dimensions: brightness, contrast, and structure, within a local neighborhood, aiming to preserve the texture details and structural integrity of the image. This embodiment calculates the SSIM value within a 13×13 local window, using the following formula:

[0124]

[0125] in and This represents the mean of a local area. and Represents local variance. Describing covariance, , It is a stability constant.

[0126] The final comprehensive loss function L is given by the following formula:

[0127]

[0128] Where ω1 and ω2 are weighting coefficients. In this embodiment, the preferred weighting coefficients are ω1=0.4 and ω2=0.6, to emphasize the preservation of structural details, balance pixel reconstruction accuracy and structural consistency, and assign slightly higher weights to structural similarity. This guides the network to focus more on restoring the texture details and edge structure of underwater images while ensuring pixel accuracy, avoiding structural distortion caused by focusing solely on pixel errors. Through optimized training using this comprehensive loss function, the network can effectively maintain the natural structure and visual quality of the image while removing color cast and increasing brightness.

[0129] In one embodiment, the underwater image enhancement backbone network is trained using the training set and the comprehensive loss function to obtain the trained underwater image enhancement backbone network. The step of deploying the trained backbone network to a target embedded device further includes:

[0130] Configure the initial learning rate, batch size, and number of training rounds for network training, and use the Adam optimizer as the parameter update algorithm to iteratively optimize the weights of the underwater image enhancement backbone network during training.

[0131] The underwater images in the training set are input in pairs into the underwater image enhancement backbone network for forward propagation. The combined loss function value between the enhanced output image and the real reference image is calculated. Then, the gradient is calculated through the backpropagation algorithm, and the parameters of the backbone network are updated using the optimizer until the loss function converges or reaches the preset number of training rounds, so as to obtain the trained backbone network model.

[0132] The trained backbone network model parameter file is exported in an inference format suitable for the target embedded platform, loaded into the target embedded device, and the input / output interface is configured to enable the embedded device to acquire underwater video images in real time.

[0133] Specifically, in this embodiment, step S4 includes the following training and deployment process:

[0134] First, configure the training hyperparameters and optimizer. In this embodiment, hyperparameters are initialized before training begins. Specifically, the Adam optimizer is used as the parameter update algorithm, and its decay coefficients β1=0.5 and β2=0.999 are set to balance gradient estimation bias correction and convergence stability. The initial learning rate is set to 1×10⁻⁶. -4 This value strikes a balance between convergence speed and training stability. The batch size is set to 8, and the number of training epochs is set to 200 to ensure that the network has sufficient iterations to learn the mapping relationship between underwater image degradation and enhancement. Before training, all input images are cropped to a fixed size of 128×128 pixels, and the pixel values ​​are normalized to the [0,1] interval to accelerate model convergence.

[0135] Next, iterative training and parameter updates are performed. Underwater degraded images and their corresponding ground truth reference images are paired and input into the backbone network. For each batch of data, forward propagation is first performed. After the network outputs an enhanced image, the combined loss function L between the enhanced image and the ground truth reference image is calculated. Then, the gradient of the loss function with respect to the parameters of each network layer is calculated using the backpropagation algorithm, and the Adam optimizer is used to update all learnable parameters of the backbone network (including the encoder, decoder, improved residual block, lightweight feature calibration unit, and channel scaling module). After each training round, the model performance is evaluated on the validation set to monitor for overfitting. This process is repeated until the loss function converges or the preset 200 training rounds are reached. Finally, the weights of the trained backbone network model are saved.

[0136] Finally, model export and embedded deployment. The trained model parameter file is exported to an inference format suitable for the target embedded platform. For example, for NVIDIA Jetson platforms, it can be exported as a TensorRT engine file, and for ARM architecture platforms, it can be converted to ONNX Runtime format. The optimized model file is loaded into the inference engine of the embedded device, and standard input / output interfaces are configured, enabling the device to receive underwater video image frames in real time, call the model for forward inference enhancement, and output the enhancement results to a display terminal or subsequent processing module, achieving end-to-end real-time underwater video enhancement functionality.

[0137] In one embodiment, the steps of acquiring real-time underwater video images, performing size padding and normalization on the underwater video images through preprocessing, inputting the preprocessed underwater video images into the trained backbone network for feature extraction and enhancement, and outputting a preliminary enhanced image further include:

[0138] The video acquisition unit deployed on the embedded device acquires continuous frame images of underwater video in real time, and inputs each frame of raw image data into the preprocessing module.

[0139] The preprocessing module obtains the original size information of the input image, performs edge-padding and expansion processing on the width and height of the image according to the preset network input size alignment rules, so that the expanded image size meets the specific multiple of the network input requirements, and uses a constant filling method to fill the expanded area with pixels, while normalizing the pixel values ​​to a preset numerical range, generating a preprocessed image that meets the backbone network input requirements.

[0140] The preprocessed image is input into the trained backbone network deployed on the embedded device. The encoder of the backbone network performs multi-scale feature extraction, the decoder performs resolution restoration, and the multi-scale skip connections perform feature fusion. Finally, the output layer of the backbone network generates and outputs the preliminarily enhanced underwater image.

[0141] Specifically, in this embodiment, step S5 includes the following real-time inference process:

[0142] First, real-time image acquisition and input. A video acquisition unit deployed on an embedded device, such as an underwater camera connected to an AUV or ROV, acquires continuous video frames of the underwater environment in real time. The video acquisition unit acquires image data at a fixed frame rate (e.g., 30 FPS) and transmits each raw frame directly to the input buffer of the preprocessing module in a preset data format (e.g., BGR or RGB), ensuring that image acquisition and subsequent processing pipelines run in parallel, minimizing end-to-end latency.

[0143] Secondly, image preprocessing alignment. After receiving the original image frame, the preprocessing module first reads the original height H and width W of the image. Since the backbone network in this embodiment requires that the width and height of the input image be multiples of 16 (i.e., integer multiples of the network downsampling factor), the preprocessing module calculates the size to be filled according to this preset alignment rule. Specifically, it calculates the target height H' = ceil(H / 16)×16 and the target width W' = ceil(W / 16)×16, and determines the number of pixel rows and columns to be filled at the bottom and right. Subsequently, the preprocessing module uses a constant filling method, usually filling the extended area with pixel values ​​of 0 (corresponding to black). This filling method has minimal computational overhead and does not introduce additional noise. After completing the size filling, the module linearly maps the pixel values ​​of the image from the original integer range of [0,255] to the floating-point range of [0,1]. This normalization operation helps maintain the numerical stability of network inference and keeps the distribution of input data consistent with the training phase. After the above processing, the original image is converted into a preprocessed image tensor with a size of H'×W'×3 and a value range of [0,1], which fully meets the input requirements of the backbone network.

[0144] Finally, the backbone network is enhanced through forward inference. The preprocessed image tensor is input into the trained backbone network deployed on an embedded device. This tensor first enters the encoder, passing through initial convolutional layers and a three-level downsampling module (each level followed by an improved residual block), extracting multi-scale features layer by layer, gradually reducing the spatial resolution to 1 / 8 of the original input, and expanding the number of channels to 512. The deep features are then input into the decoder, passing through a three-level transposed convolutional module (each level followed by an improved residual block, with lightweight feature calibration units interspersed between modules) to restore the resolution layer by layer. Simultaneously, through multi-scale skip connections, the shallow detail features of the corresponding levels of the encoder are fused with the decoder features at the channel dimension, effectively compensating for the edge and texture information lost during downsampling. Finally, the feature map is passed through the transposed convolution and Tanh activation function of the output layer to generate and output a preliminary enhanced image with the same spatial size as the input image. At a resolution of 640×480, the entire forward inference process takes only 28.52 ms, corresponding to a frame rate of 36.23 FPS, fully meeting the requirements of real-time video processing.

[0145] In one embodiment, the step of inputting the preliminary enhanced image to the adaptive brightness adjustment post-processing module, compensating for brightness attenuation through brightness statistics and dynamic stretching, and outputting the final enhanced underwater video image further includes:

[0146] The system receives the initial enhanced image output from the backbone network and extracts the global brightness statistics of the image through a global average pooling operation, which serves as the basis for subsequent brightness adjustments.

[0147] The extracted global brightness statistics are input into a micro-network consisting of two 1×1 convolutional layers. After nonlinear mapping, brightness adjustment coefficients corresponding to the number of channels in the input image are generated. A learnable scaling factor is introduced to adaptively adjust the amplitude of the brightness adjustment coefficients in order to control the intensity of brightness adjustment.

[0148] The generated brightness adjustment coefficients are extended to the same spatial dimension as the input image through a broadcast mechanism, and then multiplied or weighted with the preliminary enhanced image element-wise to achieve dynamic stretching compensation of the image brightness, correct the brightness attenuation caused by the nonlinear mapping of the network, and finally output an underwater video image with enhanced visual comfort.

[0149] Specifically, in this embodiment, step S6 includes the following adaptive brightness adjustment process:

[0150] First, global brightness feature extraction is performed. The adaptive brightness adjustment post-processing module receives the preliminary enhanced image output from the backbone network as input. This module first extracts global brightness statistics from the input image, specifically using global average pooling. For an input feature map of size C×H×W (where C is the number of channels, and H and W are the spatial height and width), global average pooling calculates the pixel mean of each channel along the spatial dimension, outputting a statistical vector of dimension C×1×1. This operation effectively captures the overall brightness distribution trend of the image, reflecting the brightness of each color channel, while requiring minimal computation, making it suitable for the real-time processing needs of embedded platforms. Through this step, the module obtains the global brightness baseline information required for subsequent brightness adjustment.

[0151] Secondly, adaptive adjustment coefficients are generated. The extracted global brightness statistics are input into a lightweight micro-network consisting of two 1×1 convolutional layers. The first 1×1 convolutional layer performs channel compression and nonlinear transformation on the input statistical vector, reducing dimensionality to decrease computation and introducing nonlinear mapping capability; the second 1×1 convolutional layer maps the features back to the same dimension as the number of channels C of the input image, generating initial brightness adjustment coefficients. Each channel value of this coefficient corresponds to the brightness compensation intensity of the corresponding channel in the original image. To further enhance adaptability, this embodiment introduces a learnable scaling factor s into the micro-network. This factor is initialized to 1 and automatically optimized through backpropagation during network training. The generated initial brightness adjustment coefficients are multiplied by this learnable scaling factor s to achieve dynamic adjustment of the compensation amplitude, enabling the module to adaptively adjust the brightness compensation strength according to different types of underwater images, avoiding overexposure or underexposure.

[0152] Finally, brightness compensation and image output are performed. The brightness adjustment coefficients, after scaling factor adjustment, are expanded spatially through a broadcast mechanism, increasing their size from C×1×1 to C×H×W, consistent with the input image, resulting in a complete brightness compensation parameter matrix. This compensation parameter matrix is ​​then multiplied element-wise with the initial enhanced image; each pixel value in the output image equals the input pixel value multiplied by the corresponding compensation coefficient. This operation is equivalent to linearly stretching the image's dynamic range, effectively correcting the overall brightness attenuation caused by the multi-layer nonlinear mapping of the deep network. To further preserve the original color balance, this embodiment can also employ a weighted fusion method, fusing the compensated image with the original initial enhanced image at a certain ratio, improving brightness while avoiding color distortion. Experimental data shows that after processing by this module, the average image brightness increases from 109.45 to 122.88, closer to the real-world 121.27, significantly improving visual comfort. Ultimately, the module outputs an enhanced underwater video image with natural brightness, clear details, and realistic colors, which can be directly used for subsequent target detection, tracking, and other visual tasks, or presented to operators.

[0153] A lightweight multi-scale image enhancement system for embedded real-time underwater video, the system comprising:

[0154] The dataset building module is used to build and segment underwater image datasets;

[0155] The network construction module is used to build the downsampling module, upsampling module, and underwater image enhancement backbone network without a batch normalization layer.

[0156] The training module is used to train the backbone network using the comprehensive loss function;

[0157] The preprocessing module, deployed on the embedded device, is used to perform size filling and normalization processing on the real-time acquired underwater images.

[0158] The lightweight enhanced backbone network module is deployed on an embedded device to load the trained backbone network model and enhance the preprocessed image to output a preliminary enhanced image.

[0159] The adaptive brightness adjustment post-processing module, deployed on the embedded device, is used to perform brightness compensation on the initially enhanced image and output the final enhanced image.

[0160] Specifically, in this embodiment, this application also provides a lightweight multi-scale image enhancement system for embedded real-time underwater video. The system includes the following functional modules, which work together to implement the image enhancement process in the aforementioned method embodiment:

[0161] The dataset construction module is used to build and partition underwater image datasets. This module is responsible for collecting underwater image samples from multiple public data sources (including UIEB, EUVP, RUIE, and SUID datasets) and the self-built O200 dataset, forming a diverse sample library covering different water depths, turbidity, lighting conditions, and degradation types. Following a pre-defined stratified sampling strategy, the module divides the dataset into training, validation, and test sets, ensuring the balance of scene distribution across each data subset and providing a high-quality sample foundation for subsequent network training.

[0162] The network construction module is used to build downsampling and upsampling modules, as well as the underwater image enhancement backbone network, without batch normalization layers. This module first constructs downsampling and upsampling modules based on stacked convolutional layers and the Leaky ReLU activation function, with no batch normalization layers in any of the modules. Based on this, a backbone network with a U-shaped encoder-decoder architecture is built: the encoder consists of multiple cascaded downsampling modules, extracting multi-scale features layer by layer; the decoder consists of multiple cascaded upsampling modules, gradually restoring image resolution; multi-scale skip connections are introduced between corresponding layers of the encoder and decoder to fuse shallow details and deep semantic features. Simultaneously, improved residual blocks are introduced after each layer of the encoder and decoder, lightweight feature calibration units are interspersed between the transposed convolutional modules of the decoder, and a channel scaling module is integrated into the network, collectively forming a complete lightweight enhancement network structure.

[0163] The training module is used to train the backbone network using a comprehensive loss function. This module configures the Adam optimizer and hyperparameters such as initial learning rate, batch size, and number of training epochs. Underwater degraded images and real reference images from the training set are input into the backbone network in pairs. Forward propagation calculates the augmented output, and the augmentation quality is evaluated using a comprehensive loss function that fuses L2 and SSIM losses. Then, backpropagation updates the network parameters. Iterative training continues until the loss converges, and the weights of the trained backbone network model are saved.

[0164] The preprocessing module, deployed on the embedded device, is used to perform size padding and normalization processing on the real-time acquired underwater images. This module receives the raw image frames input from the video acquisition unit, calculates the number of pixel rows and columns to be filled according to the network input size requirements (such as multiples of 16), expands the image using a constant padding method, and normalizes the pixel values ​​from [0,255] to the [0,1] range, generating a preprocessed image tensor that meets the input requirements of the backbone network.

[0165] A lightweight enhanced backbone network module, deployed on an embedded device, is used to load the trained backbone network model and enhance preprocessed images. This module inputs the preprocessed image into the encoder-decoder architecture, and through multi-scale feature extraction, feature enhancement using improved residual blocks, detail calibration by a lightweight feature calibration unit, feature recalibration by a channel scaling module, and feature fusion via multi-scale skip connections, the output layer generates the initial enhanced image. Experiments show that at a resolution of 640×480, this module achieves a processing speed of 36.23 FPS.

[0166] An adaptive brightness adjustment post-processing module, deployed on an embedded device, performs brightness compensation on the initially enhanced image. This module extracts image brightness statistics using global average pooling, generates brightness adjustment coefficients through two 1×1 convolution layers, introduces a learnable scaling factor to adjust the compensation magnitude, and then expands the coefficients to the image space dimension via a broadcast mechanism before multiplying them element-wise with the initially enhanced image. This corrects for brightness attenuation caused by nonlinear mapping in the network, outputting the final enhanced image. The average brightness of the processed image increases from 109.45 to 122.88, significantly improving visual comfort. These modules work collaboratively in a pipeline on the embedded device, achieving end-to-end real-time enhancement from image acquisition, preprocessing, network inference to brightness post-processing, providing an efficient and high-quality underwater video enhancement solution for resource-constrained platforms such as underwater robots.

[0167] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0168] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0169] It should be particularly noted that, through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, or of course, by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A lightweight multi-scale image enhancement method for embedded real-time underwater video, characterized in that, Includes the following steps: Step S1: Construct an underwater image dataset and divide the dataset into a training set, a validation set, and a test set; Step S2: Construct a downsampling module and an upsampling module based on convolution operations and nonlinear activation functions. The downsampling module and the upsampling module do not contain a batch normalization layer. Based on the downsampling module and the upsampling module, build an underwater image enhancement backbone network with a U-shaped encoder-decoder architecture. Step S3: Construct a comprehensive loss function that combines the L2 loss function and the structural similarity loss function; Step S4: Train the underwater image enhancement backbone network using the training set and the comprehensive loss function to obtain the trained underwater image enhancement backbone network, and deploy the trained backbone network into the target embedded device; Step S5: Acquire real-time underwater video images, perform size padding and normalization on the underwater video images through preprocessing; input the preprocessed underwater video images into the trained backbone network for feature extraction and enhancement, and output a preliminary enhanced image; Step S6: Input the preliminary enhanced image into the adaptive brightness adjustment post-processing module, and output the final enhanced underwater video image by compensating for brightness attenuation through brightness statistics and dynamic stretching.

2. The method according to claim 1, characterized in that, The construction of downsampling and upsampling modules based on convolution operations and nonlinear activation functions does not include a batch normalization layer. The steps of constructing an underwater image enhancement backbone network with a U-shaped encoder-decoder architecture based on the downsampling module and the upsampling module include: A downsampling module for spatial downsampling and an upsampling module for spatial upsampling are constructed by stacking convolutional layers and Leaky ReLU activation functions, and neither the downsampling module nor the upsampling module contains a batch normalization layer; Multiple downsampling modules are cascaded to form an encoder, which is used to extract features from the input image layer by layer and reduce the spatial resolution; multiple upsampling modules are cascaded to form a decoder, which is used to reconstruct the deep features output by the encoder layer by layer and restore the image spatial resolution; and multi-scale skip connections are introduced between corresponding levels of the encoder and the decoder to fuse the shallow features extracted by the encoder with the deep features in the decoder. An improved residual block is connected after each downsampling module of the encoder and after each upsampling module of the decoder; the improved residual block consists of two sets of cascaded convolutional layers and Leaky ReLU activation functions, and the input is directly added to the output through skip connections.

3. The method according to claim 2, characterized in that, The backbone network in step S2 also includes: A lightweight feature calibration unit is interspersed among the transposed convolutional modules of the decoder. The calibration unit first compresses the channel dimension through 1×1 convolution, then extracts spatial features through 3×3 depth convolution, and finally restores the channel dimension through 1×1 convolution, thereby achieving lightweight calibration of the decoder features.

4. The method according to claim 2, characterized in that, The backbone network in step S2 also includes: By initializing a scaling vector that matches the number of input feature channels and broadcasting it to the spatial dimension of the input features, and then multiplying it element-wise with the input features, adaptive control of the activation intensity of each channel can be achieved.

5. The method according to claim 1, characterized in that, The comprehensive loss function in step S4 consists of two parts: the L2 loss function and the structural similarity loss function. The L2 loss function is used to calculate the Euclidean distance between the enhanced image and the reference image, constraining low-level visual features; the specific formula is as follows: ； U is the enhanced image, I is the real reference image, where H and W represent the height and width of the image, respectively, and (m, n) represent the spatial coordinates of the pixel; The structural similarity loss function is used to calculate the structural similarity within a 13×13 local neighborhood, preserving the texture and structural features of the image. The formula is as follows: ； in and This represents the mean of a local area. and Represents local variance. Describing covariance, , It is the stability constant; The final comprehensive loss function L is given by the following formula: ； Where ω1 and ω2 are weighting coefficients.

6. The method according to claim 1, characterized in that, The step of training the underwater image enhancement backbone network using the training set and the comprehensive loss function to obtain the trained underwater image enhancement backbone network, and deploying the trained backbone network into the target embedded device, further includes: Configure the initial learning rate, batch size, and number of training rounds for network training, and use the Adam optimizer as the parameter update algorithm to iteratively optimize the weights of the underwater image enhancement backbone network during training. The underwater images in the training set are input in pairs into the underwater image enhancement backbone network for forward propagation. The combined loss function value between the enhanced output image and the real reference image is calculated. Then, the gradient is calculated through the backpropagation algorithm, and the parameters of the backbone network are updated using the optimizer until the loss function converges or reaches the preset number of training rounds, so as to obtain the trained backbone network model. The trained backbone network model parameter file is exported in an inference format suitable for the target embedded platform, loaded into the target embedded device, and the input / output interface is configured to enable the embedded device to acquire underwater video images in real time.

7. The method according to claim 1, characterized in that, The steps of acquiring real-time underwater video images, performing size padding and normalization on the underwater video images through preprocessing, and inputting the preprocessed underwater video images into the trained backbone network for feature extraction and enhancement, and outputting a preliminary enhanced image, further include: The video acquisition unit deployed on the embedded device acquires continuous frame images of underwater video in real time, and inputs each frame of raw image data into the preprocessing module. The preprocessing module obtains the original size information of the input image, performs edge-padding and expansion processing on the width and height of the image according to the preset network input size alignment rules, so that the expanded image size meets the specific multiple of the network input requirements, and uses a constant filling method to fill the expanded area with pixels, while normalizing the pixel values ​​to a preset numerical range, generating a preprocessed image that meets the backbone network input requirements. The preprocessed image is input into the trained backbone network deployed on the embedded device. The encoder of the backbone network performs multi-scale feature extraction, the decoder performs resolution restoration, and the multi-scale skip connections perform feature fusion. Finally, the output layer of the backbone network generates and outputs the preliminarily enhanced underwater image.

8. The method according to claim 1, characterized in that, The step of inputting the preliminary enhanced image into the adaptive brightness adjustment post-processing module, and outputting the final enhanced underwater video image through brightness statistics and dynamic stretching to compensate for brightness attenuation, further includes: The system receives the initial enhanced image output from the backbone network and extracts the global brightness statistics of the image through a global average pooling operation, which serves as the basis for subsequent brightness adjustments. The extracted global brightness statistics are input into a micro-network consisting of two 1×1 convolutional layers. After nonlinear mapping, brightness adjustment coefficients corresponding to the number of channels in the input image are generated. A learnable scaling factor is introduced to adaptively adjust the amplitude of the brightness adjustment coefficients in order to control the intensity of brightness adjustment. The generated brightness adjustment coefficients are extended to the same spatial dimension as the input image through a broadcast mechanism, and then multiplied or weighted with the preliminary enhanced image element-wise to achieve dynamic stretching compensation of the image brightness, correct the brightness attenuation caused by the nonlinear mapping of the network, and finally output an underwater video image with enhanced visual comfort.

9. A lightweight multi-scale image enhancement system for embedded real-time underwater video for implementing the method of any one of claims 1-8, characterized in that, The system includes: The dataset building module is used to build and segment underwater image datasets; The network construction module is used to build the downsampling module, upsampling module, and underwater image enhancement backbone network without a batch normalization layer. The training module is used to train the backbone network using the comprehensive loss function; The preprocessing module, deployed on the embedded device, is used to perform size filling and normalization processing on the real-time acquired underwater images. The lightweight enhanced backbone network module is deployed on an embedded device to load the trained backbone network model and enhance the preprocessed image to output a preliminary enhanced image. The adaptive brightness adjustment post-processing module, deployed on the embedded device, is used to perform brightness compensation on the initially enhanced image and output the final enhanced image.

Images