A lightweight underwater video enhancement method and system

Abstract

This application relates to a lightweight underwater video enhancement method and system. The system includes: a teacher subnetwork, a student subnetwork, and a decoding and video reconstruction subnetwork; the output of the teacher subnetwork is used as the input of the student subnetwork for distillation training; the output of the student subnetwork is used as the input of the decoding and video reconstruction subnetwork for video enhancement. This invention constructs teacher and student subnetworks, and distills the student subnetwork using the teacher subnetwork, enabling the student network to approximate the performance of the teacher network while maintaining a lightweight design, achieving high-quality, robust, and real-time underwater video restoration. It maintains the high-quality enhancement effect of the teacher network while significantly improving inference speed, making it suitable for resource-constrained scenarios such as unmanned underwater vehicles, underwater robots, and mobile devices.

Description

Technical Field

[0001] This invention relates to the field of video processing, and more specifically, to a lightweight underwater video enhancement method and system. Background Technology

[0002] Underwater video is widely used in marine exploration, underwater robots, marine engineering inspection, and aquaculture. However, due to the non-uniform absorption and scattering of light by water, as well as the influence of turbid suspended particles, the videos often suffer from the following problems: severe color shift (often appearing bluish-green); reduced contrast and blurred structure; scattering leading to enhanced low-frequency background and loss of high-frequency details; and significant fluctuations in illumination over time, resulting in video flickering and unstable colors. In recent years, deep learning-based methods have made significant progress in underwater image enhancement / restoration, but research has mainly focused on color correction and contrast enhancement of single images. Representative works include UWCNN (a convolutional neural network based on underwater scene priors) trained with synthetic data and several real-time enhancement methods based on GANs (such as FUnIE-GAN). These methods have performed well in color correction and overall visual quality improvement, and have spawned several publicly available benchmark datasets for evaluation and training. Compared to single images, underwater video restoration / enhancement needs to simultaneously ensure temporal consistency (avoiding inter-frame jitter and flicker) and utilize redundant information from adjacent frames to recover details. The mainstream strategy in video restoration usually involves first aligning adjacent frames, and then fusing and reconstructing them.

[0003] Existing underwater image enhancement methods are mainly single-frame enhancements (such as UWCNN, UIE-DAL, FUnIE-GAN, etc.). Although they are effective in color correction, they cannot handle video temporal consistency (which causes flickering), cannot utilize the redundancy of adjacent frames to recover details, and cannot be deployed in real-time applications of underwater robots.

[0004] Common alignment methods used in video super-resolution and video restoration include: optical flow + warp (severe jitter, sensitive to turbidity) feature-level alignment (TDAN); and multi-scale deformable convolution (EDVR). Among them, TDAN is relatively robust in feature domain alignment, but its computational cost is still large and it cannot capture long-term illumination variation trends in underwater scenes. Summary of the Invention

[0005] Therefore, it is necessary to provide a lightweight underwater video enhancement method and system with low computational cost to address the aforementioned technical problems.

[0006] One type of lightweight underwater video enhancement system includes a teacher subnetwork, a student subnetwork, and a decoding and video reconstruction subnetwork; The output of the teacher subnetwork is used as the input of the student subnetwork for distillation training; The output of the student subnetwork is used as the input to the decoding and video reconstruction subnetwork for video enhancement.

[0007] Furthermore, the teacher subnetwork includes: a TDAN motion alignment module, a local spatial feature extraction module, an FFT frequency domain enhancement module, an SSCS-Mamba backbone feature extraction module, and an LSTM global temporal control module.

[0008] Furthermore, the student subnetwork includes: a multidimensional LUT main enhancement module and an α-ResNet residual compensation module.

[0009] Furthermore, the decoding and video reconstruction subnetwork is specifically a decoder, which includes: a lightweight convolutional and color fusion module. Furthermore, this invention provides a method for a teacher subnetwork to process underwater video, specifically: TDAN Motion Alignment Module: Input t±k multi-frame sequence Feature domain alignment is achieved through deformable convolution of TDAN; the output is an aligned sequence. To provide a consistent spatial information basis for subsequent processing; Local spatial feature extraction: After alignment Perform shallow convolution feature extraction to obtain spatial domain features. This feature serves as the common input for both the FFT and LSTM modules; FFT frequency domain enhancement module Frequency domain enhancement is introduced into the backbone of underwater video restoration, and Mamba's sequence modeling capabilities are used to process frequency domain spectral information. Spatial features First, downsampling is performed, then FFT transformation is performed, and the frequency domain features after high-frequency emphasis are output. Finally, the frequency domain enhancement features and Fusion:

[0010] It can be dynamically adjusted by LSTM; Global LSTM timing control module: Spatial features Perform global pooling and standard deviation pooling:

[0011] Will Inputting the data into an LSTM model, we can model the long-term trend and obtain the control vector.

[0012] Controlling LUT weights enables dynamic LUT blending; SSCS-Mamba backbone feature extraction module: Enhanced Arranged into a continuous sequence, after Residual fusion outputs deep features ; Decoder reconstructs clear underwater video: Will Decoded as underwater augmentation results .

[0013] Furthermore, this invention provides a method for student subnetworks to process underwater video, specifically: The multidimensional LUT backbone enhancement module uses a multidimensional lookup table as the main enhancement branch; based on the preprocessed underwater video, brightness L, local contrast C, and frequency domain energy E are introduced as additional dimensions to form a local description vector R; Input vector:

[0014] LUT directly outputs the preliminary enhancement results:

[0015] α-ResNet residual compensation module: A dual-branch structure using LUT-dominated + ResNet residual compensation is employed: ; Will As a lightweight feature; in: · Output of Lightweight ResNet · Dynamically adjustable, learnable Depend on control:

[0016] Distillation loss: Pixel distillation loss:

[0017] Perceived loss:

[0018] Edge consistency loss:

[0019] Temporal consistency loss

[0020] The student subnetwork after distillation loss is used as a lightweight underwater video enhancement model.

[0021] Furthermore, this invention provides a lightweight underwater video enhancement method, including: S1: Acquire the underwater video to be enhanced and preprocess it to obtain the preprocessed underwater video; S2: Construct an underwater video-enhanced distillation network, which includes a teacher subnetwork and a student subnetwork; S3: Input the preprocessed underwater video into the teacher subnetwork to generate high-quality underwater video enhancement results; S4: Distill the student subnetwork based on the high-quality video enhancement results and the preprocessed video to obtain the features in the student subnetwork. S5: The features from the student subnetwork are input into the decoder and the final enhanced frame is generated through a lightweight convolution and color fusion module.

[0022] Further, in step S1, specifically: The underwater video to be enhanced is split into frame-by-frame images and its resolution is normalized to 640×360 or 1280×720. For each current frame, select two neighboring frames before and after it to form a... The five-frame input group is used as the preprocessed underwater video.

[0023] Furthermore, in step S3, specifically: S3.1: Perform motion alignment on the underwater video preprocessed by the TDAN motion alignment module to obtain the aligned underwater video; S3.2: Based on underwater video data, the local spatial feature extraction module, and the FFT frequency domain enhancement module, feature extraction is performed to obtain high-quality video features; S3.3: Input the high-quality video features into the LSTM global temporal control module to obtain the global control vector. The high-quality video features and the global control vector are used as the high-quality underwater video enhancement result. S3.4: The high-quality video features are arranged into a continuous sequence using the SSCS-Mamba backbone feature extraction module, and then... Residual fusion outputs deep features ; by depth features And global control vectors as high-quality underwater video enhancement results.

[0024] Further, in step S3.2, specifically: Aligned underwater video Perform shallow convolution feature extraction to obtain spatial domain features. ; Processing frequency domain spectral information using Mamba's sequence modeling capabilities: Spatial features First, downsampling is performed, then FFT transformation is performed, and the frequency domain features after high-frequency emphasis are output. Finally, the frequency domain enhancement features and Fusion:

[0025] Will As a characteristic of high-quality video; Dynamically adjusted by the LSTM global timing control module; In step S3.3, specifically: Spatial features Perform global pooling and standard deviation pooling:

[0026] Will Inputting the data into an LSTM model, we can model the long-term trend and obtain the control vector. .

[0027] This invention constructs a teacher subnetwork and a student subnetwork, and distills the student subnetwork through the teacher subnetwork. This allows the student network to approach the performance of the teacher network while maintaining a lightweight architecture, achieving high-quality, robust, and real-time underwater video restoration. It preserves the high-quality enhancement effect of the teacher network while significantly improving inference speed, making it suitable for resource-constrained scenarios such as unmanned underwater vehicles, underwater robots, and mobile devices. Attached Figure Description

[0028] Figure 1 This is a flowchart of a lightweight underwater video enhancement method in one embodiment; Figure 2 This is a block diagram of a lightweight underwater video enhancement system in one embodiment; Detailed Implementation To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0029] Example 1: This embodiment provides, as follows: Figure 1 The present invention discloses a lightweight underwater video enhancement system, which includes a teacher subnetwork, a student subnetwork, and a decoding and video reconstruction subnetwork. The output of the teacher subnetwork is used as the input of the student subnetwork for distillation training; The output of the student subnetwork is used as the input to the decoding and video reconstruction subnetwork for video enhancement.

[0030] This embodiment constructs a teacher subnetwork and a student subnetwork, and distills the student subnetwork through the teacher subnetwork. This allows the student network to approach the performance of the teacher network while maintaining a lightweight design, achieving high-quality, robust, and real-time underwater video restoration. It maintains the high-quality enhancement effect of the teacher network while significantly improving inference speed, making it suitable for resource-constrained scenarios such as unmanned underwater vehicles, underwater robots, and mobile devices.

[0031] Example 2: This embodiment further discloses information based on Embodiment 1: Furthermore, the teacher subnetwork includes: a TDAN motion alignment module, a local spatial feature extraction module, an FFT frequency domain enhancement module, an SSCS-Mamba backbone feature extraction module, and an LSTM global temporal control module.

[0032] Furthermore, the student subnetwork includes: a multidimensional LUT main enhancement module and an α-ResNet residual compensation module.

[0033] Furthermore, the decoding and video reconstruction subnetwork is specifically a decoder, which includes: a lightweight convolutional and color fusion module.

[0034] This embodiment constructs a teacher subnetwork and a student subnetwork, and distills the student subnetwork through the teacher subnetwork. This allows the student network to approach the performance of the teacher network while maintaining a lightweight design, achieving high-quality, robust, and real-time underwater video restoration. It maintains the high-quality enhancement effect of the teacher network while significantly improving inference speed, making it suitable for resource-constrained scenarios such as unmanned underwater vehicles, underwater robots, and mobile devices.

[0035] Example 3: This embodiment provides a method for a teacher subnetwork to process underwater video, specifically as follows: TDAN Motion Alignment Module: Input t±k multi-frame sequence Feature domain alignment is achieved through deformable convolution of TDAN; the output is an aligned sequence. To provide a consistent spatial information basis for subsequent processing; Local spatial feature extraction: After alignment Perform shallow convolution feature extraction to obtain spatial domain features. This feature serves as the common input for both the FFT and LSTM modules; FFT frequency domain enhancement module Frequency domain enhancement is introduced into the backbone of underwater video restoration, and Mamba's sequence modeling capabilities are used to process frequency domain spectral information. Spatial features First, downsampling is performed, then FFT transformation is performed, and the frequency domain features after high-frequency emphasis are output. Finally, the frequency domain enhancement features and Fusion:

[0036] It can be dynamically adjusted by LSTM; Global LSTM timing control module: Spatial features Perform global pooling and standard deviation pooling:

[0037] Will Inputting the data into an LSTM model, we can model the long-term trend and obtain the control vector.

[0038] Controlling LUT weights enables dynamic LUT blending; SSCS-Mamba backbone feature extraction module: Enhanced Arranged into a continuous sequence, after Residual fusion outputs deep features ; Decoder reconstructs clear underwater video: Will Decoded as underwater augmentation results .

[0039] Example 4: This embodiment provides a method for student subnetworks to process underwater videos, specifically as follows: The multidimensional LUT backbone enhancement module uses a multidimensional lookup table as the main enhancement branch; based on the preprocessed underwater video, brightness L, local contrast C, and frequency domain energy E are introduced as additional dimensions to form a local description vector R; Input vector:

[0040] LUT directly outputs the preliminary enhancement results:

[0041] α-ResNet residual compensation module: A dual-branch structure using LUT-dominated + ResNet residual compensation is employed: ; Will As a lightweight feature; in: · Output of Lightweight ResNet · Dynamically adjustable, learnable Depend on control:

[0042] Distillation loss: Pixel distillation loss:

[0043] Perceived loss:

[0044] Edge consistency loss:

[0045] Temporal consistency loss

[0046] The student subnetwork after distillation loss is used as a lightweight underwater video enhancement model.

[0047] Example 5: This embodiment provides, as follows: Figure 2 The present invention discloses a lightweight underwater video enhancement method, comprising: S1: Acquire the underwater video to be enhanced and preprocess it to obtain the preprocessed underwater video; S2: Construct an underwater video-enhanced distillation network, which includes a teacher subnetwork and a student subnetwork; S3: Input the preprocessed underwater video into the teacher subnetwork to generate high-quality underwater video enhancement results; S4: Distill the student subnetwork based on the high-quality video enhancement results and the preprocessed video to obtain the features in the student subnetwork. S5: The features from the student subnetwork are input into the decoder and the final enhanced frame is generated through a lightweight convolution and color fusion module.

[0048] This embodiment constructs a teacher subnetwork and a student subnetwork, and distills the student subnetwork through the teacher subnetwork. This allows the student network to approach the performance of the teacher network while maintaining a lightweight design, achieving high-quality, robust, and real-time underwater video restoration. It maintains the high-quality enhancement effect of the teacher network while significantly improving inference speed, making it suitable for resource-constrained scenarios such as unmanned underwater vehicles, underwater robots, and mobile devices.

[0049] Example 6: This embodiment further discloses information based on Embodiment 5: Further, in step S1, specifically: The underwater video to be enhanced is split into frame-by-frame images and its resolution is normalized to 640×360 or 1280×720. For each current frame, select two neighboring frames before and after it to form a... The five-frame input group is used as the preprocessed underwater video.

[0050] Furthermore, in step S3, specifically: S3.1: Perform motion alignment on the underwater video preprocessed by the TDAN motion alignment module to obtain the aligned underwater video; S3.2: Based on underwater video data, the local spatial feature extraction module, and the FFT frequency domain enhancement module, feature extraction is performed to obtain high-quality video features; S3.3: Input the high-quality video features into the LSTM global temporal control module to obtain the global control vector. The high-quality video features and the global control vector are used as the high-quality underwater video enhancement result. S3.4: The high-quality video features are arranged into a continuous sequence using the SSCS-Mamba backbone feature extraction module, and then... Residual fusion outputs deep features ; by depth features And global control vectors as high-quality underwater video enhancement results.

[0051] Further, in step S3.2, specifically: Aligned underwater video Perform shallow convolution feature extraction to obtain spatial domain features. ; Processing frequency domain spectral information using Mamba's sequence modeling capabilities: Spatial features First, downsampling is performed, then FFT transformation is performed, and the frequency domain features after high-frequency emphasis are output. Finally, the frequency domain enhancement features and Fusion:

[0052] Will As a characteristic of high-quality video; Dynamically adjusted by the LSTM global timing control module; In step S3.3, specifically: Spatial features Perform global pooling and standard deviation pooling:

[0053] Will Inputting the data into an LSTM model, we can model the long-term trend and obtain the control vector. .

[0054] This embodiment of the invention constructs a teacher subnetwork and a student subnetwork, and distills the student subnetwork through the teacher subnetwork. This allows the student network to approach the performance of the teacher network while maintaining a lightweight design, achieving high-quality, robust, and real-time underwater video restoration. It maintains the high-quality enhancement effect of the teacher network while significantly improving inference speed, making it suitable for resource-constrained scenarios such as unmanned underwater vehicles, underwater robots, and mobile devices.

Claims

1. A lightweight underwater video enhancement system, characterized in that, It includes a teacher subnetwork, a student subnetwork, and a decoding and video reconstruction subnetwork; The output of the teacher subnetwork is used as the input of the student subnetwork for distillation training; The output of the student subnetwork is used as the input to the decoding and video reconstruction subnetwork for video enhancement.

2. The lightweight underwater video enhancement system according to claim 1, characterized in that, The teacher subnetwork includes: TDAN motion alignment module, local spatial feature extraction module, FFT frequency domain enhancement module, SSCS-Mamba backbone feature extraction module, and LSTM global temporal control module.

3. The lightweight underwater video enhancement system according to claim 1, characterized in that, The student subnetwork includes: a multidimensional LUT main enhancement module and an α-ResNet residual compensation module.

4. The lightweight underwater video enhancement system according to claim 1, characterized in that, The decoding and video reconstruction subnetwork is specifically a decoder, which includes a lightweight convolutional and color fusion module.

5. A method for processing underwater video using a teacher subnet, characterized in that, Specifically: TDAN Motion Alignment Module: Input t±k multi-frame sequence Feature domain alignment is achieved through deformable convolution of TDAN; Output alignment sequence To provide a consistent spatial information basis for subsequent processing; Local spatial feature extraction: After alignment Perform shallow convolution feature extraction to obtain spatial domain features. This feature serves as the common input for both the FFT and LSTM modules; FFT frequency domain enhancement module Frequency domain enhancement is introduced into the backbone of underwater video restoration, and Mamba's sequence modeling capabilities are used to process frequency domain spectral information. Spatial features First, downsampling is performed, then FFT transformation is performed, and the frequency domain features after high-frequency emphasis are output. Finally, the frequency domain enhancement features and Fusion: It can be dynamically adjusted by LSTM; Global LSTM timing control module: Spatial features Perform global pooling and standard deviation pooling: Will Inputting the data into an LSTM model, we can model the long-term trend and obtain the control vector. Controlling LUT weights enables dynamic LUT blending; SSCS-Mamba backbone feature extraction module: Enhanced Arranged into a continuous sequence, after Residual fusion outputs deep features ; Decoder reconstructs clear underwater video: Will Decoded as underwater augmentation results .

6. A method for processing underwater video using a student subnet, characterized in that, Specifically: The multidimensional LUT backbone enhancement module uses a multidimensional lookup table as the main enhancement branch; based on the preprocessed underwater video, brightness L, local contrast C, and frequency domain energy E are introduced as additional dimensions to form a local description vector R; Input vector: LUT directly outputs the preliminary enhancement results: α-ResNet residual compensation module: A dual-branch structure using LUT-dominated + ResNet residual compensation is employed: ； Will As a lightweight feature; in: · Output of Lightweight ResNet · Dynamically adjustable, learnable Depend on control: Distillation loss: Pixel distillation loss: Perceived loss: Edge consistency loss: Temporal consistency loss The student subnetwork after distillation loss is used as a lightweight underwater video enhancement model.

7. A lightweight underwater video enhancement method, characterized in that, include: S1: Acquire the underwater video to be enhanced and preprocess it to obtain the preprocessed underwater video; S2: Construct an underwater video-enhanced distillation network, which includes a teacher subnetwork and a student subnetwork; S3: Input the preprocessed underwater video into the teacher subnetwork to generate high-quality underwater video enhancement results; S4: Distill the student subnetwork based on the high-quality video enhancement results and the preprocessed video to obtain the features in the student subnetwork. S5: The features from the student subnetwork are input into the decoder and the final enhanced frame is generated through a lightweight convolution and color fusion module.

8. The lightweight underwater video enhancement method according to claim 1, characterized in that, In step S1, specifically: The underwater video to be enhanced is split into frame-by-frame images and its resolution is normalized to 640×360 or 1280×720. For each current frame, select two neighboring frames before and after it to form a... The five-frame input group is used as the preprocessed underwater video.

9. A lightweight underwater video enhancement method according to claim 7, characterized in that, In step S3, specifically: S3.1: Perform motion alignment on the underwater video preprocessed by the TDAN motion alignment module to obtain the aligned underwater video; S3.2: Based on underwater video data, the local spatial feature extraction module, and the FFT frequency domain enhancement module, feature extraction is performed to obtain high-quality video features; S3.3: Input the high-quality video features into the LSTM global temporal control module to obtain the global control vector. The high-quality video features and the global control vector are used as the high-quality underwater video enhancement result. S3.4: The high-quality video features are arranged into a continuous sequence using the SSCS-Mamba backbone feature extraction module, and then... Residual fusion outputs deep features ; by depth features And global control vectors as high-quality underwater video enhancement results.

10. A lightweight underwater video enhancement method according to claim 7, characterized in that, In step S3.2, specifically: Aligned underwater video Perform shallow convolution feature extraction to obtain spatial domain features. ; Processing frequency domain spectral information using Mamba's sequence modeling capabilities: Spatial features First, downsampling is performed, then FFT transformation is performed, and the frequency domain features after high-frequency emphasis are output. Finally, the frequency domain enhancement features and Fusion: Will As a characteristic of high-quality video; Dynamically adjusted by the LSTM global timing control module; In step S3.3, specifically: Spatial features Perform global pooling and standard deviation pooling: Will Inputting the data into an LSTM model, we can model the long-term trend and obtain the control vector. 。

Images