A maritime low-light video real-time enhancement system based on deep learning

Abstract

A deep learning-based real-time maritime low-light video enhancement system belongs to the field of maritime image processing and ship-aided navigation technology. The system includes: a video acquisition module for real-time acquisition of sea surface video images; an edge computing enhancement module electrically connected to the video acquisition module, which has a built-in pre-trained deep learning image enhancement model for real-time enhancement processing of the video images; a display module electrically connected to the edge computing enhancement module for real-time display of the enhanced video images; and a power supply module to power all modules. This invention can enhance sea surface video in real-time under low-light conditions such as nighttime and foggy weather, clearly presenting wave details, assisting crew members in judging wind speed, and features high system integration, strong real-time performance, and convenient operation.

Description

Technical Field

[0001] This invention belongs to the field of maritime image processing and ship-assisted navigation technology, specifically relating to a deep learning-based real-time maritime low-light video enhancement system, used to enhance sea surface videos in real time under adverse weather conditions such as night and fog, and to assist crew members in judging wind speed. Background Technology

[0002] During navigation, crew members observe the quantity, shape, and breakage of waves to help determine current wind speed and sea conditions—a long-standing empirical method in maritime practice. However, in adverse weather conditions such as nighttime, low light, and dense fog, human vision is severely obstructed, and traditional optical cameras cannot produce clear images. Although some ships are equipped with infrared thermal imaging cameras, their images generally suffer from low contrast, blurred edges, loss of detail, and significant noise, making it difficult to clearly present the texture and distribution of waves. This makes it difficult for crew members to capture wave details and fails to meet the stringent requirements for high-definition, high-real-time video streaming during navigation.

[0003] Furthermore, shipboard scenarios have strict requirements on device power consumption, size, and deployment compatibility. Existing models are often difficult to run efficiently on embedded edge platforms, which restricts the engineering application of low-light video enhancement technology in real ships. Summary of the Invention

[0004] The present invention aims to provide a real-time enhancement system for low-light maritime video based on deep learning, in order to solve the problem that existing ships cannot clearly observe the details of sea waves in adverse environments such as night and fog, and to realize the real-time enhancement and display of low-light video.

[0005] A deep learning-based real-time video enhancement system for maritime low-light conditions includes:

[0006] Video acquisition module: Installed on the exterior of the vessel or on top of the bridge, it is used to acquire real-time video images of the sea surface. The video acquisition module includes at least one camera, which is a visible light camera and / or an infrared camera.

[0007] Edge computing enhancement module: Deployed in the shipborne embedded device, electrically connected to the video acquisition module, it receives image frames output by the video acquisition module and runs a pre-trained deep learning image enhancement model to perform real-time enhancement processing on each frame. The edge computing enhancement module incorporates a neural network model with an asymmetric encoder-decoder structure. The encoder includes a frequency domain amplitude enhancement module, and the decoder includes a large receptive field spatial attention module. The edge computing enhancement module also includes a model optimization unit that converts the trained model to TensorRT or ONNX format and performs quantization compression.

[0008] The asymmetric encoder-decoder structure is specifically as follows:

[0009] The encoder consists of four downsampling stages, each containing a frequency domain amplitude enhancement module and a downsampling convolutional layer. The operation of the frequency domain amplitude enhancement module is as follows: perform a fast Fourier transform on the input feature map to separate the amplitude spectrum A and the phase spectrum P; perform a nonlinear transformation on the amplitude spectrum A through a two-layer multilayer perceptron to obtain the enhanced amplitude spectrum A' = MLP(A); combine the enhanced amplitude spectrum A' with the original phase spectrum P, and recover the spatial domain features through an inverse Fourier transform.

[0010] The decoder consists of four upsampling stages, each containing a large receptive field spatial attention module; the large receptive field spatial attention module is specifically as follows:

[0011] Three parallel, depthwise separable convolutions with dilation rates of 1, 4, and 9 are used to capture multi-scale contextual information. The outputs of the three branches are concatenated along the channel dimension and then fused through a 1×1 convolution. Simplified channel attention is applied to recalibrate the fused features. Finally, the output is passed through a gated feedforward network.

[0012] Features from each layer of the encoder are passed to the corresponding layer of the decoder through skip connections to recover high-frequency details.

[0013] Display module: Installed on the ship's bridge, electrically connected to the edge computing enhancement module, used to display enhanced video images in real time. The display module includes a touch screen with image enhancement on / off, brightness adjustment, and image zoom functions.

[0014] Power module: Electrically connected to the video acquisition module, edge computing enhancement module, and display module, providing operating power.

[0015] Data transmission module: includes a wired transmission unit and / or a wireless transmission unit, used for data transmission between the video acquisition module and the edge computing enhancement module, and between the edge computing enhancement module and the display module. The wired transmission unit includes a GMSI interface, an HDMI interface, or a network cable interface, and the wireless transmission unit includes a Wi-Fi module or a Bluetooth module.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] Integrated design: It organically integrates three major modules: video acquisition, edge computing enhancement, and display, to form a complete maritime video enhancement system. It is easy to install and plug and play.

[0018] High real-time performance: It adopts a lightweight deep learning model deployed on edge computing devices, with end-to-end processing latency of less than 50 milliseconds, which meets the low latency requirements for video streams during navigation.

[0019] Adaptable to harsh environments: Supports input from visible light and infrared cameras, and can operate normally under adverse weather conditions such as night, fog, rain, and snow.

[0020] User-friendly interface: The display module has touch control functionality, allowing crew members to turn image enhancement on / off with a single click, and supports zooming in and out to observe wave details more closely. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall system architecture.

[0022] Figure 2 This is a schematic diagram of the neural network model structure for the edge computing enhancement module.

[0023] Figure 3 This is a block diagram of the internal structure of the edge computing enhancement module. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0027] Example 1: System Overall Architecture

[0028] like Figure 1 As shown, this embodiment provides a real-time maritime low-light video enhancement system based on deep learning, including: a video acquisition module, an edge computing enhancement module, a display module, a power supply module, and a data transmission module.

[0029] Video acquisition module: Installed on top of the ship's bridge or mast, approximately 5-15 meters above the sea surface, with a downward angle of approximately 15-30 degrees, to cover the main sea surface observation area. This module includes at least one camera, specifically:

[0030] Infrared thermal imaging camera: model Hikvision DS-2CD5A26FWD-IZS, minimum illumination 0.004 lux, maximum resolution 2560×1440, frame rate 30fps, supports wide dynamic range (WDR) function, adapts to sea surface reflection scenes, and is suitable for dense fog and completely dark environments.

[0031] The camera housing has a protection rating of no less than IP66 and is resistant to salt spray corrosion.

[0032] Edge computing enhancement module: Deployed in a shipborne embedded device, this embodiment uses an NVIDIA Jetson OrinNX 16GB module integrated into a waterproof and shockproof chassis. This module is electrically connected to the video acquisition module 100 via the data transmission module 500 to receive the raw video stream. The edge computing enhancement module internally runs a Linux operating system and comes pre-installed with Python 3.10, PyTorch 2.5, and TensorRT runtime environments.

[0033] The edge computing enhancement module contains a finely tuned neural network model. Specifically, this model employs an asymmetric encoder-decoder structure (32 channels, 3.31M parameters) and has been finely trained on the maritime spray dataset. The model is stored in the form of a TensorRT engine and supports FP16 inference.

[0034] The workflow of the edge computing enhancement module is as follows:

[0035] The video capture module receives raw video frames (1920×1080 resolution) via the GMSI interface.

[0036] Each frame of image is scaled to 384×384 pixels and normalized to the [0,1] range;

[0037] The image is fed into the model for forward inference and the enhanced image (384×384×3) is obtained.

[0038] The enhanced image is scaled back to its original resolution (1920×1080) and output to the display module 300 via the HDMI interface.

[0039] The end-to-end latency of the entire processing pipeline (from camera capture to display) is less than 50 milliseconds, meeting real-time requirements.

[0040] Display module: Mounted on the ship's bridge control panel, it is a 15.6-inch industrial-grade touchscreen LCD monitor with a resolution of 1920×1080 and a brightness of 1000 nits (suitable for bright daylight environments), featuring an anti-glare coating. The monitor connects to the edge computing enhancement module 200 via an HDMI interface to display enhanced sea surface video in real time.

[0041] The monitor provides the following interactive controls (via touch or physical buttons):

[0042] Enhancement switch: One-click to turn the image enhancement function on / off, making it easy for crew members to compare the before and after enhancement effects;

[0043] Brightness / contrast adjustment: adapts to different ambient lighting conditions;

[0044] Screen zoom: Supports two-finger touch to zoom in, making it easier to observe details of the waves;

[0045] Wind speed reference scale: A legend showing the relationship between wave characteristics and wind speed at the edge of the screen.

[0046] Power supply module: The input is the ship's 24V DC power supply. Through a DC-DC conversion module, it outputs stable 12V and 5V voltages to power the video acquisition module (12V), the edge computing enhancement module (12V / 5A), and the display module (12V), respectively. The power supply module has built-in overvoltage, overcurrent, and short-circuit protection.

[0047] Data transmission module: includes wired transmission unit and wireless transmission unit.

[0048] Wired transmission unit: The video acquisition module and the edge computing enhancement module use GMSI (Automotive Serial Interface) coaxial cable, with a maximum transmission distance of 15 meters and resistance to electromagnetic interference. The edge computing enhancement module and the display module use HDMI 1.4 standard cable.

[0049] Wireless transmission unit: Optional Wi-Fi module (802.11ac) for wirelessly transmitting the enhanced video stream to crew members' tablets or mobile phones for secondary screen viewing. Wireless transmission uses H.264 encoding compression with adjustable bitrate.

[0050] Example 2: Internal Structure of Edge Computing Enhancement Module

[0051] like Figure 2 As shown, its specific structure is as follows:

[0052] The encoder (EBlock) consists of four downsampling stages, each containing a frequency domain amplitude enhancement module and a downsampling convolutional layer. The operation flow of the frequency domain amplitude enhancement module is as follows:

[0053] Perform a Fast Fourier Transform (FFT) on the input feature map to separate the amplitude spectrum A and the phase spectrum P;

[0054] The amplitude spectrum A is nonlinearly transformed through a two-layer multilayer perceptron (MLP) to obtain the enhanced amplitude spectrum A' = MLP(A);

[0055] The enhanced amplitude spectrum A' is combined with the original phase spectrum P, and the spatial characteristics are recovered by inverse Fourier transform (IFFT).

[0056] This module leverages the high correlation between frequency domain information and lighting conditions to effectively enhance the brightness of dark areas, while requiring far less computation than spatial domain attention.

[0057] Decoder (DBlock): The decoder consists of four upsampling stages, each containing a large receptive field spatial attention module (Di-SpAM). The Di-SpAM module is specifically as follows:

[0058] Three parallel depthwise separable convolutions with dilation rates of 1, 4, and 9 are used to capture multi-scale contextual information.

[0059] The outputs of the three branches are concatenated along the channel dimension and then fused using a 1×1 convolution.

[0060] Simplified Channel Attention (SCA) is applied to recalibrate the fused features;

[0061] Finally, the output is passed through a gated feedforward network (Gated FFN).

[0062] Skip connections: Features from each layer of the encoder are passed to the corresponding layer of the decoder through skip connections, which helps to recover high-frequency details.

[0063] Model parameter configuration: This embodiment uses a medium-sized version with 32 channels and 4 codec blocks. The total number of model parameters is approximately 3.31M, and the computational cost is 7.25 GMACs (for 384×384 input). This configuration can achieve a processing speed of over 30fps on edge devices such as NVIDIA Jetson Orin NX.

[0064] like Figure 3 As shown, the edge computing enhancement module further includes:

[0065] Video input interface: GMSI deserializer chip (such as MAX96717) deserializes the serial signal on the coaxial cable into a parallel video signal and outputs it in RGB format.

[0066] Image preprocessing unit: Implemented using CUDA kernel functions based on the GPU (NVIDIA Jetson Orin NX built-in GPU), including size scaling (using bilinear interpolation), normalization (subtracting the mean and dividing the standard deviation), and conversion to tensor format.

[0067] AI Inference Unit: Loads the TensorRT-optimized model engine and performs inference. This unit utilizes the GPU's Tensor Cores for FP16 matrix operations, with a single-frame inference time of approximately 15-20ms.

[0068] Image post-processing unit: Converts the inference result tensor back to image format, performs denormalization, scales back to the original resolution, and optionally performs color correction (such as adjusting saturation to enhance the white areas of the waves).

[0069] Video output interface: HDMI transmitter chip (such as IT66121) outputs the processed video frames to the display module at a rate of 60fps.

[0070] Data is transferred between units via a high-speed on-chip bus (such as PCIe or internal video memory) to avoid additional latency.

[0071] Example 3: System Installation Method

[0072] The installation steps for this system are as follows:

[0073] Select a suitable location on top of the ship's bridge to install the camera bracket, ensuring an unobstructed view and avoiding the ship's structure. Secure the camera to the bracket and adjust the pitch angle to a suitable position.

[0074] Lay a GMSI coaxial cable from the camera location along the ship's bulkhead to the edge computing box installation location inside the bridge, with the two ends connected to the GMSI interfaces of the camera and the edge computing module, respectively.

[0075] The edge computing module is mounted on a 19-inch cabinet or dedicated bracket inside the bridge and connected to the ship's 24V power supply.

[0076] The display module is embedded in an opening in the dashboard control panel, connected to the edge computing module via an HDMI cable, and then connected to a power source.

[0077] After all devices are powered on, an automatic startup script runs on the edge computing module to load the model and begin processing the video stream.

[0078] Observe the image on the display module, and adjust the enhancement switch and image parameters via the touch screen if necessary.

[0079] Example 4: System Workflow

[0080] After the system is powered on, the following loop will be executed automatically:

[0081] Step A: The video acquisition module acquires images of the sea surface in real time and sends them to the edge computing module via GMSI at a rate of 30fps;

[0082] Step B: The edge computing module preprocesses each frame of image and sends it to the model for inference to obtain the enhanced image;

[0083] Step C: The enhanced image is displayed in real time on the driver's console monitor via HDMI;

[0084] Step D: The crew observes the enhanced wave details and, in conjunction with the wind speed reference scale, determines the current wind speed;

[0085] Repeat steps A through D.

[0086] When the crew turns off the enhancement switch, the edge computing module directly transmits the original image to the display module without enhancement processing.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time video enhancement system for maritime low-light conditions based on deep learning, characterized in that, include: A video acquisition module, installed on the exterior of the vessel or on top of the bridge, is used to acquire real-time video images of the sea surface. The video acquisition module includes a visible light camera and / or an infrared camera. The edge computing enhancement module is deployed on the shipborne embedded device and is electrically connected to the video acquisition module. It has a built-in pre-trained deep learning image enhancement model for real-time enhancement processing of video images. The display module is installed on the ship's bridge and is electrically connected to the edge computing enhancement module for real-time display of enhanced video images. The power supply module is electrically connected to the video acquisition module, the edge computing enhancement module, and the display module.

2. The system according to claim 1, characterized in that, The video acquisition module includes a visible light camera and / or an infrared camera.

3. The system according to claim 1, characterized in that, The edge computing enhancement module has a built-in deep learning image enhancement model, the DarkIR neural network model, which adopts an asymmetric encoder-decoder structure.

4. The system according to claim 3, characterized in that, The edge computing enhancement module also includes a model optimization unit, which is used to convert the trained model into TensorRT or ONNX format and perform quantization compression.

5. The system according to claim 4, characterized in that, The asymmetric encoder-decoder structure is specifically as follows: The encoder consists of four downsampling stages, each containing a frequency domain amplitude enhancement module and a downsampling convolutional layer. The operation of the frequency domain amplitude enhancement module is as follows: perform a fast Fourier transform on the input feature map to separate the amplitude spectrum A and the phase spectrum P. The amplitude spectrum A is nonlinearly transformed through a two-layer multilayer perceptron to obtain the enhanced amplitude spectrum A' = MLP(A); the enhanced amplitude spectrum A' is combined with the original phase spectrum P, and the spatial characteristics are recovered through inverse Fourier transform. The decoder contains four upsampling stages, each containing a large receptive field spatial attention module; The large receptive field spatial attention module specifically includes: Three parallel, depthwise separable convolutions with dilation rates of 1, 4, and 9 are used to capture multi-scale contextual information. The outputs of the three branches are concatenated along the channel dimension and then fused through a 1×1 convolution. Simplified channel attention is applied to recalibrate the fused features. Finally, the output is passed through a gated feedforward network. Features from each layer of the encoder are passed to the corresponding layer of the decoder through skip connections to recover high-frequency details.

6. The system according to claim 5, characterized in that, The workflow of the edge computing enhancement module is as follows: S1. Receive raw video frames from the video acquisition module via the GMSI interface; S2. Scale each frame of the image and normalize it to the [0,1] interval; S3. The image is fed into the DarkIR neural network model of the deep learning image enhancement model for forward inference to obtain the enhanced image; S4. Scale the enhanced image back to its original resolution and output it to the display module via the HDMI interface.

7. The system according to claim 1, characterized in that, The display module has image enhancement switch, brightness adjustment, and screen zoom functions.

8. The system according to claim 1, characterized in that, It also includes a data transmission module, which includes a wired transmission unit and / or a wireless transmission unit.

Images