An underwater image defogging method and system based on a full-optical diffraction neural network

Abstract

The application discloses an underwater image defogging method and system based on a full-optical diffraction neural network. The method comprises the following steps: acquiring a real underwater degraded image and a clear image; constructing a full-optical diffraction neural network to establish an end-to-end optical mapping from the degraded image to the defogging image; determining the physical propagation law of light between network layers based on the Huygens-Fresnel diffraction principle; taking a certain number of cascaded diffraction neural network layers as a passive encoder of the full-optical diffraction neural network, and taking a single lens imaging structure with a fixed focal length as a passive decoder of the full-optical diffraction neural network; running the full-optical diffraction neural network by using Python and Pytorch programming, and training the full-optical diffraction neural network by using a supervised training and a gradient descent method; and inputting the real underwater degraded image into the trained full-optical diffraction neural network to generate a defogging image. The application has application prospects in the fields of underwater environment and biological monitoring, precision engineering measurement, archaeological research and national defense security.

Description

Technical Field

[0001] This invention relates to the fields of image processing, deep learning and electromagnetic control technology, and more specifically to an underwater image dehazing method and system based on an all-optical diffraction neural network. Background Technology

[0002] Underwater imaging is a challenging branch of marine research, widely used in underwater environmental and biological monitoring, precision engineering measurement, archaeological research, and national defense. However, due to the nature of light propagation in water, different wavelengths of light attenuate to varying degrees, with red light attenuating more severely than blue or green light. Therefore, underwater images are often displayed in a blue-green hue. Furthermore, scattering by water and suspended particles can cause blurring, distortion, and turbidity in underwater optical images. Traditional methods for image dehazing include image enhancement techniques, which use filters and dehazing techniques to enhance contrast or reduce noise, thereby improving perception and imaging quality. The drawback of these methods is that they only provide an enhancement effect for underwater images and cannot easily recover high-quality images from degraded images using known information, often resulting in residual haze and blurred facial features.

[0003] Deep learning is a rapidly developing and significant branch of machine learning methods. Its powerful algorithms in abstract feature recognition and extraction, model building, and generalization have led to its widespread application in underwater image dehazing. However, these methods are generally limited by complex algorithms and network architectures, making it difficult to generalize trained models to the complex physical scenarios of real underwater images, and computational latency makes them unsuitable for real-time processing. As a bridge between deep learning and optical computing, diffractive neural networks (DNNs) retain the representational capabilities and feature extraction advantages of traditional neural networks while offering inherent physical advantages—light-speed inference, low energy consumption, and massive parallelism. Recently, research on DNNs has garnered significant attention. DNNs are excellent accelerators for simulating artificial intelligence, using photons instead of electrons for computation. In this architecture, specific inference and computation tasks are parameterized as a series of physical property functions of engineered surfaces or layers, interconnected through light diffraction. DNNs have proven successful in achieving various optical tasks, such as image classification, phase retrieval, data encryption, spectral processing, image reconstruction, 3D vision tasks, and logical operations. Applying DNNs to underwater image dehazing is an attempt to combine them with more complex image processing, further expanding the applications of DNNs.

[0004] Therefore, proposing a high-performance dehazing strategy based on all-optical diffraction neural network (AODNN) is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, this invention provides an underwater image dehazing method and system based on an all-optical diffraction neural network. It utilizes wavefront modulation and spatial diffraction propagation in a multi-layer physical network to achieve feature separation and enhancement of degraded underwater images, thereby establishing a direct end-to-end optical mapping from degraded input to sharp output. Using pairs of degraded and sharp underwater images captured by a light field camera, a multi-task learning framework based on sparse samples is constructed, and the underwater image dehazing task is designed and implemented using Python and PyTorch. This invention also verifies the unique advantages of all-optical diffraction neural networks (AODNN) in processing degraded underwater images, providing a high-performance computing paradigm for high-speed, low-power, and real-time underwater visual enhancement.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A method for underwater image dehazing based on an all-optical diffraction neural network includes: Step 1: Acquire realistic underwater degraded images and clear images; Step 2: Construct an all-optical diffraction neural network to establish an end-to-end optical mapping from the degraded image to the dehazed image; Step 3: Based on the Huygens-Fresnel diffraction principle, determine the physical propagation law of light between network layers, use a certain number of cascaded diffraction neural network layers as the passive encoder of the all-optical diffraction neural network, and use a single lens imaging structure with a fixed focal length as the passive decoder of the all-optical diffraction neural network. Step 4: Run the all-optical diffraction neural network using Python and PyTorch, and train it using supervised training and gradient descent. Step 5: Input the real underwater degradation image into the trained all-optical diffraction neural network to generate a dehazed image.

[0007] Preferably, the passive encoder includes a three-layer diffractive neural network cascaded part, each diffractive neural network layer is capable of phase modulation, and each neuron is regarded as a secondary wave source during light propagation. According to the Huygens-Fresnel diffraction principle, light carries information and diffracts between neurons, thereby realizing the forward propagation of the network.

[0008] Preferably, a single lens imaging structure is used as a passive decoder of the all-optical diffraction neural network and cascaded after the passive encoder. The distance between the lens and the last diffraction neural network layer and the distance between the lens and the imaging receiving surface are the same as the focal length of the lens. After the light field output by the last diffraction neural network layer passes through the passive decoder, it undergoes Fourier transform and presents a specific light spot pattern of a clear image after defogging on the imaging receiving surface.

[0009] Preferably, the loss function used for training with supervised training and gradient descent is the mean squared error, and the error calculation formula is as follows: ; in, It is a dehazed image. It is a clear image. It is the number of pixels in the image. When the training of the loss function converges and stabilizes, the final phase parameter is the optimized phase distribution of the diffraction layer.

[0010] Preferably, all-optical diffraction neural networks are run using Python and PyTorch programming, including: Write the main.py function as the main function of the project. By calling the function library, it can realize the reading of batch data and network elements, training loop, calculation of loss function, and output of loss value and interactive image. Write a function mydataset.py as the data input function to implement the preprocessing process of reading, cropping and normalizing underwater degraded images and their corresponding clear images; Write the nn_module.py function as the core function of the all-optical diffraction neural network, which includes phase modulation, Fresnel diffraction, lens and receiving screen classes; Write the eval.py function as the prediction function of the all-optical diffraction neural network. By reading the test dataset, as well as the trained network and parameters, it performs inference prediction, calculates the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) of the output result and the clear image, and saves the output result as a PNG image.

[0011] Preferably, an underwater image dehazing system based on an all-optical diffraction neural network includes: Image acquisition module: Acquires both realistic underwater degraded images and clear images; Encoder module: Based on the Huygens-Fresnel diffraction principle, the physical propagation law of light between network layers is determined, and a certain number of cascaded diffraction neural network layers are used as passive encoders of all-optical diffraction neural networks; Decoder module: Uses a single lens imaging structure with a fixed focal length as a passive decoder for an all-optical diffraction neural network; Model training module: The all-optical diffraction neural network is run using Python and PyTorch programming, and supervised training and gradient descent are used for training.

[0012] As can be seen from the above technical solution, compared with the prior art, the present invention discloses an underwater image dehazing method and system based on all-optical diffraction neural network, which has the following beneficial effects: This invention innovatively proposes an optical coding strategy for dehazing underwater degraded images, constructs a multi-task learning framework with sparse samples (extremely simple and practical), and utilizes an all-optical inference mechanism to achieve low-energy, high-speed optical computation enhancement and image dehazing. The results are presented intuitively, verifying the unique advantages of all-optical diffraction neural networks in processing underwater degraded images. It provides a high-performance computing paradigm for high-speed, low-power real-time visual enhancement underwater, and has application prospects in underwater environment and biological monitoring, precision engineering measurement, archaeological research, and national defense security. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0014] Figure 1 The system framework diagram for underwater image dehazing using AODNN provided by this invention is shown.

[0015] Figure 2 Example images of a portion of the dataset provided by this invention.

[0016] Figure 3 This is a schematic diagram of the training results provided by the present invention.

[0017] Figure 4 The method flowchart provided by the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 4 As shown, this invention discloses an underwater image dehazing method based on an all-optical diffraction neural network (AODNN), comprising: Step 1: Acquire realistic underwater degraded images and clear images; Step 2: Construct an all-optical diffraction neural network to establish an end-to-end optical mapping from the degraded image to the dehazed image; Step 3: Based on the Huygens-Fresnel diffraction principle, determine the physical propagation law of light between network layers, use a certain number of cascaded diffraction neural network layers as the passive encoder of the all-optical diffraction neural network, and use a single lens imaging structure with a fixed focal length as the passive decoder of the all-optical diffraction neural network. Step 4: Run the all-optical diffraction neural network using Python and PyTorch, and train it using supervised training and gradient descent. Step 5: Input the real underwater degradation image into the trained all-optical diffraction neural network to generate a dehazed image.

[0020] In a specific embodiment of the present invention, the physical propagation law of light between network layers is determined based on the Huygens-Fresnel diffraction principle, and a certain number of cascaded diffraction neural network layers are combined to form a passive encoder of AODNN, and the diffraction layer realizes the physical phase modulation of the network. Based on the principles of minimal network architecture, reduced computational latency, and low energy consumption, a single lens imaging structure with a focal length of 200mm is used as the passive decoder of the AODNN. This decoder, together with the encoder, forms the complete AODNN. An end-to-end optical mapping is established from the degraded image to the dehazed image. The AODNN is implemented using Python and PyTorch programming. The diffraction neural network employs supervised training and gradient descent, where the learnable parameters are the phase arrangement of each diffraction layer. A total of 430,923 phases are available for learning.

[0021] Furthermore, the real underwater degraded images and clear images in this invention are from Skinner KA, Johnson-Roberson M. Underwater image dehazing with a light field camera [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017: 62-69.

[0022] Specifically, the passive encoder includes a three-layer diffractive neural network cascade. Each diffractive neural network layer is capable of phase modulation. Each neuron is regarded as a secondary wave source during light propagation. According to the Huygens-Fresnel diffraction principle, light carries information and diffracts between neurons, thereby realizing the forward propagation of the network.

[0023] In a specific embodiment of the present invention, the passive encoder is part of an AODNN network architecture composed of three cascaded diffractive neural networks, each diffractive layer having phase modulation capability, and its construction method includes the following steps: The target pixels of the diffraction layer are simulated as neurons in a neural network layer. Each neuron carries information such as the complex transmission coefficient. Together, they can perform phase modulation on the received light field. The modulated light field can be regarded as the light field physically transmitted through this network layer. Each neuron can be considered a secondary wave source during light propagation. According to the Huygens-Fresnel diffraction principle, light carries information and diffracts between neurons, thereby enabling the forward propagation of the network. The interlayer spacing of the diffraction layer is 200 mm, and the dimensions are... An array of pixels.

[0024] Specifically, a single lens imaging structure is used as a passive decoder of the all-optical diffraction neural network and cascaded after the passive encoder. The distance between the lens and the last diffraction neural network layer, as well as the distance between the lens and the imaging receiving surface, are the same as the focal length of the lens. The light field output by the last diffraction neural network layer is subjected to Fourier transform after passing through the passive decoder, and presents a specific light spot pattern of a clear image after defogging on the imaging receiving surface.

[0025] In a specific embodiment of the present invention, the passive decoder is part of an AODNN network architecture composed of a single lens imaging structure with a focal length of 200mm, and its construction method includes: The goal of this invention is to simplify the network architecture as much as possible, reduce computational latency and energy consumption, so as to facilitate underwater operations and research. Therefore, when constructing the decoder, priority is given to optical devices or systems that do not require additional power. The lens imaging structure is selected as the passive decoder and cascaded after the encoder of AODNN. The distance between the lens and the last diffraction layer and the imaging receiving surface is equal to its focal length of 200mm. The light field output from the last diffraction layer is decoded and then subjected to Fourier transform. The specific spot pattern of the clear image after dehazing will be visually presented on the imaging receiving surface (i.e., the Fourier spectrum surface).

[0026] Specifically, the loss function used for training with supervised training and gradient descent is the mean squared error, and the error calculation formula is as follows: ; in, It is a dehazed image. It is a clear image. It is the number of pixels in the image. When the training of the loss function converges and stabilizes, the final phase parameter is the optimized phase distribution of the diffraction layer.

[0027] Specifically, running all-optical diffraction neural networks using Python and PyTorch programming includes: Write the main.py function as the main function of the project. By calling the function libraries you wrote yourself, such as mydataset and nn_module, as well as commonly used function libraries such as PyTorch and numpy, you can realize the reading of batch data and network elements, training loop, calculation of loss function, and output of loss value and interactive image. Write a function mydataset.py to serve as the data input function, and implement preprocessing such as reading, cropping and normalizing underwater degraded images (with fog) and their corresponding clear images (without fog); Write the nn_module.py function as the core implementation of AODNN, which includes classes such as phase modulation, Fresnel diffraction, lens and receiving screen, and implements the corresponding physical functions. Write the eval.py function as the prediction implementation of AODNN. By reading the test dataset, as well as the trained network and parameters, it performs inference prediction, calculates the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) of the output result (dehazed image) and the label value (clear image), and saves the output result as a PNG image. The underwater image dehazing algorithm is embedded into the AODNN computation process using the four functions mentioned above. No additional data processing is required. The inference and prediction mechanism of AODNN is entirely optical, and the training and learning of the diffraction layer phase parameters are computer-based.

[0028] Specifically, an underwater image dehazing system based on an all-optical diffraction neural network includes: Image acquisition module: Acquires both realistic underwater degraded images and clear images; Encoder module: Based on the Huygens-Fresnel diffraction principle, the physical propagation law of light between network layers is determined, and the three-layer cascaded diffraction neural network layer is used as a passive encoder of the all-optical diffraction neural network; Decoder module: The single-lens imaging structure with a fixed focal length is used as a passive decoder for the all-optical diffraction neural network; Model training module: The all-optical diffraction neural network is run using Python and PyTorch programming, and supervised training and gradient descent are used for training.

[0029] In the specific embodiments provided by the present invention, such as Figure 1As shown, real images completely preserve the complex degradation characteristics and random disturbances in the actual underwater environment, which is difficult for synthetic images to accurately simulate. However, acquiring real images is challenging. Therefore, the underwater image dehazing system based on all-optical diffraction neural network (AODNN) provided in this embodiment can be applied to real underwater degradation images with sparse samples. Five real underwater degradation images captured using a light field camera were used in the training, including five corresponding clear images (proposed by Skinner KA et al.). Some data examples are shown below. Figure 2 As shown, this specifically includes: training AODNN on underwater degraded images using a real dataset captured by a light field camera.

[0030] The implementation example design metrics are as follows: Supervised training and gradient descent are used to train AODNN on underwater degraded images. The training data is... The system reads five samples per batch, including both true grayscale and color images of the pixels; the loss function is the mean squared error (MSE). It is a dehazed image. Yes and clear images, This refers to the number of pixels in the image. The error calculation formula is as follows:

[0031] When the training of the loss function converges and stabilizes, the final phase parameters are the optimized phase distribution of the diffraction layer.

[0032] Train AODNN-Gray to study its dehazing effect on grayscale images.

[0033] With the network architecture and training parameters unchanged, AODNN-RGB was trained on RGB images to study its dehazing effect. For the image dehazing task, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) were used to evaluate the dehazing quality of the images.

[0034] The training results of grayscale images are as follows Figure 3 As shown in (a), qualitative analysis shows that the AODNN model can adaptively handle non-uniform haze distribution and degradation patterns in complex scenes through end-to-end feature learning, and has a good effect on restoring detailed information of hazy images and improving global contrast.

[0035] The training results of RGB images are as follows Figure 3As shown in (b), qualitative analysis shows that the AODNN model can learn multiple tasks in parallel, such as dehazing, detail restoration, edge processing, contrast enhancement, and color restoration. It demonstrates powerful image processing capabilities under sparse samples. Existing underwater image dehazing work often requires cascading several complex networks and using complex loss functions to achieve these functions. However, AODNN's minimalist architecture and extremely fast training speed demonstrate its unique advantages.

[0036] Quantitative analysis: The quantitative results of PSNR and SSIM are shown in Table 1. AODNN is still convincing in terms of objective indicators. At the same time, the generated dehazed images are visually clear and natural, with delicate details and edges, and no obvious color distortion or halo artifacts.

[0037] Table 1 Quantitative results using AODNN

[0038] This invention verifies the proposed high-performance dehazing strategy at the simulation level, demonstrating that AODNN can handle degraded images with complex features. Furthermore, the results are experimentally feasible, with the following approach: the diffraction layer is physically implemented using a reconfigurable intelligent metasurface (RIS) or a spatial light modulator (SLM), both of which possess programmability and reconfigurability. This allows for adaptive design of network layers based on the task, fabrication of either a RIS or an SLM based on the phase distribution obtained during training, and the establishment of a simple experimental platform to verify the results.

[0039] The AODNN proposed in this invention uses only basic optical devices to form the encoder and decoder. It employs a small number of network layers, equivalent to a fully connected linear network, and uses a common training method with only the phase distribution of the diffraction layers as the training parameter. Despite this, it still achieves excellent image dehazing performance on sparse samples. Further verification of AODNN's generalization ability on large datasets is highly likely, with the following approaches: 1) Introducing optical domain enhancements to the input image (such as random rotation, scaling, and noise addition); 2) Increasing trainable parameters, such as the spacing, geometry, and mechanical arrangement of the diffraction layers; 3) Recently, many studies on nonlinear DNNs have emerged. Replacing a linear DNN with a nonlinear DNN can achieve more efficient performance.

[0040] In summary, this embodiment proposes an underwater image dehazing method and system based on all-optical diffraction neural networks, providing a high-performance computing paradigm for high-speed, low-power real-time underwater visual enhancement, with promising applications in underwater environment and biological monitoring, precision engineering measurement, archaeological research, and national defense security.

[0041] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0042] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for underwater image dehazing based on an all-optical diffraction neural network, characterized in that, include: Step 1: Acquire realistic underwater degraded images and clear images; Step 2: Construct an all-optical diffraction neural network to establish an end-to-end optical mapping from the degraded image to the dehazed image; Step 3: Based on the Huygens-Fresnel diffraction principle, determine the physical propagation law of light between network layers, use a certain number of cascaded diffraction neural network layers as the passive encoder of the all-optical diffraction neural network, and use a single lens imaging structure with a fixed focal length as the passive decoder of the all-optical diffraction neural network. Step 4: Run the all-optical diffraction neural network using Python and PyTorch, and train it using supervised training and gradient descent. Step 5: Input the real underwater degraded image into the trained all-optical diffraction neural network to generate a dehazed image; The passive encoder includes a three-layer diffraction neural network cascade. Each diffraction neural network layer is capable of phase modulation. Each neuron is regarded as a secondary wave source during light propagation. According to the Huygens-Fresnel diffraction principle, light carries information and diffracts between neurons, thereby realizing the forward propagation of the network. A single lens imaging structure is used as a passive decoder of the all-optical diffraction neural network and cascaded after the passive encoder. The distance between the lens and the last diffraction neural network layer and the distance between the lens and the imaging receiving surface are the same as the focal length of the lens. The light field output by the last diffraction neural network layer is transformed by the passive decoder and presents a specific light spot pattern of a clear image after defogging on the imaging receiving surface. The loss function used for training with supervised training and gradient descent is the mean squared error, and the error calculation formula is as follows: ； in, It is a dehazed image. It is a clear image. It is the number of pixels in the image. When the training of the loss function converges and stabilizes, the final phase parameter is the optimized phase distribution of the diffraction layer.

2. The underwater image dehazing method based on an all-optical diffraction neural network according to claim 1, characterized in that, Running all-optical diffraction neural networks using Python and PyTorch includes: Write the main.py function as the main function of the project. By calling the function library, it can realize the reading of batch data and network elements, training loop, calculation of loss function, and output of loss value and interactive image. Write a function mydataset.py as the data input function to implement the preprocessing process of reading, cropping and normalizing underwater degraded images and their corresponding clear images; Write the nn_module.py function as the core function of the all-optical diffraction neural network, which includes phase modulation, Fresnel diffraction, lens and receiving screen classes; Write the eval.py function as the prediction function of the all-optical diffraction neural network. By reading the test dataset, as well as the trained network and parameters, it performs inference prediction, calculates the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) of the output result and the clear image, and saves the output result as a PNG image.

3. An underwater image dehazing system based on an all-optical diffraction neural network, used to implement the method described in any one of claims 1-2, characterized in that, include: Image acquisition module: Acquires both realistic underwater degraded images and clear images; Encoder module: Based on the Huygens-Fresnel diffraction principle, the physical propagation law of light between network layers is determined, and a certain number of cascaded diffraction neural network layers are used as passive encoders of all-optical diffraction neural networks; Decoder module: Uses a single lens imaging structure with a fixed focal length as a passive decoder for an all-optical diffraction neural network; Model training module: The all-optical diffraction neural network is run using Python and PyTorch programming, and supervised training and gradient descent are used for training.

Images