A thermal disturbance removal system for underwater images based on unsupervised learning
Through the unsupervised learning underwater image thermal perturbation removal system, the feature extraction and fusion modules are used to solve the problems of poor processing effects and missing labeled data sets in the prior art, and efficient image de-perturbation in real world environments is achieved Treatment, suitable for complex underwater environments.
Patent Information
- Application Number
- CN202411471261.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-22
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2044-10-22
AI Technical Summary
The existing underwater image thermal perturbation removal algorithm has poor processing effect, limited application scenarios, and cannot be processed in real time. The supervised learning method relies on labeled data sets, making it difficult to generalize to the real world environment.
Adopting an underwater image thermal perturbation removal system based on unsupervised learning, including feature extraction, feature fusion and image reconstruction modules, the network model VideoDeTurb is trained through unsupervised learning, and feature fusion and image reconstruction is used to use RNN and GRU for feature fusion and image reconstruction, using unsupervised loss function Model training is performed to eliminate dependence on the annotated dataset.
Real-time processing of thermal disturbances of underwater images is realized, the clarity and processing effect of images are improved, and can effectively generalize in complex and changeable real-world environments, and is suitable for still and motion shooting scenes.
Smart Images
Figure CN119027353B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of underwater image thermal disturbance removal, and in particular to an underwater image thermal disturbance removal system based on unsupervised learning. Background Art
[0002] Since water is an excellent coolant and nuclear radiation shielding material, in the design of modern nuclear power plants, important work processes and scenes such as nuclear fuel transfer, reactor cores and spent fuel pools are almost completely submerged in water. In the operation and maintenance of nuclear power plants, there are a large number of underwater applications and work scenes. Among them, underwater image thermal disturbance is an important technical difficulty and challenge in the underwater operation and maintenance of nuclear power plants.
[0003] Thermal disturbance of underwater images is an important challenge in underwater imaging of cameras. It can significantly degrade the video footage taken by the camera underwater, causing problems such as image distortion, blurring, and decreased contrast. This severely limits the availability and value of underwater images from cameras in application scenarios such as nuclear power plant observation and measurement.
[0004] Underwater thermal disturbances are mainly caused by two factors: the existence of temperature gradients. When there are temperature differences in the water, the densities of different temperature layers will also be different; the turbulent movement of the water body. The flow of the water body will aggravate the change of the temperature gradient, thereby aggravating the disturbance effect. These factors work together to cause the light to refract and scatter when passing through the water body, and the light propagation path is nonlinearly bent, causing the image to become blurred, distorted and distorted.
[0005] The underwater image thermal disturbance removal algorithm aims to improve the image blur and distortion problems caused by thermal disturbance through image algorithms. Most of the existing underwater thermal disturbance processing algorithms are based on atmospheric turbulence or improved turbulence models to model the underwater imaging process, and then use traditional image processing algorithms such as time domain mean filtering, bilateral filtering, lucky frame selection, etc. to restore the underwater thermal disturbance distorted images. Traditional algorithms face many difficulties and limitations in dealing with such problems, and they have the following obvious defects:
[0006] 1. Poor processing effect: Most of the existing algorithms are based on artificially designed simplified turbulence models, which work well on specific test sets. However, they do not work well when faced with the complex and changeable real underwater turbulence environment in the real world.
[0007] 2. Limited application scenarios: Most existing thermal disturbance removal algorithms require that the camera and the target remain stationary when shooting, which severely limits the application scenarios of the algorithms.
[0008] 3. Limited real-time processing capabilities: Most existing technologies focus on offline processing and cannot achieve real-time removal during underwater shooting.
[0009] In recent years, with the development of deep learning technology, deep learning has made great progress in the field of image processing. When using supervised learning to train underwater thermal disturbance image restoration models, there are the following problems:
[0010] 1. Missing labeled datasets: When using supervised learning to train network models, it is necessary to input "non-thermal disturbance images" that are paired with "thermal disturbance images" so that the network model can learn how to predict the correct output from the input data. Due to the complexity and unpredictability of the underwater turbulence environment, in the real world, labeled datasets of underwater thermal disturbance images, including pairs of thermal disturbance-non-thermal disturbance images, are almost impossible to obtain, resulting in the lack of labeled training datasets for underwater thermal disturbance removal models.
[0011] 2. Underwater thermal disturbance simulation is extremely difficult. Due to the extremely complex turbulent motion patterns of fluids, accurate mathematical models describing the turbulent motion mechanics have not yet been found in physics. The current main research method is to use finite element analysis to approximate the simulation of fluid motion. Due to the obvious differences between the image features of simulated data and the image turbulence features of the real world, the network model trained with simulated data is almost ineffective on complex and changeable real-world data sets and cannot be used for real-world image restoration. Summary of the invention
[0012] The present application provides an underwater image thermal disturbance removal system based on unsupervised learning to solve the problems in the above-mentioned background technology.
[0013] The technical solution adopted by the present application to solve the technical problem is: an underwater image thermal disturbance removal system based on unsupervised learning, comprising:
[0014] The underwater image thermal disturbance removal network model VideoDeTurb is obtained by training using an unsupervised learning training method. The underwater image thermal disturbance removal network model VideoDeTurb includes a feature extraction module, a feature fusion module and an image reconstruction module.
[0015] The feature extraction module is used to extract spatial features and temporal features from N video images taken continuously underwater by the camera, respectively, to obtain spatial features and temporal features, and to connect the extracted spatial features and temporal features using the concat operator and output them after 2D convolution processing to obtain an image feature map;
[0016] The feature fusion module is used to use the GRU gated recurrent unit in the RNN recurrent neural network to achieve effective feature fusion between video frames to obtain an image fusion feature map and transmit sequence information;
[0017] The image reconstruction module is used to reduce the dimension of the image fusion feature map and visualize the image to obtain an image with thermal disturbance removed.
[0018] This underwater image thermal disturbance removal system based on unsupervised learning uses an unsupervised learning training method, which eliminates the dependence of network model training on labeled data sets, and solves the problem of insufficient generalization ability of existing underwater image thermal disturbance removal algorithms, and the obvious lack of generalization ability, poor processing effect and missing labeled data sets when facing the complex and changeable underwater thermal disturbance environment in the real world.
[0019] Preferably, the number of video images N is adjusted by the user between 6-16 frames according to the severity of thermal disturbance in the actual underwater environment. When the thermal disturbance effect of the image is relatively mild, N can be selected as a smaller number. When the thermal disturbance effect of the image is relatively severe, N can be selected as a larger number.
[0020] Preferably, several 2D residual modules are used for spatial feature extraction to extract image spatial features in each frame, and the convolution kernel size of the 2D residual module is 3*3. Several 3D residual modules are used for temporal feature extraction between multiple frames, focusing on the differences and changes between video image frames. The convolution kernel size of the 3D residual module is 3*5*5. In order to improve computational efficiency, 2D convolution is used to downsample the feature map before 3D convolution calculation, and the feature map size is (N, 32, 128, 128).
[0021] Preferably, the feature fusion module includes fusion layer 1, fusion layer 2, fusion layer 3 and fusion layer 4, the input of the GRU unit of fusion layer 1 is the image feature map output by the feature extraction module and the hidden state update of the previous GRU unit, the input of the GRU units of fusion layers 2, 3, and 4 is the output of the GRU unit of the previous layer and the hidden state update of the previous GRU unit, and the hidden state update input of the first GRU cycle unit in each layer is the same as the feature map input.
[0022] Preferably, the image reconstruction module includes three 2D residual modules 1 and three 2D convolutions. The input of the image reconstruction module is the image fusion feature map output by the image fusion module and N video images taken continuously underwater by a camera used by the feature extraction module to extract features. The output is an image with thermal disturbance removed after model processing.
[0023] Preferably, the unsupervised learning training method comprises:
[0024] Step 1: Collect image frames of underwater thermal disturbance video and input them into VideoDeTurb network and PoseNet network respectively. The original images of adjacent frames of the current frame are , VideoDeTurb network output image frame ;
[0025] Step 2: PoseNet for image frames Estimate and return the relative camera pose [RT] between the current frame and its adjacent frames;
[0026] Step 3: According to the relative camera pose returned by PoseNet, perform perspective transformation on the 6 adjacent frames before and after the current frame, and combine them with the image frame Perform image reconstruction to obtain 6 images reconstructed by perspective transformation ;
[0027] Step 4: Use 6 reconstructed images As the learning target for the current frame, use Loss function for unsupervised model training.
[0028] As a preference, the reconstructed image is obtained include:
[0029] A. Get the intrinsic parameter matrix of the camera that captured the image , ,in, and are the focal lengths of the camera in the x-axis and y-axis directions of the pixel coordinate system, and It is the coordinate of the center of the image in the pixel coordinate system. VideoDeTurb and PoseNet are jointly learned and trained. The output of PoseNet obtains the relative pose matrix [RT] of the camera when shooting adjacent frames relative to the camera when shooting the current frame;
[0030] B. Get the projection matrix when the camera takes adjacent frames , ;
[0031] C. Obtain the homography matrix of adjacent frames , ;
[0032] D. Reconstructed image , .
[0033] Preferably, in step 4, The loss functions include:
[0034] ,
[0035] in, The definition is as follows,
[0036] ,
[0037] in, is the image frame at time t in the input video image sequence The output result after being processed by the network model VideoDeTurb,
[0038] They are The reconstructed image corresponding to the adjacent frames of
[0039] is the weighting coefficient, where The definition is as follows:
[0040] ,
[0041] In the input video image frame sequence, for any time t, define the corresponding image frame is the current frame, the current frame The six adjacent frames before and after are defined as its adjacent frames, which are recorded as , The definition is as follows:
[0042] ,
[0043] is the weighting coefficient, where The definition is as follows:
[0044] ,
[0045] Where x is the image and The difference between the corresponding pixels, is a positive constant, and the function CW-SSIM is the image complex wavelet structure similarity index, and its calculation formula is as follows:
[0046] ,
[0047] in, and Respectively, images and The complex wavelet transform of Indicates plural The complex conjugate of is a normal number, For the contextual loss of the image, use the loss function Perform unsupervised model training, when the loss function When the value of no longer decreases, the model training ends.
[0048] The substantial effects of this application are:
[0049] 1. This underwater image thermal disturbance removal system based on unsupervised learning first implements an efficient underwater image thermal disturbance removal network model VideoDeTurb. The network model VideoDeTurb realizes real-time processing of underwater thermal disturbance images and improves the image distortion and distortion problems caused by underwater thermal disturbances.
[0050] 2. This underwater image thermal disturbance removal system based on unsupervised learning implements an unsupervised training method for an underwater image thermal disturbance removal network model. It uses the implicit continuity and similarity of the content of continuous image frames of video taken by a camera in time and space as constraints to achieve unsupervised training of the network model VideoDeTurb, eliminating the dependence on labeled data sets. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 is a network structure diagram of a feature extraction module in an embodiment of the present application;
[0052] Figure 2 2D residual module network structure diagram of the feature extraction module of the embodiment of the present application;
[0053] Figure 3 is a 3D residual module network structure diagram of the feature extraction module of an embodiment of the present application;
[0054] Figure 4 is a network structure diagram of a feature fusion module in an embodiment of the present application;
[0055] Figure 5 is a network structure diagram of an image reconstruction module according to an embodiment of the present application;
[0056] Figure 6 This is a flow chart of unsupervised training using an embodiment of the present application;
[0057] Figure 7 It is a comparison diagram of images before and after processing by the embodiment of the present application. DETAILED DESCRIPTION
[0058] The technical solution of the present application is further described below through specific embodiments.
[0059] An underwater image thermal disturbance removal system based on unsupervised learning, an underwater image thermal disturbance removal network model VideoDeTurb trained by an unsupervised learning training method, the underwater image thermal disturbance removal network model VideoDeTurb comprising a feature extraction module, a feature fusion module and an image reconstruction module,
[0060] The feature extraction module is used to extract spatial features and temporal features from the N video images taken continuously underwater by the camera, and then connect the extracted spatial features and temporal features using the concat operator and output them after 2D convolution processing to obtain an image feature map, where the number of video images N is adjusted by the user between 6 and 16 frames according to the severity of thermal disturbance in the actual underwater environment. When the thermal disturbance effect of the image is relatively mild, N can be selected as a smaller number. When the thermal disturbance effect of the image is relatively severe, N can be selected as a larger number.
[0061] like Figure 1-3 As shown in the figure, five 2D residual modules are used to extract spatial features in each frame of the image, focusing on the spatial structural relationship between pixels in a single frame. The convolution kernel size of the 2D residual module is 3*3. Five 3D residual modules are used to extract temporal features between multiple frames of images, focusing on the differences and changes between video image frames. The convolution kernel size of the 3D residual module is 3*5*5. In order to improve the computational efficiency, the feature map is downsampled using 2D convolution before the 3D convolution calculation. The feature map size is (N, 32, 128, 128). The 2D convolution is the standard image Conv2D convolution, and the transposed 2D convolution is the standard image ConvTranspose2D transposed convolution. Finally, the spatial and temporal features extracted from the video image frame are connected using the concat operator and output after 2D convolution.
[0062] In this embodiment, the feature fusion module is an RNN recurrent neural network, which utilizes the GRU gated recurrent unit in the RNN recurrent neural network to achieve effective feature fusion and sequence information transmission between video frames. Figure 4 As shown in the figure, the feature fusion module uses a total of 4 layers of RNN feature fusion layers composed of GRU gated recurrent units. The feature fusion module takes advantage of the GRU to achieve efficient inter-frame information transfer and feature fusion when processing video sequences. This not only improves the model's de-thermal disturbance performance, but also simplifies the model structure and reduces the demand for computing resources.
[0063] The input of the GRU unit in fusion layer 1 is the image feature map output by the feature extraction module and the hidden state update of the previous GRU unit. The input of the GRU units in fusion layers 2, 3, and 4 is the output of the GRU unit in the previous layer and the hidden state update of the previous GRU unit. The hidden state update input of the first GRU recurrent unit in each layer is the same as the feature map input.
[0064] like Figure 5As shown in the figure, the image reconstruction module is responsible for dimensionality reduction and visualization image reconstruction of the image fusion feature map. The image reconstruction module consists of three 2D residual modules 1 and three 2D convolutions. The input is the output of the image fusion module (N, 64, 256, 256) and the original video frame image (N, 3, 256, 256). The module output is the image (N, 3, 256, 256) after model processing and the image thermal disturbance is removed.
[0065] It should be noted that if Figure 6 As shown, the unsupervised learning training method includes:
[0066] Step 1: Collect image frames of underwater thermal disturbance video and input them into VideoDeTurb network and PoseNet network respectively. The original images of adjacent frames of the current frame are , VideoDeTurb network output image frame ;
[0067] Step 2: PoseNet for image frames Estimate and return the relative camera pose [RT] between the current frame and its adjacent frames;
[0068] Step 3: According to the relative camera pose returned by PoseNet, perform perspective transformation on the 6 adjacent frames before and after the current frame, and combine them with the image frame Perform image reconstruction to obtain 6 images reconstructed by perspective transformation ;
[0069] Step 4: Use 6 reconstructed images As the learning target for the current frame, use Loss function for unsupervised model training.
[0070] Among them, the reconstructed image is obtained include:
[0071] A. Get the intrinsic parameter matrix of the camera that captured the image , ,in, and are the focal lengths of the camera in the x-axis and y-axis directions of the pixel coordinate system, and is the coordinate of the image center in the pixel coordinate system. VideoDeTurb and PoseNet are jointly learned and trained. The output of PoseNet is the relative pose matrix [RT] of the camera when shooting adjacent frames relative to the current frame. When the camera shoots adjacent frames, the rotation matrix relative to the current frame is R, which is a 3×3 rotation matrix. When the camera shoots adjacent frames, the relative translation vector relative to the current frame is T, which is a 3×1 translation vector.
[0072] B. Get the projection matrix when the camera takes adjacent frames , ;
[0073] C. Obtain the homography matrix of adjacent frames , ;
[0074] D. Reconstructed image , the original image of the adjacent frame of the current frame is , the image of the current frame after being processed by the network model VideoDeTurb is , the reconstructed image of the current frame reconstructed using adjacent frames is , .
[0075] In addition, in step 4, The loss functions include:
[0076] ,
[0077] in, The definition is as follows,
[0078] ,
[0079] in, is the image frame at time t in the input video image sequence The output result after being processed by the network model VideoDeTurb,
[0080] They are The reconstructed image corresponding to the adjacent frames of
[0081] is the weighting coefficient, where The definition is as follows:
[0082] ,
[0083] In the input video image frame sequence, for any time t, define the corresponding image frame is the current frame, the current frame The six adjacent frames before and after are defined as its adjacent frames, which are recorded as , The definition is as follows:
[0084] ,
[0085] is the weighting coefficient, The function is the image Charbonnier loss function, where The definition is as follows:
[0086] ,
[0087] Where x is the image and The difference between the corresponding pixels, is a positive constant, and the function CW-SSIM is the image complex wavelet structure similarity index, and its calculation formula is as follows:
[0088] ,
[0089] in, and Respectively, images and The complex wavelet transform of Indicates plural The complex conjugate of is a normal number, For the contextual loss of the image, use the loss function Perform unsupervised model training, when the loss function When the value of no longer decreases, the model training ends.
[0090] The present invention solves the problem of missing labeled data sets when training underwater image thermal disturbance removal models for deep learning, eliminates the reliance on labeled data sets, and enables network model training to proceed. It also implements a neural network model VideoDeTurb based on unsupervised removal of underwater image thermal disturbances. Compared with thermal disturbance removal algorithms based on artificial feature design, the network model VideoDeTurb algorithm of the present invention is significantly superior to previous algorithm implementations in terms of image processing effects, algorithm robustness, and environmental generalization capabilities. At the same time, the VideoDeTurb algorithm can not only process camera still shooting scenes, but also camera motion shooting scenes. Figure 7 As shown, the contrast effect is removed by using this embodiment, wherein the upper figure is the original image of underwater thermal disturbance image acquisition, and the lower figure is the image after algorithm processing and thermal disturbance removal, and the clarity is significantly improved.
Claims
1. An underwater image thermal disturbance removal system based on unsupervised learning, characterized in that: include: The underwater image thermal disturbance removal network model VideoDeTurb is obtained by training using an unsupervised learning training method. The underwater image thermal disturbance removal network model VideoDeTurb includes a feature extraction module, a feature fusion module and an image reconstruction module. The feature extraction module is used to extract spatial features and temporal features from N video images taken continuously underwater by the camera, respectively, to obtain spatial features and temporal features, and to connect the extracted spatial features and temporal features using the concat operator and output them after 2D convolution processing to obtain an image feature map; The feature fusion module is used to use the GRU gated recurrent unit in the RNN recurrent neural network to achieve effective feature fusion between video frames to obtain an image fusion feature map and transmit sequence information; The image reconstruction module is used to reduce the dimension of the image fusion feature map and reconstruct the visual image to obtain an image with the thermal disturbance of the image removed; Unsupervised learning training methods include: Step 1: Collect image frames of underwater thermal disturbance video and input them into VideoDeTurb network and PoseNet network respectively. The original images of adjacent frames of the current frame are , VideoDeTurb network output image frame ; Step 2: PoseNet for image frames Estimate and return the relative camera pose [RT] between the current frame and its adjacent frames; Step 3: According to the relative camera pose returned by PoseNet, perform perspective transformation on the 6 adjacent frames before and after the current frame, and combine them with the image frame Perform image reconstruction to obtain 6 images reconstructed by perspective transformation ; Step 4: Use 6 reconstructed images As the learning target for the current frame, use Loss function for unsupervised model training; Get the reconstructed image include: A. Get the intrinsic parameter matrix of the camera that captured the image , ,in, and are the focal lengths of the camera in the x-axis and y-axis directions of the pixel coordinate system, and It is the coordinate of the center of the image in the pixel coordinate system. VideoDeTurb and PoseNet are jointly learned and trained. The output of PoseNet obtains the relative pose matrix [RT] of the camera when shooting adjacent frames relative to the camera when shooting the current frame; B. Get the projection matrix when the camera takes adjacent frames , ; C. Obtain the homography matrix of adjacent frames , ; D. Reconstructed image , ; In step four, The loss functions include: ， in, The definition is as follows, ， in, is the image frame at time t in the input video image sequence The output result after being processed by the network model VideoDeTurb, They are The reconstructed image corresponding to the adjacent frames of is the weighting coefficient, where The definition is as follows: ， In the input video image frame sequence, for any time t, define the corresponding image frame is the current frame, the current frame The six adjacent frames before and after are defined as its adjacent frames, which are recorded as , The definition is as follows: ， is the weighting coefficient, where The definition is as follows: ， Where x is the image and The difference between the corresponding pixels, is a positive constant, and the function CW-SSIM is the image complex wavelet structure similarity index, and its calculation formula is as follows: ， in, and Respectively, images and The complex wavelet transform of Indicates plural The complex conjugate of is a normal number, For the contextual loss of the image, use the loss function Perform unsupervised model training, when the loss function When the value of no longer decreases, the model training ends.
2. According to the underwater image thermal disturbance removal system based on unsupervised learning according to claim 1, it is characterized in that: The number of video images N is adjusted by the user between 6 and 16 frames according to the severity of thermal disturbance in the actual underwater environment. When the thermal disturbance effect of the image is relatively mild, N can be selected as a smaller number. When the thermal disturbance effect of the image is relatively severe, N can be selected as a larger number.
3. The underwater image thermal disturbance removal system based on unsupervised learning according to claim 1, characterized in that: When extracting spatial features, several 2D residual modules are used to extract the spatial features of each frame. When extracting temporal features, several 3D residual modules are used to extract temporal features between multiple frames, focusing on the differences and changes between video image frames.
4. The underwater image thermal disturbance removal system based on unsupervised learning according to claim 1, characterized in that: The feature fusion module includes fusion layer 1, fusion layer 2, fusion layer 3 and fusion layer 4. The input of the GRU unit of fusion layer 1 is the image feature map output by the feature extraction module and the hidden state update of the previous GRU unit. The input of the GRU units of fusion layers 2, 3, and 4 is the output of the GRU unit of the previous layer and the hidden state update of the previous GRU unit. The hidden state update input of the first GRU cycle unit in each layer is the same as the feature map input.
5. The underwater image thermal disturbance removal system based on unsupervised learning according to claim 1, characterized in that: The image reconstruction module includes three 2D residual modules 1 and three 2D convolutions. The input of the image reconstruction module is the image fusion feature map output by the image fusion module and N video images taken continuously underwater by a camera used to extract features by the feature extraction module. The output is an image with thermal disturbance removed after model processing.