Underwater image enhancement method and device
Through the combination method of neural network based on attention mechanism and variational automatic encoder, the problem of poor underwater image enhancement effect is solved, efficient enhancement of underwater images is achieved, and image processing is suitable for complex underwater environments.
Patent Information
- Application Number
- CN202411197219.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-29
AI Technical Summary
The prior art cannot effectively improve the enhancement effect of underwater images, especially in issues such as blurring and color distortion, and the generalization ability is insufficient, so it cannot be applied to specific scenarios in underwater environments.
By obtaining the underwater image to be enhanced and encoding it into a low-dimensional feature vector, using a neural network model based on attention mechanism for target mapping, combining with a variational automatic encoder for encoding and decoding optimization, the enhanced underwater image is generated, and the enhanced underwater image is improved Image quality and clarity.
It improves the visual effect of underwater images, enhances the contrast and clarity of images, improves the problem of color distortion, improves the quality and speed of image enhancement, and is suitable for image processing in different underwater environments.
Smart Images

Figure CN118710528B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to an underwater image enhancement method and an underwater image enhancement device. Background Art
[0002] Underwater image enhancement has a very broad application scenario. For many underwater operations, such as underwater robot operations, the visual system is required to obtain relevant information about the surrounding environment, just like the human eye. Among them, the visual system based on optical vision is a more attractive way of perception. However, due to the complexity of the underwater environment, the refraction and absorption of light, and the influence of factors such as suspended matter in the water, the underwater images obtained have poor imaging effects, severe degradation, low contrast, blur and distortion, etc., and underwater operations that rely on visual perception face huge challenges. Therefore, it is of great significance to enhance underwater images.
[0003] There are many traditional methods for image enhancement. Among them, the spatial domain image processing method processes the pixels of the image and uses grayscale mapping as the basis to improve the grayscale level, such as histogram equalization, grayscale world hypothesis, etc.; the image can also be denoised by filtering, such as median filtering, mean filtering, etc.; frequency domain image enhancement can indirectly enhance the image through various frequency domain transformations, such as Fourier transform, wavelet transform, etc., or combine spatial domain and frequency domain methods for image enhancement. Traditional image enhancement algorithms can improve image blur and enhance image details to a certain extent, but they do not provide more targeted enhancement processing for the specific underwater scene.
[0004] In the prior art, for example, the patent disclosure document with publication number CN117011194A mainly focuses on image enhancement under low-light conditions, but does not take into account the particularity of the underwater environment, and cannot simultaneously solve the problems of image blurring and color distortion. The generalization ability of this method is not strong enough. When the input low-quality image is different from the training sample, the enhancement effect will decrease. For another example, the patent disclosure document with publication number CN116993621A discloses a relatively limited direction of image enhancement, which has a better effect in low light conditions, but has poor enhancement effects for other problems of the image, and does not take into account the contextual information in the image during the processing process. Therefore, the image still has many problems such as high noise, low clarity, and color distortion.
[0005] It can be seen that the image augmentation technology in the prior art still has limitations such as color distortion and cannot be applied to underwater images. Therefore, how to effectively improve the underwater image enhancement effect has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention
[0006] The present invention provides an underwater image enhancement method and an underwater image enhancement device, which solve the problem in the related art that the underwater image enhancement effect cannot be improved.
[0007] As a first aspect of the present invention, a method for underwater image enhancement is provided, which includes: acquiring an underwater image to be enhanced, and encoding the underwater image to be enhanced through a first image enhancement encoder to obtain a low-dimensional feature vector to be enhanced, wherein the first image enhancement encoder is obtained after optimization according to first encoding and decoding optimization data, and the first encoding and decoding optimization data is obtained after encoding and decoding respectively according to a data set of an image to be enhanced and a degraded image data set, and the degraded image data set is obtained after degradation processing of a target image data set; inputting the low-dimensional feature vector to be enhanced into a target mapping network to obtain a target quality low-dimensional feature vector, wherein the target mapping network is to use the first encoding and decoding process data as an attention-based The input data of the neural network model of the attention mechanism and the second encoding and decoding process data are used as the output data of the neural network model based on the attention mechanism, and the neural network model based on the attention mechanism is trained to obtain the data, the first encoding and decoding process data are obtained after encoding according to the image data set to be enhanced and the degraded image data set respectively, and the second encoding and decoding process data are obtained after encoding according to the target image data set; the target quality low-dimensional feature vector is decoded by the second image enhancement encoder to obtain the enhanced underwater image, the second image enhancement encoder is obtained after optimization according to the second encoding and decoding optimization data, and the second encoding and decoding optimization data is obtained after optimization according to the target image data set.
[0008] Furthermore, the first image enhancement encoder is obtained after optimization according to first encoding and decoding optimization data, including: obtaining an image data set to be enhanced and a target image data set; performing degradation processing on the target image data set to obtain a degraded image data set; performing encoding and decoding processing on the image data set to be enhanced and the degraded image data set respectively through a first variational autoencoder to obtain first encoding and decoding optimization data; and optimizing the current first image enhancement encoder to be optimized according to the first encoding and decoding optimization data to obtain a first image enhancement encoder.
[0009] Furthermore, the image data set to be enhanced and the degraded image data set are respectively encoded and decoded by a first variational autoencoder to obtain first encoding and decoding optimization data, including: encoding the image data set to be enhanced and the degraded image data set by a first variational autoencoder to obtain a low-dimensional feature vector set to be enhanced and a degraded low-dimensional feature vector set; decoding the low-dimensional feature vector set to be enhanced and the degraded low-dimensional feature vector set to obtain a decoded image data set to be enhanced and a decoded degraded image data set; constructing constraints so that the low-dimensional feature vector set to be enhanced and the degraded low-dimensional feature vector set both satisfy Gaussian distribution, and constructing a first loss function according to the constraints; constructing a second loss function according to the error between the image data set to be enhanced and the decoded image data set to be enhanced and the error between the degraded image data set and the decoded degraded image data set; optimizing the first loss function and the second loss function to obtain first encoding and decoding optimization data.
[0010] Further, performing degradation processing on the target image dataset to obtain a degraded image dataset includes: performing degradation processing on the target image dataset according to a Gaussian degradation model to obtain a degraded image dataset, and the expression of the degradation processing is: ,in, represents the degraded image dataset, represents the target image dataset, represents the degradation noise, represents the Gaussian degradation model, ,in, represents the normalization constant, express The circular support domain.
[0011] Furthermore, the second image enhancement encoder is obtained after optimization according to second encoding and decoding optimization data, including: encoding and decoding the target image data set through a second variational autoencoder to obtain second encoding and decoding optimization data; optimizing the second image enhancement encoder to be optimized according to the second encoding and decoding optimization data to obtain a second image enhancement encoder.
[0012] Furthermore, the target image data set is encoded and decoded by a second variational autoencoder to obtain second encoding and decoding optimization data, including: encoding the target image data set by a second variational autoencoder to obtain a target low-dimensional feature vector set; decoding the target low-dimensional feature vector set to obtain a decoded target image data set; constructing a third loss function based on the error between the target image data set and the decoded target image data set; optimizing the third loss function to obtain second encoding and decoding optimization data.
[0013] Furthermore, the target mapping network is obtained by using the first encoding and decoding process data as input data of a neural network model based on an attention mechanism and using the second encoding and decoding process data as output data of a neural network model based on an attention mechanism and training the neural network model based on the attention mechanism, including: encoding the image data set to be enhanced and the degraded image data set through a first variational autoencoder to obtain a low-dimensional feature vector set to be enhanced and a degraded low-dimensional feature vector set, respectively, and the low-dimensional feature vector set to be enhanced and the degraded low-dimensional feature vector set are used as the first encoding and decoding process data; encoding the target image data set through a second variational autoencoder to obtain a target low-dimensional feature vector set, and the target low-dimensional feature vector set is used as the second encoding and decoding process data; inputting the first encoding and decoding process data as input data into a neural network model based on an attention mechanism to obtain a predicted feature vector data set; constructing a mapping network loss function according to the predicted feature vector data set and the second encoding and decoding process data; optimizing the mapping network loss function to obtain a target mapping network.
[0014] Furthermore, a mapping network loss function is constructed based on the predicted feature vector data set and the second encoding and decoding process data, including: performing mean square error calculation based on the predicted feature vector data set and the target low-dimensional feature vector set to obtain a fourth loss function; decoding the predicted feature vector data set through a second variational autoencoder to obtain a predicted image data set, and decoding the target low-dimensional feature vector set through the second variational autoencoder to obtain a decoded target image data set; performing mean absolute error calculation based on the predicted image data set and the decoded target image data set to obtain a fifth loss function; and constructing the mapping network loss function based on the fourth loss function and the fifth loss function.
[0015] Furthermore, the expression of the mapping network loss function is: ,in, represents the fourth loss function, , represents the fifth loss function, .
[0016] As another aspect of the present invention, an underwater image enhancement device is provided for implementing the underwater image enhancement method described above, which includes: a coding module for acquiring an underwater image to be enhanced, and encoding the underwater image to be enhanced through a first image enhancement encoder to obtain a low-dimensional feature vector to be enhanced, wherein the first image enhancement encoder is obtained after optimization according to first coding optimization data, and the first coding optimization data is obtained after coding and decoding respectively according to a data set of an image to be enhanced and a degraded image data set, and the degraded image data set is obtained after degradation processing of a target image data set; a mapping module for inputting the low-dimensional feature vector to be enhanced into a target mapping network to obtain a target quality low-dimensional feature vector, wherein the target mapping network is to input the first coding optimization data ... The encoding process data is used as the input data of the neural network model based on the attention mechanism, and the second encoding and decoding process data is used as the output data of the neural network model based on the attention mechanism, and the neural network model based on the attention mechanism is trained to obtain the data, the first encoding and decoding process data is obtained after encoding according to the image data set to be enhanced and the degraded image data set respectively, and the second encoding and decoding process data is obtained after encoding according to the target image data set; a decoding module is used to decode the target quality low-dimensional feature vector through a second image enhancement encoder to obtain an enhanced underwater image, the second image enhancement encoder is obtained after optimization according to the second encoding and decoding optimization data, and the second encoding and decoding optimization data is obtained after optimization according to the target image data set.
[0017] The underwater image enhancement method provided by the present invention is to encode the underwater image to be enhanced through a first image enhancement encoder, and then obtain a target quality low-dimensional feature vector through a target mapping network, and finally decode it through a second image enhancement encoder to obtain an enhanced underwater image. In the underwater image enhancement method, since the first image enhancement encoder is specifically obtained by performing encoding and decoding optimization on the image data set to be enhanced and the degraded image data set respectively, the second image enhancement encoder is specifically obtained after optimizing the second encoding and decoding optimization data, the target mapping network is specifically obtained by training a neural network model based on an attention mechanism, and the input data of the neural network model based on the attention mechanism is specifically the first encoding and decoding process data, and the output data is specifically the second encoding and decoding process data, and the first encoding and decoding process data is specifically the data obtained after encoding the image data set to be enhanced and the degraded image data set respectively, and the second encoding and decoding process data is specifically the data obtained after encoding the target image data set. Therefore, the underwater image enhancement method of the present invention generates a degraded image by performing degradation processing on the high-quality target image, and at the same time performs the same encoding and decoding processing on the degraded image and the original image with poor quality, which can effectively narrow the difference between the two in a low-dimensional space. In addition, by performing additional encoding and decoding processing on the high-quality target image and training the target mapping network through the process data of the two encoding and decoding processes, the trained target mapping network can restore high-quality data that meets the quality requirements, thereby improving the quality and speed of image enhancement. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.
[0019] Figure 1 The present invention provides a flow chart of the underwater image enhancement method.
[0020] Figure 2 A flow chart of a method for obtaining the first image enhancement encoder provided by the present invention.
[0021] Figure 3 A flow chart of a method for obtaining first encoding and decoding optimization data provided by the present invention.
[0022] Figure 4 The flowchart of the first variational autoencoder provided by the present invention.
[0023] Figure 5 A flow chart of a method for obtaining the second image enhancement encoder provided by the present invention.
[0024] Figure 6 A flow chart of a method for obtaining second encoding and decoding optimization data provided by the present invention.
[0025] Figure 7 A flow chart of the method for obtaining a target mapping network provided by the present invention.
[0026] Figure 8 This is a schematic diagram of the network structure of ECANet provided by the present invention.
[0027] Fig. 9 A flow chart of the method for constructing a mapping network loss function provided by the present invention.
[0028] Fig.10a A schematic diagram of an underwater image to be enhanced provided by the present invention.
[0029] Fig.10b for Fig.10a The corresponding enhanced image effect diagram.
[0030] Fig.11a A schematic diagram of another underwater image to be enhanced provided by the present invention.
[0031] Fig.11b for Fig.11a The corresponding enhanced image effect diagram.
[0032] Fig.12 This is a structural block diagram of the underwater image enhancement device provided by the present invention. DETAILED DESCRIPTION
[0033] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0034] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.
[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0036] There are huge differences between underwater and terrestrial environments. First, underwater, especially in deep water, mainly relies on artificial light sources for illumination, and the absorption and scattering of light by water leads to limited illumination range and insufficient clarity, making it difficult to extract underwater image features and low feature matching. Secondly, when the submersible is operating or observing near the bottom, the disturbance of the body or propeller will lift up the sediments on the seabed. These lifted sediments will cause image information loss, blurring and other problems. Due to the existence of the above problems, the quality of underwater image data is greatly affected. In addition, due to the diversity of underwater environments, image enhancement technology can effectively solve data problems in different water environments. This requires image enhancement technology to have a relatively strong generalization ability and to effectively and reasonably solve existing image problems according to the characteristics of different scenes. Through underwater image enhancement technology, the overall or local characteristics of the image are purposefully emphasized, the color, brightness and contrast of the image are improved, the originally unclear image becomes clear or certain features of interest are emphasized, the difference between the features of different objects in the image is enlarged, the features of no interest are suppressed, and the visual effect of underwater images is improved.
[0037] Based on this, an underwater image enhancement method is provided in this embodiment. Figure 1 is a flow chart of an underwater image enhancement method provided according to an embodiment of the present invention. Figure 1 As shown, it includes: S100, acquiring an underwater image to be enhanced, and encoding the underwater image to be enhanced through a first image enhancement encoder to obtain a low-dimensional feature vector to be enhanced, wherein the first image enhancement encoder is obtained after optimization according to first encoding and decoding optimization data, and the first encoding and decoding optimization data is obtained after encoding and decoding respectively according to a data set of an image to be enhanced and a degraded image data set, and the degraded image data set is obtained after degradation processing is performed on a target image data set.
[0038] In the embodiment of the present invention, the underwater image to be enhanced is firstly encoded to be encoded into a low-dimensional feature vector for processing, thereby effectively improving the processing efficiency and speed.
[0039] Specifically, when encoding, it can be implemented by a first image enhancement encoder, which can be obtained by performing encoding and decoding optimization on the image data set to be enhanced and the degraded image data set, respectively, wherein the degraded image data set can be obtained by performing degradation processing on the target image data set.
[0040] S200. Input the low-dimensional feature vector to be enhanced into the target mapping network to obtain a target quality low-dimensional feature vector, wherein the target mapping network uses the first encoding and decoding process data as the input data of the neural network model based on the attention mechanism and the second encoding and decoding process data as the output data of the neural network model based on the attention mechanism, and trains the neural network model based on the attention mechanism, wherein the first encoding and decoding process data is obtained by encoding the image data set to be enhanced and the degraded image data set respectively, and the second encoding and decoding process data is obtained by encoding the target image data set.
[0041] In an embodiment of the present invention, the low-dimensional feature vector to be enhanced obtained after the above encoding is input into the target mapping network, and the target quality low-dimensional feature vector can be obtained through the target mapping network, that is, the target mapping network can obtain the target quality low-dimensional feature vector that meets the requirements after processing the input low-dimensional feature vector to be enhanced. Specifically, since the target mapping network is obtained by training a neural network model based on an attention mechanism, and the input data of the neural network model based on the attention mechanism is specifically the first encoding and decoding process data, and the output data is specifically the second encoding and decoding process data, and the first encoding and decoding process data is specifically the data obtained after encoding the image data set to be enhanced and the degraded image data set, respectively, and the second encoding and decoding process data is specifically the data obtained after encoding the target image data set, by using the first encoding and decoding process data as input data and the second encoding and decoding process data as output data to train the neural network model based on the attention mechanism, a target quality low-dimensional feature vector that meets the requirements can be obtained.
[0042] S300. Decode the target quality low-dimensional feature vector through a second image enhancement encoder to obtain an enhanced underwater image, wherein the second image enhancement encoder is obtained after optimization according to second encoding and decoding optimization data, and the second encoding and decoding optimization data is obtained after encoding and decoding according to the target image data set.
[0043] In the embodiment of the present invention, the target quality low-dimensional feature vector obtained above is decoded by a second image enhancement encoder to obtain an enhanced underwater image that meets the target quality.
[0044] It should be understood that the second image enhancement encoder is specifically obtained after optimization according to the second encoding and decoding optimization data, and the second encoding and decoding optimization data is specifically optimized data after encoding and decoding the target image data set.
[0045] It should be noted that the target image data set in the embodiment of the present invention may specifically include a plurality of ideal flawless high-quality target images, the image data set to be enhanced specifically includes a plurality of real original images of poor quality, and the degraded image data set specifically includes a plurality of degraded images after degradation processing of high-quality target images.
[0046] In summary, the underwater image enhancement method provided by the present invention is to encode the underwater image to be enhanced through the first image enhancement encoder, and then obtain the target quality low-dimensional feature vector through the target mapping network, and finally decode it through the second image enhancement encoder to obtain the enhanced underwater image. In the underwater image enhancement method, since the first image enhancement encoder is specifically obtained by respectively performing encoding and decoding optimization on the image data set to be enhanced and the degraded image data set, the second image enhancement encoder is specifically obtained after optimizing the second encoding and decoding optimization data, the target mapping network is specifically obtained by training a neural network model based on an attention mechanism, and the input data of the neural network model based on the attention mechanism is specifically the first encoding and decoding process data, and the output data is specifically the second encoding and decoding process data, and the first encoding and decoding process data is specifically the data obtained after encoding the image data set to be enhanced and the degraded image data set, and the second encoding and decoding process data is specifically the data obtained after encoding the target image data set. Therefore, the underwater image enhancement method of the present invention generates a degraded image by performing degradation processing on the high-quality target image, and at the same time performs the same encoding and decoding processing on the degraded image and the original image with poor quality, which can effectively narrow the difference between the two in a low-dimensional space. In addition, by performing additional encoding and decoding processing on the high-quality target image and training the target mapping network through the process data of the two encoding and decoding processes, the trained target mapping network can restore high-quality data that meets the quality requirements, thereby improving the quality and speed of image enhancement.
[0047] In the embodiment of the present invention, the first image enhancement encoder is obtained after being optimized according to the first codec optimization data, such as Figure 2 As shown, it includes: S110, obtaining a to-be-enhanced image dataset and a target image dataset.
[0048] Specifically, the image data set to be enhanced is an image set formed by a plurality of real original images of poor quality, and the target image data set is an image set formed by a plurality of ideal, flawless, high-quality target images.
[0049] S120: Perform degradation processing on the target image data set to obtain a degraded image data set.
[0050] In an embodiment of the present invention, the above-mentioned high-quality target image is subjected to degradation processing to obtain a degraded image. Here, the degraded image data set is specifically an image set formed by a plurality of degraded images after the high-quality target image is subjected to degradation processing.
[0051] Specifically, performing degradation processing on the target image data set to obtain a degraded image data set includes: performing degradation processing on the target image data set according to a Gaussian degradation model to obtain a degraded image data set, wherein the degradation processing expression is: ,in, , , represents the degraded image dataset, represents the target image dataset, represents the degradation noise, represents the Gaussian degradation model, ,in, represents the normalization constant, express The circular support domain.
[0052] S130 , respectively performing encoding and decoding processing on the to-be-enhanced image dataset and the degraded image dataset through a first variational autoencoder to obtain first encoding and decoding optimized data.
[0053] In an embodiment of the present invention, both the image dataset to be enhanced and the degraded image dataset are encoded and decoded by a first variational autoencoder. Specifically, the image dataset to be enhanced and the degraded image dataset are encoded and decoded by a first variational autoencoder to obtain first encoding and decoding optimized data, such as Figure 3 As shown, it includes: S131, encoding the image data set to be enhanced and the degraded image data set through the first variational autoencoder respectively to obtain a low-dimensional feature vector set to be enhanced and a degraded low-dimensional feature vector set; in an embodiment of the present invention, the image data set to be enhanced is encoded through the first variational autoencoder to obtain the low-dimensional feature vector set to be enhanced, and the degraded image data set is encoded through the first variational autoencoder to obtain the degraded low-dimensional feature vector set.
[0054] S132: Decode the low-dimensional feature vector set to be enhanced and the degraded low-dimensional feature vector set respectively to obtain a decoded image data set to be enhanced and a decoded degraded image data set.
[0055] Specifically, the low-dimensional feature vector set to be enhanced and the degraded low-dimensional feature vector set are decoded by a first variational autoencoder to obtain a decoded image data set to be enhanced and a decoded degraded image data set.
[0056] like Figure 4 The following is the workflow diagram of the first variational autoencoder. Specifically, the same variational autoencoder is used to act on the degraded image at the same time. and real low-quality images , map them into feature vectors of the low-dimensional space of Gaussian distribution, that is, the low-dimensional feature vector to be enhanced and the degenerate low-dimensional feature vector , and then decode them into and .
[0057] S133, constructing constraint conditions so that both the to-be-enhanced low-dimensional feature vector set and the degraded low-dimensional feature vector set satisfy Gaussian distribution, and constructing a first loss function according to the constraint conditions.
[0058] In the embodiment of the present invention, the low-dimensional feature vector to be enhanced in the low-dimensional space and the degenerate low-dimensional feature vector Fits a Gaussian distribution: .
[0059] To minimize , assuming The distribution is normal, and For a standard normal distribution, calculate the KL divergence between two normal distributions: ,in, express The standard deviation of express The mean of .
[0060] When the first variational autoencoder receives When , the first variational autoencoder does not directly output the result , but will output The mean and standard deviation of the distribution of , so the above formula can be used as the loss function to update the encoder parameters.
[0061] Based on this, the first loss function is defined as: .
[0062] When training the decoder of the first variational autoencoder, random sampling is performed from the standard normal distribution, multiplied by the variance obtained above, and added to the mean above to construct the input of the encoder, so that the decoder in the first variational autoencoder has better stability.
[0063] S134. Construct a second loss function according to an error between the to-be-enhanced image dataset and the decoded to-be-enhanced image dataset and an error between the degraded image dataset and the decoded degraded image dataset.
[0064] In the embodiment of the present invention, by minimizing the mean square error between the input and the output, the value decoded by the first variational autoencoder can be As close as possible to the encoder input Close to, that is, define the second loss function as: .
[0065] S135. Optimize the first loss function and the second loss function to obtain first encoding and decoding optimization data.
[0066] By optimizing the above first loss function and the second loss function, the total loss function of the first image enhancement encoder can be obtained as follows: .
[0067] S140. Optimize the first image enhancement encoder to be optimized according to the first encoding and decoding optimization data to obtain a first image enhancement encoder.
[0068] Due to the degraded image Compared with the real low-quality image Sharing a variational autoencoder, the first encoding and decoding optimization data of the two can gradually bring the latent spaces of the two closer. The above total loss function can continuously narrow the difference between the two in the latent space, and improve the generalization ability and stability of the first image enhancement encoder.
[0069] In the embodiment of the present invention, the second image enhancement encoder is obtained after being optimized according to the second codec optimization data, such as Figure 5 As shown, it includes: S310, encoding and decoding the target image data set through a second variational autoencoder to obtain second encoding and decoding optimized data.
[0070] It should be understood that the encoding and decoding processing of the target image data set by the second variational autoencoder is based on the same principle as the first variational autoencoder. Here, it is still necessary to obtain the second encoding and decoding optimization data through the encoding and decoding processing of the target image data set, so as to optimize the second image enhancement encoder through the second encoding and decoding optimization data.
[0071] Specifically, the target image data set is encoded and decoded by a second variational autoencoder to obtain second encoding and decoding optimized data, such as Figure 6As shown, it includes: S311, encoding the target image data set through a second variational autoencoder to obtain a target low-dimensional feature vector set.
[0072] In the embodiment of the present invention, the target image data set specifically includes a plurality of high-quality target images , multiple target images are encoded through the second variational autoencoder to obtain a target low-dimensional feature vector set.
[0073] S312: Decode the target low-dimensional feature vector set to obtain a decoded target image data set.
[0074] The target low-dimensional feature vector set is decoded by the second variational autoencoder to obtain a decoded target image dataset.
[0075] Specifically, multiple high-quality target images The target low-dimensional feature vector is obtained by mapping it to the latent space through the second variational autoencoder ,and It conforms to the standard normal distribution, and then through the decoding function of the second variational autoencoder, it can be well restored to the decoded target image data .
[0076] S313: construct a third loss function according to the error between the target image data set and the decoded target image data set.
[0077] Here, the third loss function is constructed for the error between the above target image dataset and the decoded target image dataset: ,in, represents the standard normal distribution.
[0078] S314: Optimize the third loss function to obtain second encoding and decoding optimization data.
[0079] By optimizing the third loss function, second encoding and decoding optimization data can be obtained, and optimization processing of the second image enhancement encoder can be achieved based on the second encoding and decoding optimization data.
[0080] S320: Optimize the second image enhancement encoder to be optimized according to the second encoding and decoding optimization data to obtain a second image enhancement encoder.
[0081] In an embodiment of the present invention, the target mapping network is obtained by using the first encoding and decoding process data as input data of a neural network model based on an attention mechanism and using the second encoding and decoding process data as output data of a neural network model based on an attention mechanism and training the neural network model based on the attention mechanism, such as Figure 7As shown, it includes: S210, encoding the image data set to be enhanced and the degraded image data set through a first variational autoencoder to obtain a low-dimensional feature vector set to be enhanced and a degraded low-dimensional feature vector set, and the low-dimensional feature vector set to be enhanced and the degraded low-dimensional feature vector set are used as the first encoding and decoding process data.
[0082] In an embodiment of the present invention, the low-dimensional feature vector set to be enhanced obtained after the above-mentioned image data set is encoded by the first variational autoencoder and the degraded low-dimensional feature vector set obtained after the degraded image data set is encoded by the first variational autoencoder is used as the first encoding and decoding process data for training the target mapping network.
[0083] S220. Encode the target image data set through a second variational autoencoder to obtain a target low-dimensional feature vector set, and use the target low-dimensional feature vector set as second encoding and decoding process data.
[0084] In an embodiment of the present invention, the target low-dimensional feature vector set obtained after the target image data set is encoded by the second variational autoencoder is used as the second encoding and decoding process data for training the target mapping network.
[0085] S230: Input the first encoding and decoding process data as input data into a neural network model based on an attention mechanism to obtain a predicted feature vector data set.
[0086] In the embodiment of the present invention, a target mapping network is constructed, and specifically an ECANet network structure based on an attention mechanism is used to complete the task of mapping the target feature vector from the low-dimensional feature vector to be enhanced. To the target low-dimensional feature vector The mapping, such as Figure 8 The figure shows the network structure of ECANet. The channel attention mechanism is introduced in the convolution operation to improve the feature representation capability. The weights of the channel features are adaptively adjusted so that the network can better focus on important features and suppress unimportant features. Through this mechanism, the representation capability of the network is effectively enhanced while avoiding the addition of too many parameters and computational costs. In this process, the low-dimensional feature vector to be enhanced is Generate prediction feature vector data through this network , and expects to predict the feature vector data and the target low-dimensional feature vector Getting closer.
[0087] S240: construct a mapping network loss function according to the predicted feature vector data set and the second encoding and decoding process data.
[0088] In an embodiment of the present invention, a mapping network loss function is constructed according to the predicted feature vector data set and the second encoding and decoding process data, such as Fig. 9 As shown, it includes: S241, calculating the mean square error between the predicted feature vector data set and the target low-dimensional feature vector set to obtain a fourth loss function.
[0089] S242. Decode the predicted feature vector data set through a second variational autoencoder to obtain a predicted image data set, and decode the target low-dimensional feature vector set through the second variational autoencoder to obtain a decoded target image data set.
[0090] S243. Calculate the mean absolute error between the predicted image data set and the decoded target image data set to obtain a fifth loss function.
[0091] S244. Construct the mapping network loss function according to the fourth loss function and the fifth loss function.
[0092] Specifically, the expression of the mapping network loss function is: ,in, represents the fourth loss function, , represents the fifth loss function, .
[0093] It should be understood that represents the output of the ECANet network, represents the high-quality variables in the latent space, that is, The predicted and true values of the model output in the sample low-dimensional space, is the number of samples. Acting in the low-dimensional space, the mapping generates The second variational decoder code acts on Encoded , so that You can also use The decoding process recovers a good . calculate The variables recovered by the second image enhancement encoder with by Restored variables The gap between It can also recover well .
[0094] S250: Optimize the mapping network loss function to obtain a target mapping network.
[0095] It should be understood that in the above process, the real low-quality image In low-dimensional space Keep getting closer, and Is Degenerate synthesis can be better approached through network mapping , It can be well restored to high-quality data through variational autoencoders ,therefore With the help of the above methods, a good quality Secondly, it is simpler to transform in a low-dimensional space than in a complex high-dimensional space, so that a clear image can be generated from an underwater distorted image. The underwater image enhancement method of the embodiment of the present invention can solve the existing problems of underwater distorted images in a targeted manner according to the different characteristics of the problems, thereby generating an ideal clear image.
[0096] Combination Figures 10a to 11b As shown, the underwater image enhancement method provided by the embodiment of the present invention can effectively improve the quality of underwater image data, greatly improve the problems existing in underwater images, provide strong visual support for underwater operations, and improve the efficiency of underwater related activities.
[0097] In summary, the underwater image enhancement method provided by the present invention has the following advantages: 1) The underwater image enhancement method for underwater environments proposed by the present invention realizes the enhancement of low-quality underwater images. 2) The present invention encodes the image into feature variables in the latent space through a variational autoencoder. Compared with the use of an autoencoder, which has no special constraints and may have problems such as uncontrollable encoding results, the embodiment of the present invention uses a variational autoencoder for encoding, so that the encoding of the latent space conforms to a certain distribution, which is more convenient and easy to reduce the gap between subsequent variables. It is convenient for subsequent spatial alignment, and then the decoder restores the original image data, extracts various features of the image, and improves the accuracy of image feature extraction. 3) The transformation is performed in a low-dimensional latent space instead of a complex high-dimensional space, which reduces the processing difficulty, improves the enhancement efficiency and speed, and improves the effect of subsequent image enhancement. 4) The present invention generates degraded images by degrading real high-quality data, aligns the domains of synthetic degraded data and real low-quality data in low-dimensional latent space by sharing the same variational autoencoder with real low-quality data and designing loss functions, indirectly increases the sample types of low-quality data, improves the algorithm image enhancement effect, improves the generalization ability of the algorithm, and enhances the adaptability of the algorithm to different input images. 5) The present invention realizes the mapping relationship between synthetic degraded data and low-quality real data and ideal high-quality output images in low-dimensional space through ECANet based on the attention mechanism, extracts different features of low-quality input images through variational autoencoders, restores high-quality enhanced images, and improves various types of problems such as color distortion and blurred details in low-quality input images, thereby improving image quality from multiple angles and multiple aspects.
[0098] As another embodiment of the present invention, an underwater image enhancement device 10 is provided for implementing the underwater image enhancement method described above, wherein: Fig.12As shown, it includes: an encoding module 100, which is used to obtain an underwater image to be enhanced, and encode the underwater image to be enhanced through a first image enhancement encoder to obtain a low-dimensional feature vector to be enhanced, wherein the first image enhancement encoder is obtained after optimization according to first encoding and decoding optimization data, and the first encoding and decoding optimization data is obtained after encoding and decoding according to a data set of an image to be enhanced and a degraded image data set, respectively, and the degraded image data set is obtained after degradation processing of a target image data set; a mapping module 200, which is used to input the low-dimensional feature vector to be enhanced into a target mapping network to obtain a target quality low-dimensional feature vector, wherein the target mapping network is a neural network model based on an attention mechanism, which uses the first encoding and decoding process data as a neural network model based on an attention mechanism. The input data of the type and the second encoding and decoding process data are used as the output data of the neural network model based on the attention mechanism, and the neural network model based on the attention mechanism is trained to obtain the data, the first encoding and decoding process data are obtained after encoding according to the image data set to be enhanced and the degraded image data set respectively, and the second encoding and decoding process data are obtained after encoding according to the target image data set; the decoding module 300 is used to decode the target quality low-dimensional feature vector through the second image enhancement encoder to obtain an enhanced underwater image, the second image enhancement encoder is obtained after optimization according to the second encoding and decoding optimization data, and the second encoding and decoding optimization data is obtained after optimization according to the target image data set.
[0099] The underwater image enhancement device provided by the present invention encodes the underwater image to be enhanced through a first image enhancement encoder, and then obtains a target quality low-dimensional feature vector through a target mapping network, and finally decodes through a second image enhancement encoder to obtain an enhanced underwater image. In the underwater image enhancement device, since the first image enhancement encoder is specifically obtained by performing encoding and decoding optimization on the image data set to be enhanced and the degraded image data set respectively, the second image enhancement encoder is specifically obtained after optimizing the second encoding and decoding optimization data, the target mapping network is specifically obtained by training a neural network model based on an attention mechanism, and the input data of the neural network model based on the attention mechanism is specifically the first encoding and decoding process data, and the output data is specifically the second encoding and decoding process data, and the first encoding and decoding process data is specifically the data obtained after encoding the image data set to be enhanced and the degraded image data set respectively, and the second encoding and decoding process data is specifically the data obtained after encoding the target image data set. Therefore, the underwater image enhancement device of the present invention generates a degraded image by performing degradation processing on the high-quality target image, and at the same time performs the same encoding and decoding processing on the degraded image and the original image with poor quality, which can effectively narrow the difference between the two in a low-dimensional space. In addition, by performing additional encoding and decoding processing on the high-quality target image and training the target mapping network through the process data of the two encoding and decoding processes, the trained target mapping network can restore high-quality data that meets the quality requirements, thereby improving the quality and speed of image enhancement.
[0100] The specific working principle of the underwater image enhancement device provided by the present invention can be referred to the description of the underwater image enhancement method above, which will not be repeated here.
[0101] In summary, the underwater image enhancement method and device provided by the present invention have the following advantages: (1) The present invention uses a variational autoencoder to act on real high-quality data and low-quality data, as well as synthetic degraded data, to encode the image into a feature variable in the latent space. Compared with the use of an autoencoder, which has no special constraints and may have problems such as uncontrollable encoding results, the variational autoencoder is used here for encoding, so that the encoding of the latent space conforms to a certain distribution, which is more convenient and easy to reduce the gap between subsequent variables. It is convenient for subsequent spatial alignment, and then the decoder restores the original image data, extracts various features existing in the image, and improves the accuracy of image feature extraction. (2) The present invention performs degradation processing on high-quality images to generate a synthetic degraded data set, and uses the same variational autoencoder and loss function to act on degraded data and real low-quality data, that is, extracts image features, and compresses the image into a feature vector in a low-dimensional space for subsequent conversion, which can also make the degraded data and the real low-quality data continue to approach each other, that is, indirectly increase the sample types of low-quality data, improve the generalization ability of the algorithm, and improve the quality and efficiency of subsequent image enhancement. (3) The present invention uses an ECANet network structure based on an attention mechanism to transform feature variables in a low-dimensional latent space, so as to better focus on important features and suppress unimportant features. Through this mechanism, the representation ability of the network is effectively enhanced while avoiding the addition of too many parameters and computational costs. Generative adversarial networks are used to enhance the domain alignment of sample data and input data, and data is transformed in a low-dimensional latent space to improve the enhancement effect and speed. (4) The present invention maps data in a low-dimensional latent space, so that in the latent space of high-quality data, with the help of the decoder of its variational autoencoder, low-quality data can be well restored to a high-quality enhanced image.
[0102] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.
Claims
1. A method for underwater image enhancement, characterized in that: include: Acquire an underwater image to be enhanced, and encode the underwater image to be enhanced through a first image enhancement encoder to obtain a low-dimensional feature vector to be enhanced, wherein the first image enhancement encoder is obtained after being optimized according to first encoding and decoding optimization data, the first encoding and decoding optimization data is obtained after encoding and decoding respectively according to a data set of an image to be enhanced and a data set of a degraded image, and the degraded image data set is obtained after degradation processing is performed on a target image data set; Inputting the low-dimensional feature vector to be enhanced into the target mapping network to obtain a target quality low-dimensional feature vector, wherein the target mapping network is obtained by taking the first encoding and decoding process data as the input data of the neural network model based on the attention mechanism and taking the second encoding and decoding process data as the output data of the neural network model based on the attention mechanism, and training the neural network model based on the attention mechanism, wherein the first encoding and decoding process data is obtained by encoding the image data set to be enhanced and the degraded image data set respectively, and the second encoding and decoding process data is obtained by encoding the target image data set; The target quality low-dimensional feature vector is decoded by a second image enhancement encoder to obtain an enhanced underwater image, wherein the second image enhancement encoder is obtained after optimization according to second encoding and decoding optimization data, and the second encoding and decoding optimization data is obtained after encoding and decoding according to the target image data set.
2. The underwater image enhancement method according to claim 1, characterized in that: The first image enhancement encoder is obtained after being optimized according to the first encoding and decoding optimization data, and includes: Obtaining a to-be-enhanced image dataset and a target image dataset; Performing degradation processing on the target image data set to obtain a degraded image data set; The to-be-enhanced image dataset and the degraded image dataset are respectively subjected to encoding and decoding processing by a first variational autoencoder to obtain first encoding and decoding optimized data; The first image enhancement encoder to be optimized is optimized according to the first encoding and decoding optimization data to obtain a first image enhancement encoder.
3. The underwater image enhancement method according to claim 2, characterized in that: The method further comprises: performing encoding and decoding processing on the to-be-enhanced image dataset and the degraded image dataset respectively through a first variational autoencoder to obtain first encoding and decoding optimized data, including: Encoding the to-be-enhanced image dataset and the degraded image dataset through a first variational autoencoder to obtain a to-be-enhanced low-dimensional feature vector set and a degraded low-dimensional feature vector set respectively; Decoding the low-dimensional feature vector set to be enhanced and the degraded low-dimensional feature vector set respectively to obtain a decoded image data set to be enhanced and a decoded degraded image data set; Constructing constraint conditions so that both the to-be-enhanced low-dimensional feature vector set and the degraded low-dimensional feature vector set satisfy Gaussian distribution, and constructing a first loss function according to the constraint conditions; constructing a second loss function according to an error between the to-be-enhanced image dataset and the decoded to-be-enhanced image dataset and an error between the degraded image dataset and the decoded degraded image dataset; The first loss function and the second loss function are optimized to obtain first encoding and decoding optimization data.
4. The underwater image enhancement method according to claim 2, characterized in that: Performing degradation processing on the target image data set to obtain a degraded image data set includes: The target image data set is subjected to degradation processing according to the Gaussian degradation model to obtain a degraded image data set. The expression of the degradation processing is: , in, represents the degraded image dataset, represents the target image dataset, represents the degradation noise, represents the Gaussian degradation model, , in, represents the normalization constant, express The circular support domain.
5. The underwater image enhancement method according to claim 1, characterized in that: The second image enhancement encoder is obtained after being optimized according to the second encoding and decoding optimization data, and includes: The target image data set is encoded and decoded by a second variational autoencoder to obtain second encoding and decoding optimized data; The second image enhancement encoder to be optimized is optimized according to the second encoding and decoding optimization data to obtain a second image enhancement encoder.
6. The underwater image enhancement method according to claim 5, characterized in that: The target image data set is encoded and decoded by a second variational autoencoder to obtain second encoding and decoding optimized data, including: Encoding the target image data set through a second variational autoencoder to obtain a target low-dimensional feature vector set; Decoding the target low-dimensional feature vector set to obtain a decoded target image data set; constructing a third loss function according to an error between the target image dataset and the decoded target image dataset; The third loss function is optimized to obtain second encoding and decoding optimization data.
7. The underwater image enhancement method according to any one of claims 1 to 6, characterized in that: The target mapping network is obtained by taking the first encoding and decoding process data as input data of the neural network model based on the attention mechanism and taking the second encoding and decoding process data as output data of the neural network model based on the attention mechanism and training the neural network model based on the attention mechanism, including: Encoding the to-be-enhanced image dataset and the degraded image dataset respectively through a first variational autoencoder to obtain a to-be-enhanced low-dimensional feature vector set and a degraded low-dimensional feature vector set, wherein the to-be-enhanced low-dimensional feature vector set and the degraded low-dimensional feature vector set are used as first encoding and decoding process data; Encoding the target image data set through a second variational autoencoder to obtain a target low-dimensional feature vector set, wherein the target low-dimensional feature vector set serves as second encoding and decoding process data; Inputting the first encoding and decoding process data as input data into a neural network model based on an attention mechanism to obtain a prediction feature vector data set; Constructing a mapping network loss function according to the predicted feature vector data set and the second encoding and decoding process data; The mapping network loss function is optimized to obtain a target mapping network.
8. The underwater image enhancement method according to claim 7, characterized in that: Constructing a mapping network loss function according to the predicted feature vector data set and the second encoding and decoding process data, including: Obtaining a fourth loss function by performing mean square error calculation on the predicted feature vector data set and the target low-dimensional feature vector set; Decoding the predicted feature vector data set through a second variational autoencoder to obtain a predicted image data set, and decoding the target low-dimensional feature vector set through the second variational autoencoder to obtain a decoded target image data set; Calculate the mean absolute error between the predicted image data set and the decoded target image data set to obtain a fifth loss function; The mapping network loss function is constructed according to the fourth loss function and the fifth loss function.
9. The underwater image enhancement method according to claim 8, characterized in that: The expression of the mapping network loss function is: , in, represents the fourth loss function, , represents the fifth loss function, , represents the output of the ECANet network, represents high-quality variables in the latent space, represents the number of samples, and Acting in low-dimensional space can make the mapping generated The second variational decoder code acts on Encoded , so that With the help of The decoding process recovers , calculate The variables recovered by the second image enhancement encoder with by Restored variables The gap makes Recover .
10. An underwater image enhancement device, used to implement the underwater image enhancement method according to any one of claims 1 to 9, characterized in that: include: A coding module, used for acquiring an underwater image to be enhanced, and encoding the underwater image to be enhanced through a first image enhancement encoder to obtain a low-dimensional feature vector to be enhanced, wherein the first image enhancement encoder is obtained after being optimized according to first coding and decoding optimization data, the first coding and decoding optimization data is obtained after encoding and decoding the image data set to be enhanced and the degraded image data set, respectively, and the degraded image data set is obtained after degradation processing is performed on the target image data set; A mapping module, used for inputting the low-dimensional feature vector to be enhanced into a target mapping network to obtain a target quality low-dimensional feature vector, wherein the target mapping network is to use the first encoding and decoding process data as the input data of the neural network model based on the attention mechanism and the second encoding and decoding process data as the output data of the neural network model based on the attention mechanism, and to train the neural network model based on the attention mechanism, wherein the first encoding and decoding process data is obtained by encoding the image data set to be enhanced and the degraded image data set respectively, and the second encoding and decoding process data is obtained by encoding the target image data set; A decoding module is used to decode the target quality low-dimensional feature vector through a second image enhancement encoder to obtain an enhanced underwater image, wherein the second image enhancement encoder is obtained after optimization according to second encoding and decoding optimization data, and the second encoding and decoding optimization data is obtained after optimization after encoding and decoding according to the target image data set.
