An underwater image enhancement method based on context decomposition feature fusion

Abstract

The present invention provides an underwater image enhancement method based on context decomposition feature fusion, which relates to the field of underwater image enhancement technology, including: obtaining distorted underwater images to form a training set and a test set; constructing an underwater image enhancement network model based on context decomposition feature fusion; wherein the underwater image enhancement network model adopts a context decomposition feature fusion method during feature fusion; inputting the training set into the constructed underwater image enhancement network model for training; inputting the test set image into the trained underwater image enhancement network model, and outputting a high-quality underwater image after image enhancement. The present invention expands the feature fusion method in image processing by fusing context feature information, and also establishes a link between local and global analysis of the model, improves the effect of underwater image enhancement, and avoids the occurrence of adverse effects such as loss of details, color desaturation, and artifacts.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater image enhancement, and in particular to an underwater image enhancement method based on context decomposition feature fusion. Background Art

[0002] As land resources become scarce, population increases, and environmental damage become increasingly prominent, research and development of marine resources are on the rise worldwide. However, the acquisition and processing of marine information has become the first step in the development of marine resources. Therefore, underwater image processing technology has received more and more attention. Underwater image enhancement technology plays an important role in improving the quality of underwater images, which provides help for the subsequent processing of underwater information.

[0003] Among the existing underwater image enhancement technologies, the use of an attention-based multi-scale underwater image enhancement network is a very effective method for underwater image enhancement. It first extracts multi-scale features through downsampling, connects feature layers of the same size in the deconvolution stage, and finally introduces an attention mechanism module. This method enables the network to focus on features of different scales and enhance the restoration of overall color and details.

[0004] However, the above methods are limited to the intermediate layer connecting downsampling and upsampling when fusing feature information, ignoring the scene-level contextual information in the spatial scale, resulting in many adverse effects, such as detail loss, color desaturation, artifacts, etc. Summary of the invention

[0005] In view of this, the purpose of the present invention is to propose an underwater image enhancement method based on context decomposition feature fusion, and to achieve the generation and application of high-quality underwater images, so as to solve the technical problem that in the existing underwater image enhancement technology, adverse effects are caused by ignoring scene-level context information in the spatial scale when fusing feature information.

[0006] To this end, the present invention provides the following technical solutions:

[0007] On the one hand, the present invention provides an underwater image enhancement method based on context decomposition feature fusion, the method comprising:

[0008] Obtain distorted underwater images to form training sets and test sets;

[0009] Construct an underwater image enhancement network model based on context decomposition feature fusion; wherein the underwater image enhancement network model takes a distorted underwater image as input and outputs an enhanced high-quality underwater image, and adopts a context decomposition feature fusion method during feature fusion, wherein the context decomposition feature fusion includes: fusing the downsampled features with the corresponding upsampled features at the same scale, and cascading the obtained results to the next layer;

[0010] Inputting the training set into the constructed underwater image enhancement network model for training;

[0011] The test set images are input into the trained underwater image enhancement network model, and high-quality underwater images after image enhancement are output.

[0012] Furthermore, the network structure of the underwater image enhancement network model includes:

[0013] A two-layer estimation mechanism consisting of two Unet network structures, two processes simultaneously optimize the image; the UNet network structure contains 10 basic blocks, each block consists of 3 convolutional layers, where each downsampling stage includes two Conv+BN+ReLU layers and one Max Pooling layer, and each upsampling stage includes one Up Sampling layer and two Conv+ReLU layers.

[0014] Furthermore, when fusing features, the features at the same scale of the lower Unet network structure are multiplied with the corresponding upsampled features of the upper Unet network structure, and then the results are cascaded to the next layer, thereby realizing the fusion of context decomposition features.

[0015] Furthermore, the Unet network structure also includes: a channel attention module and a pixel attention module.

[0016] Furthermore, in the channel attention module, global average pooling, two convolutional layers and an activation function layer are used to convert global information into channel weight information, so that different channel features have different weighted information.

[0017] Furthermore, in the pixel attention module, two convolutional layers and an activation function layer are used to weight each pixel value.

[0018] Furthermore, the loss functions used in the underwater image enhancement network model during training include: L1 loss function and contrast loss function.

[0019] In another aspect, the present invention further provides an underwater image enhancement device based on context decomposition feature fusion, the device comprising:

[0020] An image acquisition unit, used for acquiring distorted underwater images to form a training set and a test set;

[0021] A model building unit is used to build an underwater image enhancement network model based on context decomposition feature fusion; wherein the underwater image enhancement network model takes a distorted underwater image as input and outputs an enhanced high-quality underwater image, and the feature fusion adopts a context decomposition feature fusion method, and the context decomposition feature fusion includes: fusing the downsampled features with the corresponding upsampled features at the same scale, and cascading the obtained results to the next layer;

[0022] A training unit, used for inputting the training set into the constructed underwater image enhancement network model for training;

[0023] The output unit is used to input the test set image into the trained underwater image enhancement network model and output the high-quality underwater image after image enhancement.

[0024] Furthermore, the network structure of the underwater image enhancement network model includes:

[0025] A two-layer estimation mechanism consisting of two Unet network structures, two processes simultaneously optimize the image; the UNet network structure contains 10 basic blocks, each block consists of 3 convolutional layers, where each downsampling stage includes two Conv+BN+ReLU layers and one Max Pooling layer, and each upsampling stage includes one Up Sampling layer and two Conv+ReLU layers.

[0026] On the other hand, the present invention also provides a computer-readable storage medium, which stores a computer instruction set. When the computer instruction set is executed by a processor, the above-mentioned underwater image enhancement method based on context decomposition feature fusion is implemented.

[0027] Advantages and positive effects of the present invention: In the present invention, the feature fusion method in image processing is expanded by fusing context feature information, and a link is established for local and global analysis of the model, which improves the effect of underwater image enhancement and avoids the occurrence of adverse effects such as loss of details, color desaturation, and artifacts. Secondly, the underwater image enhancement technology in the present invention will also promote related two-dimensional image processing and applications, such as underwater target detection, recognition, reconstruction and other applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 is a flow chart of underwater image enhancement based on context decomposition feature fusion in an embodiment of the present invention;

[0030] Figure 2 is a network structure diagram of underwater image enhancement based on context decomposition feature fusion in an embodiment of the present invention;

[0031] Figure 3 is a schematic diagram of the structure of a context decomposition feature fusion module in an embodiment of the present invention;

[0032] Figure 4 It is a structural diagram of the feature attention module in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Aiming at the color cast and dispersion phenomena in underwater images, the present invention proposes an underwater image enhancement method based on context decomposition feature fusion, which can be applied to underwater navigation, underwater target detection, recognition and seabed resource exploration.

[0036] like Figure 1 As shown, an underwater image enhancement method based on context decomposition feature fusion in an embodiment of the present invention includes the following specific steps:

[0037] S1, obtain distorted underwater images to form training sets and test sets;

[0038] S2. Construct an underwater image enhancement network model based on context decomposition feature fusion;

[0039] Among them, the underwater image enhancement network model takes the distorted underwater image as input and outputs the enhanced high-quality underwater image. The feature fusion adopts the context decomposition feature fusion method, which specifically includes: fusing the downsampled features with the corresponding upsampled features at the same scale, and cascading the results to the next layer. Through this fusion of context decomposition features, the context dependencies at multiple scales can be fully explored, and the context information can be aggregated to obtain more valuable feature information, so as to accurately estimate the image, and the image can have vivid colors, sufficient details and little noise.

[0040] S3, input the training set into the constructed underwater image enhancement network model for training;

[0041] Through a limited number of iterative learning, it is determined whether the loss has reached the minimum value, and the optimization parameters are continuously back-propagated during the learning process. In terms of network loss, the L1 loss function and the contrast loss function are adopted in the embodiment of the present invention. The loss function can constrain the generated image to be closer and closer to the high-quality underwater image during the training stage.

[0042] The L1 loss function is:

[0043] Among them, X and Y are blurry underwater images and clear underwater images respectively, and N(X) is the output image after the input image passes through the network.

[0044] Contrast loss function L C for:

[0045] Here, V is the ReLU 5_3 layer of the VGG network.

[0046] After training, the distorted underwater images are input into the network, and the images will reduce blur while maintaining normal colors and improve image contrast.

[0047] S4. Input the test set images into the trained underwater image enhancement network model and output high-quality underwater images after image enhancement.

[0048] For ease of understanding, the above-mentioned underwater image enhancement network model based on context decomposition feature fusion is described in detail below.

[0049] The network structure diagram of the underwater image enhancement network model based on context decomposition feature fusion is as follows: Figure 2As shown in the figure, in order to realize the fusion of contextual decomposition features, a two-layer estimation mechanism consisting of two UNet structures is introduced, and the two processes optimize the image simultaneously. The encoder of Unet is used to extract target features and reduce the amount of parameters, and the decoder of Unet is used to reconstruct features and gradually restore the image structure and texture detail information. The network structure of UNet contains 10 basic blocks, each of which is composed of 3 convolutional layers, where each downsampling stage includes two Conv+BN+ReLU layers and a Max Pooling layer, and each upsampling stage includes an Up Sampling layer and two Conv+ReLU layers. When fusion of features, the contextual decomposition feature fusion method is adopted. The latent space features learned by the decoder of the upper Unet network are used as a guide, and are fused with the multi-scale features extracted by the lower Unet network, which helps to enhance global and local information. Specifically, as Figure 3 As shown, the context decomposition feature fusion module (i.e. Figure 2 The MSP in the network structure includes: performing multiplication operations on the features of the same scale of the lower Unet network structure and the corresponding upsampled features of the upper Unet network structure, and then performing a cascade operation on the results to the next layer.

[0050] The two-layer Unet network can be expressed as:

[0051]

[0052]

[0053] Where: N TUN , N LUN They are upper Unet network and lower Unet network respectively; is the feature extracted by the i-th upsampling of the upper Unet network, M i , are the features extracted by the i-th downsampling and upsampling of the lower Unet network; X u is the input image.

[0054] Context decomposition feature fusion can be expressed as:

[0055]

[0056]

[0057]

[0058] in: To perform the features generated by the multiplication operation, C is the cascade operation.

[0059] In order to fully explore the real color information hidden behind the underwater image, the underwater image enhancement network weights different channel features and different pixel features to focus on more important areas, such as blurred areas of the image. The present invention adds a feature enhancement module to the minimum scale of Unet, including a channel attention module and a pixel attention module. In the underwater image enhancement network, these two modules can adaptively learn different weights of different features.

[0060] like Figure 4 As shown, the feature attention module (i.e. Figure 2 UFEB in : includes channel attention and pixel attention. In the channel attention module, global average pooling, two convolutional layers and an activation function layer are used to convert global information into channel weight information, so that different channel features have different weighted information; in the pixel attention module, two convolutional layers and an activation function layer are used to weight each pixel value.

[0061] The specific operation of the channel attention module is:

[0062] A 1＝ G(I); G is the global average pooling operation, and I is the input of the module;

[0063] A2 = CR (A1); CR is the convolution + activation function layer;

[0064] A3=C(A2); C is the convolutional layer;

[0065] A4 = sigmoid (A3); sigmoid is the activation function;

[0066] The specific operations of the pixel attention module are:

[0067] A5 = CR (A4); CR is the convolution + activation function layer;

[0068] A6=C(A5); C is the convolutional layer;

[0069] A7=sigmoid(A6); sigmoid is an activation function.

[0070] The overall algorithm of the underwater image enhancement network model based on context decomposition feature fusion is as follows:

[0071]

[0072] In the above-mentioned embodiment of the present invention, the feature fusion method in image processing is expanded by fusing contextual feature information, and a link is established for the local and global analysis of the model, thereby improving the effect of underwater image enhancement and avoiding the occurrence of adverse effects such as loss of details, color desaturation, and artifacts.

[0073] Corresponding to the underwater image enhancement method based on context decomposition feature fusion in the present invention, the present invention also provides an underwater image enhancement device based on context decomposition feature fusion, comprising:

[0074] An image acquisition unit, used for acquiring distorted underwater images to form a training set and a test set;

[0075] A model building unit, used for building an underwater image enhancement network model based on context decomposition feature fusion;

[0076] Among them, the underwater image enhancement network model takes the distorted underwater image as input and outputs the enhanced high-quality underwater image. The feature fusion adopts the context decomposition feature fusion method, which specifically includes: fusing the downsampled features with the corresponding upsampled features at the same scale, and cascading the results to the next layer. Through this fusion of context decomposition features, the context dependencies at multiple scales can be fully explored, and the context information can be aggregated to obtain more valuable feature information, so as to accurately estimate the image, and the image can have vivid colors, sufficient details and little noise.

[0077] A training unit, used for inputting the training set into the constructed underwater image enhancement network model for training;

[0078] Through a limited number of iterative learning, it is determined whether the loss has reached the minimum value, and the parameters are continuously back-propagated during the learning process. In terms of network loss, the L1 loss function and the contrast loss function are used. This loss function can constrain the generated image to be closer and closer to the high-quality underwater image during the training stage. After training, the distorted underwater image is input into the network, and the image will achieve the reduction of blur while maintaining the normal color and improving the image contrast.

[0079] The output unit is used to input the test set image into the trained underwater image enhancement network model and output the high-quality underwater image after image enhancement.

[0080] For the underwater image enhancement device based on context decomposition feature fusion in the embodiment of the present invention, since it corresponds to the underwater image enhancement method based on context decomposition feature fusion in the above embodiment, the description is relatively simple. For relevant similarities, please refer to the description of the underwater image enhancement method based on context decomposition feature fusion in the above embodiment, which will not be described in detail here.

[0081] An embodiment of the present invention further discloses a computer-readable storage medium, which stores a computer instruction set. When the computer instruction set is executed by a processor, the underwater image enhancement method based on context decomposition feature fusion as provided in any of the above embodiments is implemented.

[0082] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0083] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0084] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0085] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An underwater image enhancement method based on context decomposition feature fusion, characterized in that: The method comprises: Obtain distorted underwater images to form training sets and test sets; Construct an underwater image enhancement network model based on context decomposition feature fusion; wherein the underwater image enhancement network model takes a distorted underwater image as input and outputs an enhanced high-quality underwater image, and adopts a context decomposition feature fusion method for feature fusion, and the context decomposition feature fusion includes: fusing the downsampled features with the corresponding upsampled features at the same scale, and cascading the obtained results to the next layer; when fusing the features, multiplying the features extracted by the i-th downsample of the lower Unet network with the i-th upsampled features extracted by the upper Unet network, and multiplying the features extracted by the i-th upsample of the lower Unet network with the i-th upsampled features extracted by the upper Unet network, cascading the features obtained by the two multiplication operations to obtain the features after the context decomposition feature fusion, and passing them to the next layer, thereby realizing the fusion of the context decomposition features; Inputting the training set into the constructed underwater image enhancement network model for training; The test set images are input into the trained underwater image enhancement network model, and high-quality underwater images after image enhancement are output.

2. The underwater image enhancement method based on context decomposition feature fusion according to claim 1, characterized in that: The network structure of the underwater image enhancement network model includes: A two-layer estimation mechanism consisting of two Unet network structures, two processes simultaneously optimize the image; the UNet network structure contains 10 basic blocks, each block consists of 3 convolutional layers, where each downsampling stage includes two Conv+BN+ReLU layers and one Max Pooling layer, and each upsampling stage includes one Up Sampling layer and two Conv+ReLU layers.

3. The underwater image enhancement method based on context decomposition feature fusion according to claim 2 is characterized in that: The Unet network structure also includes: channel attention module and pixel attention module.

4. The underwater image enhancement method based on context decomposition feature fusion according to claim 3 is characterized in that: In the channel attention module, global average pooling, two convolutional layers and an activation function layer are used to convert global information into channel weight information, so that different channel features have different weighted information.

5. The underwater image enhancement method based on context decomposition feature fusion according to claim 3 is characterized in that: In the pixel attention module, each pixel value is weighted using two convolutional layers and an activation function layer.

6. The underwater image enhancement method based on context decomposition feature fusion according to claim 1, characterized in that: The loss functions used in the underwater image enhancement network model during training include: L1 loss function and contrast loss function.

7. An underwater image enhancement device based on context decomposition feature fusion, characterized in that: The device comprises: An image acquisition unit, used for acquiring distorted underwater images to form a training set and a test set; A model construction unit is used to construct an underwater image enhancement network model based on context decomposition feature fusion; wherein the underwater image enhancement network model takes a distorted underwater image as input, outputs an enhanced high-quality underwater image, and adopts a context decomposition feature fusion method during feature fusion, wherein the context decomposition feature fusion comprises: fusing the downsampled features with the corresponding upsampled features at the same scale, and cascading the obtained results to the next layer; during feature fusion, performing a multiplication operation on the feature extracted by the i-th downsample of the lower Unet network and the feature extracted by the i-th upsample of the upper Unet network, performing a multiplication operation on the feature extracted by the i-th upsample of the lower Unet network and the feature extracted by the i-th upsample of the upper Unet network, performing a cascade operation on the features obtained by the two multiplication operations to obtain the features after the context decomposition feature fusion, and passing it to the next layer, thereby realizing the fusion of the context decomposition features; A training unit, used for inputting the training set into the constructed underwater image enhancement network model for training; The output unit is used to input the test set image into the trained underwater image enhancement network model and output the high-quality underwater image after image enhancement.

8. The underwater image enhancement device based on context decomposition feature fusion according to claim 7, characterized in that: The network structure of the underwater image enhancement network model includes: A two-layer estimation mechanism consisting of two Unet network structures, two processes simultaneously optimize the image; the UNet network structure contains 10 basic blocks, each block consists of 3 convolutional layers, where each downsampling stage includes two Conv+BN+ReLU layers and one Max Pooling layer, and each upsampling stage includes one Up Sampling layer and two Conv+ReLU layers.

9. A computer-readable storage medium storing a computer instruction set, wherein the computer instruction set, when executed by a processor, implements the underwater image enhancement method based on context decomposition feature fusion as described in any one of claims 1 to 6.

Images