Style transfer underwater image enhancement method, device and equipment based on feature enhancement

Abstract

The present application is about a method, device and equipment for underwater image enhancement based on style transfer based on feature enhancement, and relates to the field of underwater image enhancement. The present application obtains a degraded underwater image to be enhanced; constructs an underwater image enhancement model; uses a red channel prior module to perform prior enhancement preprocessing on the degraded underwater image to be enhanced, separates the three channels of red, green and blue, performs channel compensation on the three channels of red, green and blue, and outputs a preliminary enhanced image; uses a feature matching style selection module to determine the target style image; inputs the target style image and the preliminary enhanced image into a style transfer module, and outputs the final enhanced image. In this case, the degraded underwater image can be enhanced, and good effects are shown in both color correction and clarity enhancement. While enhancing brightness, the edges of objects are also refined to achieve a more realistic effect.

Description

Technical Field

[0001] The present application relates to the technical field of underwater image enhancement, and in particular to a style transfer underwater image enhancement method, device and equipment based on feature enhancement. Background Art

[0002] Underwater images are one of the important ways for humans to obtain ocean information. However, since light is affected by the absorption and scattering effects of water when it propagates in water, underwater images often suffer from degradation phenomena such as low contrast, blur, and color distortion, which seriously affects the accuracy and effectiveness of subsequent processing.

[0003] In the field of underwater image enhancement, early work can be divided into two branches, based on non-physical models and based on physical models. Non-physical model-based methods aim to directly improve pixel intensity and obtain better image quality, such as color correction, contrast stretching, dehazing, etc.; physical model-based methods regard the task as an inverse problem and solve ill-posed problems through natural image priors, such as underwater dark channel prior (UDCP), generalized dark channel prior (GDCP), etc. In recent years, deep learning-based methods have made significant progress in computer vision and image processing. Compared with traditional methods, deep learning-based methods tend to design end-to-end modules or network integration combined with physical model priors to solve problems. However, learning-based methods usually require degraded and high-quality corresponding image pairs for supervised learning. In the real world, it is impractical to collect a large number of distortion-free real underwater images. Many studies based on synthetic underwater images ignore the field gap between synthetic images and real images.

[0004] Among the deep learning-based methods, style transfer methods have also achieved good results in the field of computer vision and are widely used in style transfer, image generation, and other aspects. In the field of data enhancement, Zhou et al. proposed cross-modal generalization of medical images based on cross-domain style enhancement, which effectively solved the problem of poor model generalization ability. Due to the lack of annotated underwater datasets, Mohamed E. Fathy et al. used style transfer technology to create visually credible underwater scenes using existing aerial datasets. In the field of underwater image enhancement, using the theoretical knowledge of content-style separation in style transfer, Chen et al. proposed a domain-adaptive underwater image enhancement framework with content-style separation to reduce the domain difference between the synthetic image domain and the real underwater image domain.

[0005] Defects and shortcomings of existing technology:

[0006] (1) Traditional methods are inadequate in processing different underwater image backgrounds and have limited generalization capabilities;

[0007] (2) Learning-based methods usually require degraded and high-quality corresponding image pairs for supervised learning, but it is difficult to obtain high-quality datasets in practice;

[0008] (3) Existing methods based on synthetic underwater images ignore domain differences.

[0009] To address these difficulties, we approach the underwater image enhancement problem from the perspective of style transfer through deep learning methods, and propose a feature matching style selection module to address the challenge of the difficulty in obtaining high-quality datasets; in addition, the style image is selected in the underwater domain, bridging the domain differences. Summary of the invention

[0010] The purpose of this application is to provide a style transfer underwater image enhancement method, device and equipment based on feature enhancement to solve the problems existing in the above-mentioned prior art.

[0011] To achieve the above purpose, the technical solution adopted in this application is:

[0012] In a first aspect, the present application provides a style migration underwater image enhancement method based on feature enhancement, the method is applied in a computer device, and the method comprises:

[0013] Acquiring a degraded underwater image to be enhanced;

[0014] Constructing an underwater image enhancement model, wherein the underwater image enhancement model includes a red channel prior module, a feature matching style selection module, and a style migration module;

[0015] The red channel prior module is used to perform prior enhancement preprocessing on the degraded underwater image to be enhanced, and the red, green and blue channels are separated, and channel compensation is performed on the red, green and blue channels to output a preliminary enhanced image;

[0016] Determining a target style image using the feature matching style selection module;

[0017] The target style image and the preliminary enhanced image are input into the style transfer module, and a final enhanced image is output.

[0018] In a possible implementation, the red channel prior module is used to perform prior enhancement preprocessing on the degraded underwater image to be enhanced, separates the red, green and blue channels, performs channel compensation on the red, green and blue channels, and outputs a preliminary enhanced image, including:

[0019] Using the red channel prior module to perform prior enhancement preprocessing on the degraded underwater image to be enhanced, and separating the three channels of red, green and blue;

[0020] Performing channel compensation on the green channel and the blue channel to make the pixel average values ​​of the blue channel and the green channel consistent, and outputting the corrected blue channel and green channel;

[0021] The corrected blue channel and green channel are used as increments to iteratively adjust the red channel, and a preliminary enhanced image is output.

[0022] In a possible implementation, performing channel compensation on the green channel and the blue channel to make the pixel average values ​​of the blue channel and the green channel consistent, and outputting the corrected blue channel and green channel, includes:

[0023] Determine a first compensation coefficient based on the size of the pixel average value of the blue channel and the pixel average value of the green channel;

[0024] Based on the difference between the pixel average value of the blue channel and the pixel average value of the green channel and the first compensation coefficient, compensate the one with the smaller pixel average value between the blue channel and the green channel, and output the blue channel and the green channel after the initial correction;

[0025] Based on the pixel averages of the blue channel and the green channel after the primary correction, the blue channel and the green channel after the secondary correction are output.

[0026] In a possible implementation, the iterative adjustment of the red channel using the corrected blue channel and green channel as increments to output a preliminary enhanced image includes:

[0027] Determine a second compensation coefficient based on the smaller pixel average value of the blue channel and the green channel after the secondary correction and the current pixel average value of the red channel;

[0028] Based on the smaller pixel value of the blue channel and the green channel after the secondary correction and the second compensation coefficient, the pixel value of the red channel after compensation is determined, and a preliminary enhanced image is output.

[0029] In a possible implementation, the smaller pixel average value of the blue channel and the green channel after the secondary correction is consistent with the current pixel average value of the red channel.

[0030] In a possible implementation, the determining the target style image by using the feature matching style selection module includes:

[0031] The preliminary enhanced image and the style data set are input into the feature matching style selection module, and the feature matching style selection module determines the target style image based on a pre-trained perceptual similarity measurement network; wherein the style data set is a UIEB data set.

[0032] In a possible implementation, the style transfer module adopts a RevNet encoder-decoder structure.

[0033] In a second aspect, the present application provides a style migration underwater image enhancement device based on feature enhancement, comprising:

[0034] An acquisition module, used for acquiring a degraded underwater image to be enhanced;

[0035] A construction module, used to construct an underwater image enhancement model, wherein the underwater image enhancement model includes a red channel prior module, a feature matching style selection module, and a style migration module;

[0036] An output module, used to perform a priori enhancement preprocessing on the degraded underwater image to be enhanced by using the red channel priori module, separate the red, green and blue channels, perform channel compensation on the red, green and blue channels, and output a preliminary enhanced image;

[0037] A determination module, used to determine a target style image using the feature matching style selection module;

[0038] The output module is further used to input the target style image and the preliminary enhanced image into the style transfer module, and output a final enhanced image.

[0039] In a third aspect, the present application provides a computer device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the processor can load and execute at least one instruction, at least one program, a code set or an instruction set to implement the feature enhancement-based style transfer underwater image enhancement method provided above.

[0040] In a fourth aspect, the present application provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The processor can load and execute at least one instruction, at least one program, code set or instruction set to implement the feature enhancement-based style transfer underwater image enhancement method provided above.

[0041] In a fifth aspect, the present application provides a computer program product or a computer program, the computer program product or the computer program including computer program instructions, the computer program instructions being stored in a computer-readable storage medium. The processor reads the computer instructions from the computer-readable storage medium and executes the computer instructions, so that the computer device executes the feature-enhanced style transfer underwater image enhancement method provided above.

[0042] The beneficial effects of the technical solution provided by this application include at least:

[0043] The present application obtains a degraded underwater image to be enhanced; constructs an underwater image enhancement model, which includes a red channel prior module, a feature matching style selection module, and a style transfer module; uses the red channel prior module to perform prior enhancement preprocessing on the degraded underwater image to be enhanced, separates the red, green, and blue channels, performs channel compensation on the red, green, and blue channels, and outputs a preliminary enhanced image; uses the feature matching style selection module to determine the target style image; inputs the target style image and the preliminary enhanced image into the style transfer module, and outputs the final enhanced image. In this case, the degraded underwater image can be enhanced, and good results are shown in both color correction and clarity enhancement. While enhancing brightness, the edges of objects are also refined to achieve a more realistic effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings:

[0045] Figure 1 A flow chart of a style transfer underwater image enhancement method based on feature enhancement provided by an exemplary embodiment of the present application is shown;

[0046] Figure 2 A schematic diagram of an underwater image enhancement model of a style transfer underwater image enhancement method based on feature enhancement provided by an exemplary embodiment of the present application is shown;

[0047] Figure 3 A schematic diagram of a red channel prior module of a style transfer underwater image enhancement method based on feature enhancement provided by an exemplary embodiment of the present application is shown;

[0048] Figure 4 A schematic diagram showing a style transfer module of a style transfer underwater image enhancement method based on feature enhancement provided by an exemplary embodiment of the present application is shown;

[0049] Figure 5A comparison diagram of the underwater image enhancement method based on feature enhancement provided by an exemplary embodiment of the present application and the other six deep learning methods on the data set is shown.

[0050] Figure 6 A detailed display diagram showing the enhancement effect of a style transfer underwater image enhancement method based on feature enhancement on a UIEB dataset provided by an exemplary embodiment of the present application is shown;

[0051] Figure 7 A structural block diagram of a style migration underwater image enhancement device based on feature enhancement provided by an exemplary embodiment of the present application is shown;

[0052] Figure 8 A schematic structural diagram of a computer device for executing a style transfer underwater image enhancement method based on feature enhancement is shown according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] First, a brief introduction is given to the terms involved in the embodiments of the present application:

[0055] AlexNet is a classic convolutional neural network (CNN) proposed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton in the 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC).

[0056] The UIEB (Underwater Image Enhancement Benchmark) dataset is a benchmark dataset specifically for underwater image enhancement tasks. The dataset was jointly created by researchers from Harbin Institute of Technology and City University of Hong Kong to provide researchers with a standardized platform to evaluate and compare different underwater image enhancement algorithms.

[0057] Conv layer, Convolutional layer (Conv layer for short) is one of the core components in convolutional neural network (CNN). It extracts local features of input data by applying a set of learnable filters (also called convolution kernels or weights). Convolutional layer is very effective in processing data with spatial structure such as images and audio.

[0058] ReLU layer, ReLU (Rectified Linear Unit) layer is a commonly used activation function layer, which is widely used in neural networks in deep learning, especially in convolutional neural networks (CNN) and fully connected neural networks. The main function of the ReLU layer is to introduce nonlinearity, so that the neural network can learn and express more complex patterns.

[0059] squeeze module, in deep learning, squeeze module usually refers to squeeze-and-excitation (SE) module, which is a technique for enhancing the performance of convolutional neural networks (CNN). SE module introduces a channel attention mechanism to recalibrate the importance of each channel in the feature map, thereby improving the expressiveness and generalization ability of the model.

[0060] Channel Refinement (CR) is a module or strategy for more detailed and precise operation or management of channels in neural network design. These operations include refinement of feature extraction, application of channel attention mechanism, feature fusion and selection, modular design, channel pruning and compression, etc. In this application, the main function of the channel refinement module is to prune, reduce the dimension of the channel, and delete the channel redundant information in the content and style dataset features.

[0061] The patch-wise reversible residual block is a structure designed to maintain information integrity and improve network expressiveness. It combines the characteristics of residual connection and reversible computation. In neural networks, reversibility means that the forward propagation process of the network can be reversed in a deterministic way, allowing backpropagation without storing intermediate activations. This feature can reduce memory consumption and is helpful for tasks such as generative models that require explicit modeling of data distribution.

[0062] RevNet (Reversible Network) is a deep learning architecture designed to reduce memory usage by avoiding storing intermediate activation values ​​through reversible layers. This feature is particularly useful for building very deep networks because it can significantly reduce memory requirements. Using RevNet in an encoder-decoder structure can make the model more efficient, especially when processing high-resolution images or tasks that require a lot of memory.

[0063] The EUVP (Enhanced Underwater Visual Perception) dataset is designed to solve underwater image enhancement and visual perception problems. It provides a rich underwater image resource for training and evaluating underwater image processing algorithms, especially those designed to improve image quality, visibility, and color reproduction. This dataset is very valuable for researchers because it contains a range of underwater images under different conditions, including different water clarity, lighting conditions, and shooting scenes.

[0064] The present application is further described below in conjunction with the accompanying drawings and embodiments.

[0065] Figure 1 A flowchart of a style migration underwater image enhancement method based on feature enhancement provided by an exemplary embodiment of the present application is shown. The method is applied in a computer device, and the method includes:

[0066] Step 101: Acquire a degraded underwater image to be enhanced.

[0067] In the embodiments of the present application, the degraded underwater image to be enhanced generally refers to an underwater image that has problems such as color attenuation and contrast reduction due to the influence of factors such as lighting conditions and suspended particles in the underwater environment.

[0068] Step 102, constructing an underwater image enhancement model, the underwater image enhancement model includes a red channel prior module, a feature matching style selection module, and a style transfer module.

[0069] In the embodiment of the present application, the underwater image enhancement model is used to improve the quality of images taken in an underwater environment. In general, the red channel prior module is used to perform a priori enhancement on the degraded underwater image to be enhanced, and the separated red (R), green (G), and blue (B) channels are used for channel compensation; the feature matching style selection module is used to select the target style of the underwater image after a priori enhancement, and determine the target style image to solve the problem of no corresponding GT map; the style transfer module is used to transfer the target style image to the image to be enhanced to obtain the final enhancement result.

[0070] In an example, see Figure 2:First, the degraded underwater image to be enhanced is preliminarily enhanced by the red channel prior module. The red channel correction process is to correct the blue-green channel first, and then correct the red channel by the blue-green channel, and output the preliminary enhanced image; then, the preliminary enhanced image after prior enhancement and the Style style data set are input into the feature matching style selection module together. The feature matching style selection module is based on the pre-trained perceptual similarity measurement network, with the Alex-Net convolutional neural network as the backbone network, and inputs the preliminary enhanced image and the Style style data set. For a given convolution layer, the cosine distance and the spatial dimension of the network and the average value between layers are calculated; then, it is input into the network G (pre-trained neural network model G) to predict the perceptual judgment score of the distance pair. The style image corresponding to the highest one among all scores is considered to be the image with the most perceptual similarity to the content image, that is, the target style image; finally, the target style image and the preliminary enhanced image are input into the style transfer module together, and the final enhanced image is obtained through the encoder and decoder.

[0071] Step 103, using a red channel prior module to perform prior enhancement preprocessing on the degraded underwater image to be enhanced, separating the red, green and blue channels, performing channel compensation on the red, green and blue channels, and outputting a preliminary enhanced image.

[0072] In the embodiment of the present application, the three colors of the clear image histogram are relatively balanced, while the underwater image after being affected by the water light usually appears blue or green, that is, the blue channel or green channel value in the histogram will always move to a higher intensity position, while the red channel is severely degraded, resulting in color stratification; the human visual system can still determine the true color of an object when the light source changes; the gray world algorithm is based on the gray world assumption, and it is believed that for an image with a large number of color changes, the average values ​​of its three components tend to the same gray value. Based on the gray world constancy principle, this application proposes a module for red channel prior enhancement, which prior processes the degraded underwater image to be enhanced by keeping the average value of each color channel consistent.

[0073] In the embodiments of this application, please refer to Figure 3 The red channel prior module separates the three channels of red (R), green (G), and blue (B) for the input degraded underwater image to be enhanced. First, the blue and green channels are compensated to achieve a complete balance between the average values ​​of the blue and green channels; then, the red channel is iteratively adjusted using the corrected blue and green channels as increments until the average value of the red channel reaches the desired level.

[0074] Specifically, step 103 includes:

[0075] Step 1031, using a red channel prior module to perform prior enhancement preprocessing on the degraded underwater image to be enhanced, and separating the three channels of red, green and blue;

[0076] Step 1032, performing channel compensation on the green channel and the blue channel to make the pixel average values ​​of the blue channel and the green channel consistent, and outputting the corrected blue channel and green channel;

[0077] Step 1033, using the corrected blue channel and green channel as increments to iteratively adjust the red channel, and output a preliminary enhanced image.

[0078] Further, step 1032 includes:

[0079] Step 10321, determining a first compensation coefficient based on the average pixel value of the blue channel and the average pixel value of the green channel.

[0080] In the embodiment of the present application, the color between the blue and green channels is corrected. The blue and green channels have smaller light attenuation and smaller difference in pixel average values. The above step 10321 includes:

[0081] δ s (x,y)=I l (x,y)(1―I s (x,y));

[0082] Among them, δ s (x, y) is the first compensation coefficient; I l (x, y) is the channel with the larger pixel average value in the blue channel and the green channel, which is used to compensate for the bright area of ​​the larger channel; I s (x,y) is the channel with the smaller pixel average value in the blue channel and the green channel (can be blue or green), which is used to compensate for the area that has experienced greater attenuation in the smaller channel.

[0083] Step 10322, based on the difference between the pixel average value of the blue channel and the pixel average value of the green channel and the first compensation coefficient, compensate the blue channel and the green channel with the smaller pixel average value, and output the blue channel and the green channel after the initial correction.

[0084] In the embodiment of the present application, the difference between the average pixel values ​​of the blue and green channels is used to compensate for the channel with a smaller average pixel value. The above step 10322 includes:

[0085]

[0086] Among them, I s (x, y) is the channel with the smaller pixel average value in the blue channel and the green channel (can be blue or green); δs (x, y) is the first compensation coefficient; is the pixel average of the larger one of the blue channel and the green channel; is the smaller average of the pixels in the blue and green channels.

[0087] Step 10323: Based on the pixel averages of the blue channel and the green channel after the initial correction, output the blue channel and the green channel after the secondary correction.

[0088] In the embodiment of the present application, based on the consistency principle of keeping the mean value of the color channel unchanged, we use the following formula to compensate the blue and green channels. The above step 10323 includes:

[0089]

[0090] Among them, I k (x, y) is the pixel value of the blue or green channel to be compensated; is the pixel average of the green channel; is the pixel average of the blue channel; is the pixel average value of the channel to be compensated. Through this step, we have completed the correction of the blue and green channels. This processing method not only fully retains the effective information between the two channels, but also adjusts the color deviation to ensure that the pixel average values ​​of the two channels are consistent, which is convenient for subsequent processing.

[0091] Furthermore, step 1033 includes:

[0092] Step 10331, determine the second compensation coefficient based on the smaller pixel average value of the blue channel and the green channel after the secondary correction and the pixel average value of the current red channel.

[0093] In the embodiment of the present application, the above step 10331 includes:

[0094]

[0095] Among them, Δδ is the second compensation coefficient, that is, the red channel compensation coefficient; is the pixel average value of the current red channel; is the pixel average of the smaller one of the blue channel and the green channel after quadratic correction; here, The value of Continuously update and control the criterion of stopping iteration Δδ<Δ th , ensuring that the pixel average of the red channel is close to the average value of the target brightness, that is, Δ th is the preset threshold (close to 0).

[0096] Step 10332, based on the smaller pixel value of the blue channel and the green channel after the secondary correction and the second compensation coefficient, determine the pixel value of the compensated red channel, and output a preliminary enhanced image; wherein, based on the smaller pixel average value of the blue channel and the green channel after the secondary correction, the pixel average value of the current red channel is consistent.

[0097] In the embodiment of the present application, the red channel is corrected and constrained by the blue and green channels after the secondary correction to prevent the pixel values ​​of certain areas of the red channel from being disproportionately higher than the pixel values ​​of the other two channels, and the smaller value of the other two channels is selected to iteratively compensate the value of the red channel. The above step 10332 includes:

[0098] I min (x,y)=min{I g (x,y),I b (x,y)};

[0099] I r (x,y)=I r (x,y)+Δδ*I min (x,y);

[0100] Among them, I min (x, y) is the smaller pixel value between the blue channel and the green channel after secondary correction; g (x, y) is the pixel value of the green channel; I b (x, y) is the pixel value of the blue channel; I r (x,y) is the red channel pixel value.

[0101] Step 104: Determine the target style image using a feature matching style selection module.

[0102] In the embodiment of the present application, step 104 includes:

[0103] The preliminary enhanced image and the style dataset are input into the feature matching style selection module, and the feature matching style selection module determines the target style image based on the pre-trained perceptual similarity measurement network; wherein the style dataset is the UIEB dataset.

[0104] Step 105: input the target style image and the preliminary enhanced image into a style transfer module, and output a final enhanced image.

[0105] In an embodiment of the present application, the style transfer module adopts a RevNet encoder-decoder structure.

[0106] For more information, see Figure 4, RevNet is a variant of ResNets, in which the activation map of each layer can be accurately reconstructed from the activation map of the next layer. Therefore, during backpropagation, the activation maps of most layers do not need to be stored in memory, which can reduce memory consumption and significantly improve the ability of model training. In this embodiment, a RevNet content and style encoder containing 10 residual blocks is designed, and the residual function F is implemented by a continuous Conv layer with a kernel size of 3; except for the last convolution layer, each convolution layer is followed by a relu layer; in order to capture large-scale style information, a squeeze module is used to reduce the spatial information by 2 times and increase the channel dimension by 4 times; a channel refinement module (CR) is used to reduce redundant information during forward reasoning; first, an injective filling module that increases the potential dimension is used to ensure that the image feature channel of the input content / style can be divided by the target channel; then, a patch-wise reversible residual block is used to integrate large receptive field information; then the channel information is propagated to a patch in the spatial dimension; finally, the decoder part corresponds to the encoder structure, and the fused features are mapped to the final enhanced image through reverse reasoning.

[0107] Effect test:

[0108] In a specific embodiment, the test is performed based on the above-mentioned style transfer underwater image enhancement method based on feature enhancement:

[0109] 1. Dataset preprocessing

[0110] The UIEB dataset consists of 890 paired underwater images and 60 unpaired underwater images. First, all images are resized to 128*128.

[0111] 2. Training process

[0112] Use PyCharm software to create an underwater image enhancement model in PyTorch; set the training details: use the Adam optimizer with a batch size of 1 to iterate the network for 10,000 times, set the initial learning rate to 1e-4, and decay at 5e-5; set the weight factor of the loss function to λ m =1200,λ cyc =10; Save the weights of underwater image enhancement model training.

[0113] 3. Testing

[0114] On the test set, the underwater image enhancement model is run using the trained weights, and the results are shown in the following table:

[0115] Dataset UIEB(890) EUVP(3700) Methods PSNR↑ SSIM↑ PSNR↑ SSIM↑ PUIE 21.99 0.90 17.66 0.75 FUnIE-Gan 16.89 0.73 23.43 0.79 TUDA 19.02 0.85 19.18 0.81 UIESS 18.79 0.80 23.91 0.82 WaterNet 20.75 0.86 17.41 0.70 Ucolor 17.99 0.71 19.96 0.76 Ours 25.43 0.93 26.06 0.83

[0116] It can be shown that on the UIEB dataset, the enhancement indicators of the present application reached an SSIM of 0.93 and a PSNR of 25.43; on the EUVP dataset, the enhancement indicators of the present application reached an SSIM of 0.83 and a PSNR of 26.06, achieving superior enhancement performance.

[0117] As a supplementary note, Figure 5 A comparison diagram of the underwater image enhancement method based on style migration provided by an exemplary embodiment of the present application and other 6 deep learning methods on a data set is shown. Figure 6 A detailed diagram showing the enhancement effect of a style transfer underwater image enhancement method based on feature enhancement on a UIEB dataset provided by an exemplary embodiment of the present application is shown.

[0118] Figure 7 The structure block diagram of a style migration underwater image enhancement device based on feature enhancement provided by an exemplary embodiment of the present application is shown. The style migration underwater image enhancement device based on feature enhancement includes:

[0119] An acquisition module 701 is used to acquire a degraded underwater image to be enhanced;

[0120] A construction module 702 is used to construct an underwater image enhancement model, which includes a red channel prior module, a feature matching style selection module, and a style migration module;

[0121] The output module 703 is used to perform a priori enhancement preprocessing on the degraded underwater image to be enhanced by using the red channel priori module, separate the red, green and blue channels, perform channel compensation on the red, green and blue channels, and output a preliminary enhanced image;

[0122] A determination module 704, used to determine a target style image using a feature matching style selection module;

[0123] The output module 703 is further used to input the target style image and the preliminary enhanced image into the style transfer module, and output the final enhanced image.

[0124] In some embodiments, the output module 703 is also used to perform prior enhancement preprocessing on the degraded underwater image to be enhanced using the red channel prior module, and separate the three channels of red, green and blue; perform channel compensation on the green channel and the blue channel to make the pixel average values ​​between the blue channel and the green channel consistent, and output the corrected blue channel and green channel; use the corrected blue channel and green channel as increments to iteratively adjust the red channel, and output a preliminary enhanced image.

[0125] In some embodiments, the output module 703 is also used to determine a first compensation coefficient based on the size of the pixel average value of the blue channel and the pixel average value of the green channel; based on the difference between the pixel average value of the blue channel and the pixel average value of the green channel and the first compensation coefficient, compensate the blue channel and the green channel with the smaller pixel average value, and output the blue channel and the green channel after the initial correction; based on the pixel average values ​​of the blue channel and the green channel after the initial correction, output the blue channel and the green channel after the secondary correction.

[0126] In some embodiments, the output module 703 is further used to determine the second compensation coefficient based on the smaller pixel average value of the blue channel and the green channel after the secondary correction and the current pixel average value of the red channel; determine the pixel value of the compensated red channel based on the smaller pixel value of the blue channel and the green channel after the secondary correction and the second compensation coefficient, and output the preliminary enhanced image. Wherein, the smaller pixel average value of the blue channel and the green channel after the secondary correction is consistent with the current pixel average value of the red channel.

[0127] In some embodiments, the determination module 704 is further used to input the preliminary enhanced image and the style data set into the feature matching style selection module, and the feature matching style selection module determines the target style image based on a pre-trained perceptual similarity measurement network; wherein the style data set is a UIEB data set.

[0128] In some embodiments, the style transfer module adopts a RevNet encoder-decoder structure.

[0129] It should be noted that the style transfer underwater image enhancement device based on feature enhancement provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0130] Figure 8 A schematic diagram of the structure of a computer device for performing a style transfer underwater image enhancement method based on feature enhancement provided by an exemplary embodiment of the present application is shown, and the computer device includes:

[0131] The processor 801 includes one or more processing cores. The processor 801 executes various functional applications and data processing by running software programs and modules.

[0132] The receiver 802 and the transmitter 803 can be implemented as a communication component, and the communication component can be a communication chip. Optionally, the communication component can be implemented to include a signal transmission function. That is, the transmitter 803 can be used to transmit a control signal to the image acquisition device and the scanning device, and the receiver 802 can be used to receive the corresponding feedback instruction.

[0133] The memory 804 is connected to the processor 801 via a bus 805 .

[0134] The memory 804 may be used to store at least one instruction, and the processor 801 may be used to execute the at least one instruction to implement each step in the above method embodiment.

[0135] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, which is loaded and executed by a processor to implement the above-mentioned feature enhancement-based style transfer underwater image enhancement method.

[0136] The present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the underwater image enhancement method based on style migration based on feature enhancement described in any of the above embodiments.

[0137] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0138] It should be understood that the specific examples herein are only intended to help those skilled in the art to better understand the present disclosure, rather than to limit the scope of the present invention.

[0139] It is understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the present disclosure.

[0140] It can be understood that the various implementation modes described in this specification can be implemented individually or in combination, and the present disclosure is not limited to this.

[0141] Unless otherwise stated, all technical and scientific terms used in this disclosure have the same meaning as those generally understood by those skilled in the art of the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0142] It can be understood that the processor of the present disclosure can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method implementation method can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (DigitalSignalProcessor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the present disclosure can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the present disclosure can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined to perform. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0143] It is understood that the memory in the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (programmable ROM, PROM), an erasable programmable read-only memory (erasable PROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0144] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0146] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0148] In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0149] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0150] The above is only a specific implementation of this specification, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in this specification, which should be included in the protection scope of this specification. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A style transfer underwater image enhancement method based on feature enhancement, characterized in that: The method is applied to a computer device, and the method comprises: Acquiring a degraded underwater image to be enhanced; Constructing an underwater image enhancement model, wherein the underwater image enhancement model includes a red channel prior module, a feature matching style selection module, and a style migration module; The red channel prior module is used to perform prior enhancement preprocessing on the degraded underwater image to be enhanced, and the red, green and blue channels are separated. Channel compensation is performed on the red, green and blue channels, and a preliminary enhanced image is output, which includes: The red channel prior module is used to perform prior enhancement preprocessing on the degraded underwater image to be enhanced, and separate the three channels of red, green and blue; Performing channel compensation on the green channel and the blue channel to make the pixel average values ​​of the blue channel and the green channel consistent, and outputting the corrected blue channel and green channel, which includes: Determine a first compensation coefficient based on the size of the pixel average value of the blue channel and the pixel average value of the green channel; Based on the difference between the pixel average value of the blue channel and the pixel average value of the green channel and the first compensation coefficient, the one with the smaller pixel average value between the blue channel and the green channel is compensated, and the blue channel and the green channel after primary correction are output; based on the pixel average values ​​of the blue channel and the green channel after primary correction, the blue channel and the green channel after secondary correction are output; Among them, I k (x, y) is the pixel value of the blue or green channel to be compensated; is the pixel average of the green channel; is the pixel average of the blue channel; is the pixel average value of the channel to be compensated; The method uses the corrected blue channel and green channel as increments to iteratively adjust the red channel and output a preliminary enhanced image, which includes: Determine a second compensation coefficient based on the smaller pixel average value of the blue channel and the green channel after the secondary correction and the current pixel average value of the red channel; Among them, Δδ is the second compensation coefficient, that is, the red channel compensation coefficient; is the pixel average value of the current red channel; is the pixel average of the smaller one of the blue channel and the green channel after secondary correction; Based on the smaller pixel value of the blue channel and the green channel after the secondary correction and the second compensation coefficient, determine the pixel value of the red channel after compensation, and output a preliminary enhanced image; I min (x,y)=min{I g (x,y),I b (x,y)}; I r (x,y)=I r (x,y)+Δδ*I min (x,y); Among them, I min (x, y) is the smaller pixel value between the blue channel and the green channel after secondary correction; g (x, y) is the pixel value of the green channel; I b (x, y) is the pixel value of the blue channel; I r (x,y) is the red channel pixel value; Determining a target style image using the feature matching style selection module; The target style image and the preliminary enhanced image are input into the style transfer module, and a final enhanced image is output.

2. The method for underwater image enhancement based on style migration according to claim 1, characterized in that: The smaller one of the blue channel and the green channel based on the secondary correction has the same pixel average as the current pixel average of the red channel.

3. The method for underwater image enhancement based on style migration according to claim 2, characterized in that: The step of using the feature matching style selection module to determine the target style image includes: The preliminary enhanced image and the style data set are input into the feature matching style selection module, and the feature matching style selection module determines the target style image based on a pre-trained perceptual similarity measurement network; wherein the style data set is a UIEB data set.

4. The method for underwater image enhancement based on style migration according to claim 3, characterized in that: The style transfer module adopts the RevNet encoder-decoder structure.

5. A style transfer underwater image enhancement device based on feature enhancement, characterized in that: include: An acquisition module, used for acquiring a degraded underwater image to be enhanced; A construction module, used to construct an underwater image enhancement model, wherein the underwater image enhancement model includes a red channel prior module, a feature matching style selection module, and a style migration module; An output module, used to perform a priori enhancement preprocessing on the degraded underwater image to be enhanced by using the red channel priori module, separate the red, green and blue channels, perform channel compensation on the red, green and blue channels, and output a preliminary enhanced image; The output module is also used to perform prior enhancement preprocessing on the degraded underwater image to be enhanced by using the red channel prior module, and separate the red, green and blue channels; Perform channel compensation on the green channel and the blue channel to make the pixel average values ​​of the blue channel and the green channel consistent, and output the corrected blue channel and green channel; Iteratively adjust the red channel using the corrected blue channel and green channel as increments, and output a preliminary enhanced image; The output module is also used to determine a first compensation coefficient based on the size of the pixel average of the blue channel and the pixel average of the green channel; based on the difference between the pixel average of the blue channel and the pixel average of the green channel and the first compensation coefficient, compensate the one with the smaller pixel average between the blue channel and the green channel, and output the blue channel and the green channel after the initial correction; Based on the pixel averages of the blue channel and the green channel after the primary correction, output the blue channel and the green channel after the secondary correction; Among them, I k (x, y) is the pixel value of the blue or green channel to be compensated; is the pixel average of the green channel; is the pixel average of the blue channel; is the pixel average value of the channel to be compensated; The output module is also used to determine a second compensation coefficient based on the smaller pixel average value of the blue channel and the green channel after the secondary correction and the current pixel average value of the red channel; Among them, Δδ is the second compensation coefficient, that is, the red channel compensation coefficient; is the pixel average value of the current red channel; is the pixel average of the smaller one of the blue channel and the green channel after secondary correction; Based on the smaller pixel value of the blue channel and the green channel after the secondary correction and the second compensation coefficient, determine the pixel value of the red channel after compensation, and output a preliminary enhanced image; I min (x,y)=min{I g (x,y),I b (x,y)}; I r (x,y)=I r (x,y)+Δδ*I min (x,y); Among them, I min (x, y) is the smaller pixel value between the blue channel and the green channel after secondary correction; g (x, y) is the pixel value of the green channel; I b (x, y) is the pixel value of the blue channel; I r (x,y) is the red channel pixel value; A determination module, used to determine a target style image using the feature matching style selection module; The output module is further used to input the target style image and the preliminary enhanced image into the style transfer module, and output a final enhanced image.

6. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the style migration underwater image enhancement method based on feature enhancement as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by a processor to implement the style migration underwater image enhancement method based on feature enhancement as described in any one of claims 1 to 4.

Images