A method and system for enhancing underwater image color constancy

Abstract

The present invention discloses a method and system for enhancing the color constancy of underwater images, and relates to the technical field of underwater image enhancement, comprising: receiving underwater image data, inputting the underwater image data into a pre-established multi-level color correction model based on an adaptive correction factor, and outputting a corrected image; inputting the corrected image into a pre-established multi-scale joint restoration network model, and outputting a restored image; inputting the restored image into a pre-established residual-based detail enhancement network model, and outputting an enhanced image; inputting the enhanced image into a pre-established underwater image enhancement network model for optimization training, and outputting a trained underwater image enhancement network model; obtaining an underwater image data set to be tested, and inputting the underwater image data set to be tested into the trained underwater image enhancement network model, and outputting an underwater image enhancement result, thereby significantly improving the color reality and naturalness of the image.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater image enhancement, and in particular to a method and system for enhancing the color constancy of underwater images. Background Art

[0002] With the breakthrough progress of deep learning methods in many low-level visual tasks, more and more researchers are applying them to the field of underwater image enhancement. In recent years, researchers have invested a lot of energy in the innovation of sample generation strategies, learning strategies and network architectures. Through automatic learning of training data, valuable information is extracted from images and applied to enhancement tasks. Although the underwater image enhancement network based on deep learning has made some progress, there is a general problem of not fully considering the physical characteristics of the image. Most of the enhancement networks ignore the underwater optical properties, such as the scattering and absorption of light. Short-wavelength blue and green light have a relatively long propagation distance in water, while long-wavelength red light is quickly absorbed. Relying solely on feature learning, it is impossible to accurately restore physically real color information. Secondly, underwater images are often accompanied by noise such as haze caused by suspended particles. Deep learning networks lack physical mechanism modeling for noise, which can easily amplify noise or cause loss of details. In addition, different water depths and water quality affect light propagation. The existing network does not explicitly incorporate these physical information, has limited generalization ability, and is difficult to adapt to diverse underwater environments.

[0003] The introduction of color balancing technology provides an effective means to alleviate the color distortion of underwater images and plays a certain role in underwater image enhancement. However, since underwater lighting conditions are far more complex than traditional color balancing scenes, methods that rely solely on color balancing are still difficult to cope with the diversity of underwater environments and maintain the color constancy of underwater images. When dealing with the color problems of underwater images, the following key challenges are still faced: first, the complex lighting changes in the underwater environment lead to unstable color correction and enhancement effects; second, existing color enhancement methods often focus on global processing and are difficult to effectively deal with local lighting and color differences. At present, there is little research on reducing color deviations and improving the color constancy of underwater images. Summary of the invention

[0004] In order to solve the deficiencies mentioned in the above background technology, the object of the present invention is to provide a method and system for enhancing the color constancy of underwater images.

[0005] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a method for enhancing the color constancy of underwater images, the method comprising the following steps:

[0006] Receiving underwater image data, inputting the underwater image data into a pre-established multi-level color correction model based on an adaptive correction factor, and outputting a corrected image;

[0007] The corrected image is input into a pre-established multi-scale joint restoration network model, and the restored image is output; the restored image is input into a pre-established residual-based detail enhancement network model, and the enhanced image is output;

[0008] The enhanced image is input into a pre-established underwater image enhancement network model for optimization training, and the trained underwater image enhancement network model is output to obtain an underwater image dataset to be tested, and the underwater image dataset to be tested is input into the trained underwater image enhancement network model to output the underwater image enhancement result.

[0009] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of inputting the underwater image data into a pre-established multi-level color correction model based on an adaptive correction factor is as follows:

[0010] The color statistical characteristics of underwater image data are used to estimate the illumination color, and the multi-level color correction is completed by combining the adaptive correction factor. The color statistical characteristics of the image include: average value, maximum value and root mean square value;

[0011] The multi-level color estimation is based on the following formula and handles global and local illumination differences in a hierarchical manner:

[0012]

[0013] in Indicates the spatial position, is the pixel value distribution function of a color channel of the image, and is a constant, Indicates the color of the light, Representation function of Power in the independent variable The integral on Represents a variable To perform integration, is the integral symbol, Indicates that the variable The variation used in the quadrature is based on the accurate estimation of the underwater illumination color, and a multi-level correction is performed on the color of the underwater image. The correction process is implemented by the following formula:

[0014]

[0015] in Is a point convolution operation, using The convolution kernel of the input image The channel information of each pixel is extracted. represents a vector of statistical features of the image color channels, and , Represents the statistical characteristics of the red channel of the image, Represents the statistical characteristics of the green channel of the image, Represents the statistical characteristics of the blue channel of the image, , , They are red, green and blue color channels. As parameters, Represents the corrected images under different correction strategies, represents pixel-level multiplication, where It is a multi-level color adaptive correction factor.

[0016] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of inputting the corrected image into a pre-established multi-scale joint restoration network model and outputting a restored image:

[0017] The pre-established multi-scale joint restoration network model MJRN is built on the Jaffe-McGlamery model, which decomposes the light energy source of underwater images as follows:

[0018]

[0019] in represents the image captured by the camera in an underwater environment, represents the original clear underwater image, represents the transmittance, Indicates background light;

[0020] Using a joint parameter estimation method, the transmittance is estimated simultaneously and background light , merged into joint parameters Optimize and restore the image. The process is as follows:

[0021]

[0022] in is a parameter, and .

[0023] Multi-scale joint restoration can be divided into three stages and restoration formulas. The basic structure of each stage consists of convolution, normalization and activation function.

[0024] In combination with the first aspect, in some implementations of the first aspect, the method further includes: introducing a dual-mode self-attention module DSM in the normalization stage, and the expression is as follows:

[0025]

[0026]

[0027] in, represents the attention operation, Represents the feature map of the output of stage 1, Indicates that the receptive field size in stage 1 is The contextual features of Represents the feature map of the output of stage 2, Indicates that the receptive field size in stage 2 is The contextual features of Indicates the size of the convolution kernel is The convolution operation, represents the combination of batch normalization and PReLU function concatenation operation, Represents channel-level connections;

[0028] The implementation process of DSM is as follows:

[0029] DSM compresses the encoded features into two channel descriptors through different pooling operations, captures the key information in the image and compresses it into a compact feature representation, captures the channel dependency between features through the compression excitation process, and is expressed by the following mathematical formula:

[0030]

[0031] In the formula and Represent the average pooling layer and the maximum pooling layer respectively, represents the combination of 3×3 convolutional layer, PReLU function and 3×3 convolutional layer, is a sigmoid function with channel weights after the channel attention module The features are fed into the next stage, where an average pooling layer is used to reduce the scale of the features and then the query vector , key vector Sum value vector To capture spatial dependencies:

[0032]

[0033] in, Indicates that the size of the convolution kernel is a 3×3 convolution operation. is a function, is the spatial weight, For learnable parameters, use deconvolution Rescale the features to the input scale, combine the spatial weights with the input features, fuse the two attention features, and combine them with the original input By performing residual connections, DSM can capture hidden features with prominent features of regional degradation. , the formula is as follows:

[0034]

[0035] In combination with the first aspect, in some implementations of the first aspect, the method further includes: introducing a residual connection in the activation function stage, and performing output processing through a multi-scale feature fusion module MFM, and the calculation process is as follows:

[0036]

[0037]

[0038] in, represents the multi-scale feature fusion operation, Represents a combination of batch normalization and PreLU function concatenation operations, represents the calibrated image, Indicates that the receptive field size in stage three is The contextual features of is the output of the branch, Represents pixel-level addition.

[0039] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of inputting the restored image into a pre-established residual-based detail enhancement network model:

[0040] The residual-based detail enhancement network model RDEN effectively transfers shallow information to deep layers by adaptively learning representative features. With the help of skip connections and DSM, RDEN weights the key features in the input image. RDEN generates feature maps through convolutional layers and activates them through PReLU activation layers. Subsequently, the image passes through three convolutional blocks in sequence. Each convolutional block contains three groups of convolutional layers and PReLU activation functions. Through skip connections, the weighted input image is combined with the output of the convolutional block.

[0041] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of inputting the enhanced image into a pre-established underwater image enhancement network model for optimization training:

[0042] Use image features to calculate errors and joint loss functions calculate:

[0043] The joint loss function consists of the following two parts, where the first part of the loss function is It is expressed as:

[0044]

[0045] in, represents the total number of samples, For the output image, is the reference image, Expressed as norm, is the index variable;

[0046] The second part is the perceptual loss function: The perceptual loss function uses the VGG16 model pre-trained on the ImageNet dataset. By inputting the enhanced image and the reference image into the pre-trained VGG network respectively, the feature difference at a specific layer is calculated. By selecting the features extracted by the Relu2_2 layer, the norm difference between the features of the enhanced image and the real underwater image at this layer is defined as the perceptual loss function. The perceptual loss function L p The formula is as follows:

[0047]

[0048] in, Represents a pre-trained network model;

[0049] Joint loss function as follows:

[0050]

[0051] in, and is the preset parameter, , .

[0052] In a second aspect, in order to achieve the above-mentioned object, the present invention discloses an underwater image color constancy enhancement system, comprising:

[0053] A color correction module is used to receive underwater image data, input the underwater image data into a pre-established multi-level color correction model based on adaptive correction factors, and output a corrected image;

[0054] An image processing module is used to input the corrected image into a pre-established multi-scale joint restoration network model to output a restored image, and input the restored image into a pre-established residual-based detail enhancement network model to output an enhanced image;

[0055] The image enhancement module is used to input the enhanced image into a pre-established underwater image enhancement network model for optimization training, output the trained underwater image enhancement network model, obtain the underwater image data set to be tested, input the underwater image data set to be tested into the trained underwater image enhancement network model, and output the underwater image enhancement result.

[0056] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, a method for enhancing underwater image color constancy as described above is adopted.

[0057] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a computer-readable storage medium is disclosed, in which a computer program is stored. When the computer program is loaded and executed by a processor, the method for enhancing the color constancy of underwater images as described above is adopted.

[0058] Beneficial effects of the present invention:

[0059] The present invention aims at the significant color deviation problem caused by the complex underwater lighting environment. The MLCC-AF layered method realizes underwater image color correction. After repairing the color distortion of the underwater image, in order to further remove the haze effect in the image, MJRN enhances the clarity and visual quality of the underwater image in a complex environment. Finally, in order to further refine the dehazed underwater image, RDEN enhances the image details. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 It is a schematic flow chart of the method of the present invention;

[0062] Figure 2 This is a general framework diagram of an underwater image color constancy enhancement network of the present invention;

[0063] Figure 3 A schematic diagram of a multi-level color correction model MLCC-AF based on an adaptive correction factor adopted by the present invention;

[0064] Figure 4 This is the effect diagram of MLCC-AF and its various branches;

[0065] Figure 5 Schematic diagram of the multi-scale joint restoration network MJRN used in the present invention;

[0066] Figure 6 Schematic diagram of the dual-mode self-attention module DSM used in the present invention;

[0067] Figure 7 A schematic diagram of a multi-scale feature fusion module MFM used in the present invention;

[0068] Figure 8 Schematic diagram of the residual-based detail enhancement network RDEN adopted by the present invention;

[0069] Fig. 9 This is the result diagram of the present invention on the UIEB dataset;

[0070] Fig.10 This is the result diagram of the present invention on the LSUI dataset;

[0071] Fig.11 This is the result diagram of the present invention on the U45 data set;

[0072] Fig.12 Schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] Embodiment 1:

[0075] like Figure 1 As shown, a method for enhancing the color constancy of underwater images comprises the following steps:

[0076] S101: receiving underwater image data, inputting the underwater image data into a pre-established multi-level color correction model based on an adaptive correction factor, and outputting a corrected image;

[0077] Specifically, the underwater image enhancement network is as follows Figure 2 As shown in the figure, the network consists of a multi-level color correction model MLCC-AF based on adaptive correction factors, a multi-scale joint restoration network MJRN, and a residual-based detail enhancement network RDEN connected in series:

[0078] First, the underwater image is input into the multi-level color correction model MLCC-AF based on adaptive correction factors. The model contains three branches. The first one is composed of a global color correction module and a 3×3 convolutional layer connected in series, the second one is composed of a local highlight area color correction module and a 3×3 convolutional layer connected in series, and the third one is composed of a channel color energy correction module and a 3×3 convolutional layer connected in series. The results of the three branches are added together to obtain the corrected image, and the corrected image is input into the multi-scale joint restoration network MJRN. The network first passes through three branches. The first branch is composed of a 3×3 convolutional layer, a batch normalization layer, and a PReLU function in sequence to obtain result one. The second branch is composed of a 5×5 convolutional layer, a batch normalization layer, and a PReLU function in sequence to obtain result two. The third branch is composed of a 7×7 convolutional layer, a batch normalization layer, and a PReLU function in sequence to obtain result three. The three results are connected at the channel level, and the obtained feature map passes through the three branches again. The first branch passes through a 3×3 convolutional layer, a batch normalization layer, and a PReLU function. LU function, and then pass through the dual-mode self-attention module. The second branch passes through a 5×5 convolution layer, a batch normalization layer, and a PReLU function, and then connects to the result two at the channel level, and then passes through the dual-mode self-attention module. The third branch passes through a 7×7 convolution layer, a batch normalization layer, and a PReLU function, and then connects to the result three at the channel level, and then passes through the dual-mode self-attention module. The final results of the three branches are connected at the channel level and then input into three branches. The first branch passes through a 3×3 convolution layer, a batch normalization layer, and a PReLU function. The second branch passes through a 5×5 convolution layer, a batch normalization layer, and a PReLU function. The third branch passes through a 7×7 convolution layer, a batch normalization layer, and a PReLU function. The results of the three branches are added to the corrected image respectively, and the three results are input into the multi-scale feature fusion module. The results and the corrected image are input into the restoration formula together to obtain the restored image. Finally, the restored image is input into the residual-based detail enhancement network RDEN to obtain the enhanced image.

[0079] The process of inputting underwater image data into the pre-established multi-level color correction model based on adaptive correction factors is as follows:

[0080] Multi-level color correction model MLCC-AF based on adaptive correction factor Figure 3 As shown, it includes a global color correction module, a local highlight area color correction module, a channel color energy correction module, a convolution layer, and a pixel addition layer.

[0081] First, the underwater image is input into the global color correction module, the local highlight area color correction module and the channel color energy correction module respectively. The global color correction module contains two branches. One branch is a 3×3 convolution layer and a Tanh function connected in series. The other branch inputs an adaptive correction factor based on The color correction framework is here For global correction, the pixel-level results of the two branches are added together to obtain the output of the global color correction module. The local highlight area color correction module contains two branches, one of which is a 3×3 convolution layer and a Tanh function connected in series, and the other branch input is based on the adaptive correction factor. The color correction framework is here For local highlight area color correction, the results of the two branches are added at the pixel level to obtain the output result of the local highlight area color correction module. The channel color energy correction module contains two branches, one of which is a 3×3 convolution layer and a Tanh function connected in series, and the other branch input is based on the adaptive correction factor. The color correction framework is here For channel color energy correction, the results of the two branches are added pixel by pixel to obtain the output result of the channel color energy correction module. After that, the results of each module of the three modules are connected to a 3×3 convolution layer, and then the obtained results are input into the element addition layer to finally obtain the corrected image.

[0082] The color correction framework based on adaptive correction factors mainly uses the color statistical characteristics of the image to estimate the illumination color and combines it with the adaptive correction factor Complete multi-level color correction. The color statistical features of the image include: average, maximum and root mean square value: The framework multi-level color estimation is based on the following formula and hierarchically handles global and local illumination differences:

[0083]

[0084] in Indicates the spatial position, is the pixel value distribution function of a color channel of the image, and is a constant, Indicates the color of the light, Representation function of Power in the independent variable The integral on Represents a variable To perform integration, is the integral symbol, Indicates that the variable The variation used in the quadrature is based on the accurate estimation of the underwater illumination color, and a multi-level correction is performed on the color of the underwater image. The correction process is implemented by the following formula:

[0085]

[0086] in Is a point convolution operation, using The convolution kernel of the input image The channel information of each pixel is extracted. represents a vector of statistical features of the image color channels, and , Represents the statistical characteristics of the red channel of the image, Represents the statistical characteristics of the green channel of the image, Represents the statistical characteristics of the blue channel of the image, , , They are red, green and blue color channels. As parameters, Represents the corrected images under different correction strategies, represents pixel-level multiplication, where It is a multi-level color adaptive correction factor.

[0087] (1) When For global correction, based on the formula:

[0088]

[0089] From this formula, by calculating each color channel ( , , ) can be used to estimate the color of the light in the underwater environment. Since the light in the underwater environment is usually blue or green, the average color of the image also tends to the corresponding hue. In the image correction stage, the following formula is used to make accurate color correction adjustments to the image:

[0090]

[0091] By dividing the pixel value of each channel by its corresponding channel mean ( , and ) to adjust the ratio and realize the normalization of the image, so that the average value of each channel is equal and the overall color cast caused by changes in ambient light is reduced.

[0092] (2) When When the correction is for the local highlight area, the formula is:

[0093]

[0094] Local highlight areas are usually affected by underwater lighting, reflecting the intensity and color characteristics of underwater lighting. By correcting the highlight part in the image, the excessive influence of the light source on the color is reduced. The following correction methods are used to correct the highlight area:

[0095]

[0096] The maximum pixel value of each color channel ( , and ) normalization can effectively compress the color range of the highlight area and enhance the color realism and visual consistency of the local area of ​​the image.

[0097] (3) When This is channel energy correction, which balances the energy distribution of different color channels to ensure smooth and natural color transitions of the image. Based on the formula:

[0098]

[0099] RMS value ( , and ) can reflect the energy distribution characteristics of each color channel and can more accurately present the color characteristics of underwater lighting. However, since the lighting in the underwater environment often leads to energy imbalance between color channels, obvious color distortion is likely to appear in the image. For this reason, correction is performed:

[0100]

[0101] The energy distribution of each color channel is adjusted by RMS normalization to achieve energy balance between channels and make color transitions smoother and more natural.

[0102] The multi-level color correction model MLCC-AF based on adaptive correction factors and the effect diagram of each branch module in the model are shown in the figure below. Figure 4 shown.

[0103] S102: inputting the corrected image into a pre-established multi-scale joint restoration network model, outputting a restored image, and inputting the restored image into a pre-established residual-based detail enhancement network model, outputting an enhanced image;

[0104] The process of inputting the corrected image into the pre-established multi-scale joint restoration network model and outputting the restored image:

[0105] Multi-scale joint restoration network MJRN Figure 5 As shown, it consists of stage one, stage two, stage three and the recovery formula to improve clarity.

[0106] In stage one, the corrected image is input into three branches respectively. The first branch is composed of a 3×3 convolution layer, a batch normalization layer, and a PReLU function. The second branch is composed of a 5×5 convolution layer, a batch normalization layer, and a PReLU function. The third branch is composed of a 7×7 convolution layer, a batch normalization layer, and a PReLU function. The results of the three branches are then connected at the channel level to obtain the output feature map of stage one. The process is expressed in the following mathematical formula:

[0107]

[0108]

[0109] in, represents the calibrated image, Indicates that the receptive field size in stage 1 is The contextual features of Represents the feature map of the output of stage 1, Represents a combination of batch normalization and PreLU function concatenation operations, Indicates the size of the convolution kernel is The convolution operation, Represents a channel-level connection.

[0110] The output feature map of stage one is input into stage two. Stage two consists of three branches. The output feature map of stage one is input into the first branch of stage two, and passes through the 3×3 convolution layer, batch normalization layer, and PReLU function in sequence. The result is connected to the first branch result of stage one at the channel level, and then the result of the first branch of stage two is obtained through the dual-mode self-attention module DSM. The output feature map of stage one is input into the second branch of stage two, and passes through the 5×5 convolution layer, batch normalization layer, and PReLU function in sequence. The result is connected to the second branch result of stage one at the channel level, and then the result of the second branch of stage two is obtained through the dual-mode self-attention module DSM. The output feature map of stage one is input into the third branch of stage two, and passes through the 7×7 convolution layer, batch normalization layer, and PReLU function in sequence. The result is connected to the third branch result of stage one at the channel level, and then the result of the third branch of stage two is obtained through the dual-mode self-attention module DSM. Finally, the results of the three branches of stage two are connected at the channel level to obtain the output feature map of stage two. The process is expressed as follows by mathematical formula:

[0111]

[0112]

[0113] in, represents the attention operation, Indicates that the receptive field size in stage 2 is The contextual features of Represents the feature map output of this stage.

[0114] The dual-mode self-attention module DSM in stage 2 is as follows Figure 6 As shown in Figure 1, it is composed of a channel self-attention module and a spatial self-attention module in series. The channel self-attention module first inputs the DSM into two parallel branches. The first branch is composed of a maximum pooling layer, a 3×3 convolutional layer, a PReLU function, and a 3×3 convolutional layer in series. The second branch is composed of an average pooling layer, a 3×3 convolutional layer, a PReLU function, and a 3×3 convolutional layer in series. The results of the two branches are added pixel-wise, and then the result is multiplied pixel-wise with the input of the DSM through the sigmoid function to obtain the output feature map of the channel self-attention module. , the above process can be expressed by the following mathematical formula:

[0115]

[0116] In the formula and Represent the average pooling layer and the maximum pooling layer respectively, represents the combination of 3×3 convolutional layer, PReLU function and 3×3 convolutional layer, is the sigmoid function, is the original input.

[0117] Specifically, after the channel attention module, with channel weight The features are fed into the next stage, where an average pooling layer is used to reduce the scale of the features and then the query vector , key vector Sum value vector To capture spatial dependencies:

[0118]

[0119] in, Indicates that the size of the convolution kernel is a 3×3 convolution operation. is a function, is the spatial weight, For learnable parameters, use deconvolution Rescale the features to the input scale, combine the spatial weights with the input features, fuse the two attention features, and combine them with the original input By performing residual connections, DSM can capture hidden features with prominent features of regional degradation. , the formula is as follows:

[0120]

[0121] Specifically, the channel self-attention module outputs the feature map The input to the spatial self-attention module first passes through the average pooling layer and then is input to three parallel branches. All three branches are composed of a 3×3 convolution layer. The results of the first and second branches are multiplied pixel by pixel, and then the result is passed through the softmax function. The result is combined with the result of the third branch and the introduced learnable parameters. , the three are multiplied at the pixel level, and the output feature map is the same as the output feature map of the channel self-attention module The feature maps obtained by the average pooling layer are added, and the output results enter the 3×3 transposed convolution layer. The obtained feature map is combined with the feature map output by the channel self-attention module. Multiply at the pixel level, and finally add the pixel level with the input of the dual-mode self-attention module DSM to get the output of the dual-mode self-attention module DSM.

[0122] The output feature map of stage two is input to stage three. Stage three consists of three branches. The output feature map of stage two is input to the first branch of stage three, and passes through the 3×3 convolution layer, batch normalization layer, and PReLU function in turn. The result is added to the pixel level of the input corrected image to obtain the result of the first branch of stage three. The output feature map of stage two is input to the second branch of stage three, and passes through the 5×5 convolution layer, batch normalization layer, and PReLU function in turn. The result is added to the pixel level of the input corrected image to obtain the result of the second branch of stage three. The output feature map of stage two is input to the third branch of stage three, and passes through the 7×7 convolution layer, batch normalization layer, and PReLU function in turn. The result is added to the pixel level of the input corrected image to obtain the result of the third branch of stage three. The results of the three branches of stage three are input to the multi-scale feature fusion module MFM to obtain the output feature map of stage three. The process is expressed as follows by mathematical formula:

[0123]

[0124]

[0125] in, represents the multi-scale feature fusion operation, Indicates that the receptive field size in stage three is The contextual features of This is the output of the branch. represents pixel-level addition, Represents the combination of batch normalization and PreLU functions in series.

[0126] The multi-scale feature fusion module MFM in stage three is as follows Figure 7As shown, the result feature graphs of the three branches in stage 3 are set as , and The multi-scale feature fusion module MFM first performs channel-level splicing on the three feature maps, and then sequentially connects the 3×3 convolution layer, PReLU function, batch normalization layer, 3×3 convolution layer, and sigmoid function. The results are respectively compared with the result feature maps of the three branches in stage three. , and Multiply and fuse to get three results, and then add the three result feature maps to output.

[0127] The output feature map of stage 3 is used as a joint parameter to input the restoration formula. The restoration formula form is explained below: The multi-scale joint restoration network MJRN is built based on the Jaffe-McGlamery model. The Jaffe-McGlamery model decomposes the light energy source of underwater images as follows:

[0128]

[0129] in represents the image captured by the camera in an underwater environment, represents the original clear underwater image, represents the transmittance, represents the background light. The present invention adopts a joint parameter estimation method to estimate the transmittance at the same time. and background light , merged into joint parameters Optimize and derive the recovery formula. The process is as follows:

[0130]

[0131] in is a parameter, and , according to the derivation, the recovery formula is , the output feature map of stage three is used as , the input of the multi-scale joint restoration network MJRN is taken as , according to the restoration formula, image restoration is performed to obtain the restored image.

[0132] The restored image is input into the pre-established residual-based detail enhancement network RDEN, and the enhanced image is output:

[0133] The residual-based detail enhancement network RDEN is Figure 8As shown in the figure, the input is divided into two branches. One branch passes through a 3×3 convolutional layer, a PReLU function and a dual-mode self-attention module DSM to obtain the feature map of the first branch. The second branch passes through a 3×3 convolutional layer, a PReLU function and a convolutional block. The convolutional block is composed of a 3×3 convolutional layer, a PReLU function, a 3×3 convolutional layer, a PReLU function, a 3×3 convolutional layer, and a PReLU function in series to obtain the feature map of the second branch. Then the output of the first branch is connected to the output of the second branch at the channel level, and the result continues to be input into the convolutional block, and the output is connected to the feature map of the first branch at the channel level. Then the result is input into the convolutional block again, and then the output is connected to the feature map of the first branch at the channel level. Finally, after a 3×3 convolutional layer, an enhanced image is output to indicate the detail performance.

[0134] S103: Input the enhanced image into a pre-established underwater image enhancement network model for optimization training, output the trained underwater image enhancement network model, obtain an underwater image dataset to be tested, input the underwater image dataset to be tested into the trained underwater image enhancement network model, and output the underwater image enhancement result.

[0135] The process of inputting the enhanced image into the pre-established underwater image enhancement network model for optimization training:

[0136] Use image features to calculate errors and joint loss functions calculate:

[0137] The joint loss function consists of the following two parts, where the first part of the loss function is It is expressed as:

[0138]

[0139] in, represents the total number of samples, For the output image, is the reference image, Expressed as norm, is the index variable;

[0140] The second part is the perceptual loss function: The perceptual loss function uses the VGG16 model pre-trained on the ImageNet dataset. By inputting the enhanced image and the reference image into the pre-trained VGG network respectively, the feature difference at a specific layer is calculated. By selecting the features extracted by the Relu2_2 layer, the norm difference between the features of the enhanced image and the real underwater image at this layer is defined as the perceptual loss function. The perceptual loss function L p The formula is as follows:

[0141]

[0142] in, Represents a pre-trained network model;

[0143] Joint loss function as follows:

[0144]

[0145] in, and is the preset parameter, , .

[0146] The effectiveness of the proposed network in underwater image enhancement tasks is verified on the UIEB, LSUI and U45 datasets.

[0147] Specifically, the present invention is further described below by way of embodiments:

[0148] The experiments were conducted in the Pycharm simulation platform. The hardware environment is as follows: Intel (R) Xeon (R) Platinum 8352V processor (CPU), 24G memory, NVIDIA RTX4090 graphics processor (GPU), operating system is Ubuntu18.04, and the program running environment is as follows: Python version is 3.8, CUDA version is 11.3. In addition, the training epoch is set to 100, and the experimental hyperparameter settings are shown in Table 1:

[0149] Table 1 Experimental hyperparameter settings

[0150]

[0151] 800 images were randomly selected from the UIEB dataset for training, and the remaining 90 images were used for testing. 3879 images were selected from the LSUI dataset for training, and the remaining 400 images were used for testing, and tested on the U45 dataset.

[0152] Quantitative analysis of results:

[0153] The present invention uses four evaluation indicators to comprehensively compare the performance of different algorithms: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), underwater image quality evaluation (UIQM) and underwater color image quality evaluation (UCIQE). The comparative analysis results of the present invention on the UIEB dataset are shown in Table 2. All evaluation indicators have achieved the best performance, which is better than the existing mainstream algorithms. Specifically, the PSNR of the present invention is improved by 22.05% compared with Ucolor and 25.43% compared with U-shape, and the SSIM reaches 0.90, which fully demonstrates its significant advantages in restoring image texture details. At the same time, on UIQM and UCIQE, the values ​​are 3.208 and 0.623, respectively, which are the highest. It shows that the present invention has excellent performance in image color representation and contrast enhancement.

[0154] Table 2 Quantitative comparison of enhancement performance on UIEB dataset

[0155]

[0156] Table 3 is the quantitative comparison results of the present invention and other algorithms on the LSUI and U45 datasets. It can be seen from the table that the present invention has achieved significant advantages in various indicators, and many evaluation indicators rank first.

[0157] Table 3 Quantitative comparison of enhancement performance on LSUI and U45 datasets

[0158]

[0159] Results visualization analysis:

[0160] Fig. 9 , Fig.10 and Fig.11 The underwater image enhancement effects of the present invention on UIEB, LSUI and U45 datasets are shown respectively. It can be seen from the figures that the present invention significantly solves the serious underwater color cast problem through stable color correction, avoids the generation of artifacts, and enhances the image contrast in the underwater environment. At the same time, the present invention maintains the color naturalness and detail clarity while enhancing the blue, green and blurred underwater images.

[0161] Embodiment 2: In the second aspect, as Fig.12 As shown, in order to achieve the above-mentioned purpose, the present invention discloses an underwater image color constancy enhancement system, comprising:

[0162] The color correction module 11 is used to receive underwater image data, input the underwater image data into a pre-established multi-level color correction model based on adaptive correction factors, and output a corrected image;

[0163] The image processing module 12 is used to input the corrected image into a pre-established multi-scale joint restoration network model, output a restored image, and input the restored image into a pre-established residual-based detail enhancement network model, output an enhanced image;

[0164] The image enhancement module 13 is used to input the enhanced image into a pre-established underwater image enhancement network model for optimization training, output the trained underwater image enhancement network model, obtain the underwater image data set to be tested, input the underwater image data set to be tested into the trained underwater image enhancement network model, and output the underwater image enhancement result.

[0165] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0166] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by the processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0167] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0168] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A method for enhancing underwater image color constancy, characterized in that: The method comprises the following steps: Receiving underwater image data, inputting the underwater image data into a pre-established multi-level color correction model based on an adaptive correction factor, and outputting a corrected image; The process of inputting the underwater image data into the pre-established multi-level color correction model based on adaptive correction factors is as follows: The color statistical characteristics of underwater image data are used to estimate the illumination color, and the multi-level color correction is completed by combining the adaptive correction factor. The color statistical characteristics of the image include: average value, maximum value and root mean square value; The multi-level color estimation is based on the following formula and handles global and local illumination differences in a hierarchical manner: Where X represents the spatial position, f(X) is the pixel value distribution function of a color channel of the image, p and k are constants, e represents the color of the light, ∫(f(X)) p dX represents the integral of the p-th power of the function f(X) on the independent variable X, ∫dX represents the integral of the variable X, ∫ is the integral symbol, and dX indicates the change used when the variable X is integrated. Based on the accurate estimation of the underwater illumination color, the color of the underwater image is corrected at multiple levels, and the correction process is implemented by the following formula: Among them, Conv 1×1 (x) is a point convolution operation that uses a 1×1 convolution kernel to extract the channel information of each pixel of the input image x. represents a vector of statistical features of the image color channels, and A R,P＝i Represents the statistical characteristics of the red channel of the image, A G,P＝i Represents the statistical characteristics of the green channel of the image, A B,P＝i Represents the statistical characteristics of the blue channel of the image. R, G, and B are the red, green, and blue color channels respectively. i is a parameter, and x i Represents the corrected images under different correction strategies, represents pixel-level multiplication, where p is a multi-level color adaptive correction factor; The corrected image is input into a pre-established multi-scale joint restoration network model, and the restored image is output; the restored image is input into a pre-established residual-based detail enhancement network model, and the enhanced image is output; The enhanced image is input into a pre-established underwater image enhancement network model for optimization training, and the trained underwater image enhancement network model is output to obtain an underwater image dataset to be tested, and the underwater image dataset to be tested is input into the trained underwater image enhancement network model to output the underwater image enhancement result.

2. The method for enhancing underwater image color constancy according to claim 1, characterized in that: The process of inputting the corrected image into the pre-established multi-scale joint restoration network model and outputting the restored image: The pre-established multi-scale joint restoration network model MJRN is built on the Jaffe-McGlamery model, which decomposes the light energy source of underwater images as follows: I λ (x,y)=J λ (x,y)T λ (x,y)+B λ (1-T λ (x,y)) Among them I λ (x, y) represents the image captured by the camera in the underwater environment, J λ (x, y) represents the original clear underwater image, T λ (x,y) represents the transmittance, B λ Indicates background light; The transmittance T is estimated by using the joint parameter estimation method. λ (x,y) and background light B λ , merged into the joint parameter K λ (x,y) is optimized and the image is restored. The process is as follows: Where b is a parameter, and b=1; Multi-scale joint restoration can be divided into three stages and restoration formulas. The basic structure of each stage consists of convolution, normalization and activation function.

3. The method for enhancing underwater image color constancy according to claim 2, characterized in that: The normalization stage introduces a dual-mode self-attention module DSM, which is expressed as follows: Among them, DSM(·) represents the attention operation, M 1 Represents the feature map of the output of stage 1, represents the context feature with a receptive field size of s at stage 1, M 2 Represents the feature map of the output of stage 2, Indicates the context feature with a receptive field size of s in stage 2, Conv i×i Indicates that the size of the convolution kernel is an i×i convolution operation, Φ BP (·) represents the combination of batch normalization and PreLU function concatenation, and ⊙ represents channel-level connection; The implementation process of DSM is as follows: DSM compresses the encoded features into two channel descriptors through different pooling operations, captures the key information in the image and compresses it into a compact feature representation, captures the channel dependency between features through the compression excitation process, and is expressed by the following mathematical formula: Where AvgP(·) and MaxP(·) represent the average pooling layer and the maximum pooling layer, respectively. cpc represents the combination of 3×3 convolutional layer, PReLU function and 3×3 convolutional layer, (·) sig is a sigmoid function with channel weight f after the channel attention module c The features are fed into the next stage, where an average pooling layer is used to reduce the scale of the features, and then the spatial dependencies are captured by the query vector Q, key vector K, and value vector V: in,(·) c Indicates that the size of the convolution kernel is 3×3 convolution operation, (·) soft is a softmax function, f s is the spatial weight, γ is a learnable parameter, and deconvolution (·) is used. dc By rescaling the features to the input scale and combining the spatial weights with the input features, fusing the two attention features and performing a residual connection with the original input f, DSM is able to capture the hidden features f with regional degradation salient features h , the formula is as follows:

4. The method for enhancing underwater image color constancy according to claim 3, characterized in that: The activation function stage introduces residual connection and performs output processing through the multi-scale feature fusion module MFM. The calculation process is as follows: Among them, MFM(·) represents the multi-scale feature fusion operation, Φ BP (·) represents the combination of batch normalization and PreLU function in series, x in represents the calibrated image, represents the context feature with a receptive field size of s in stage three, M 3 is the output of the branch, Represents pixel-level addition.

5. The method for enhancing underwater image color constancy according to claim 4, characterized in that: The process of inputting the restored image into the pre-established residual-based detail enhancement network model: The residual-based detail enhancement network model RDEN effectively transfers shallow information to deep layers by adaptively learning representative features. With the help of skip connections and DSM, RDEN weights the key features in the input image. RDEN generates feature maps through convolutional layers and activates them through PReLU activation layers. Subsequently, the image passes through three convolutional blocks in sequence. Each convolutional block contains three groups of convolutional layers and PReLU activation functions. Through skip connections, the weighted input image is combined with the output of the convolutional block.

6. The method for enhancing underwater image color constancy according to claim 1, characterized in that: The process of inputting the enhanced image into the pre-established underwater image enhancement network model for optimization training: Use image features to calculate the error and the joint loss function L total calculate: The joint loss function consists of the following two parts, where the first part of the loss function L2 is expressed as: Where N represents the total number of samples, J(x) is the output image, and J * (x) is the reference image, ||·|| represents the L2 norm, and i is the index variable; The second part is the perceptual loss function: The perceptual loss function uses the VGG16 model pre-trained on the ImageNet dataset. By inputting the enhanced image and the reference image into the pre-trained VGG network respectively, the feature difference at a specific layer is calculated. By selecting the features extracted by the Relu2_2 layer, the norm difference between the features of the enhanced image and the real underwater image at this layer is defined as the perceptual loss function. The perceptual loss function L p The formula is as follows: Among them, V(·) represents the pre-trained network model; Joint loss function L total as follows: L total =ω1·L2(J(x),J * (x))+ω2·L p (J(x),J * (x)) Among them, ω1 and ω2 are preset parameters.

7. An underwater image color constancy enhancement system, using an underwater image color constancy enhancement method according to any one of claims 1 to 6, characterized in that: include: A color correction module is used to receive underwater image data, input the underwater image data into a pre-established multi-level color correction model based on adaptive correction factors, and output a corrected image; An image processing module is used to input the corrected image into a pre-established multi-scale joint restoration network model to output a restored image, and input the restored image into a pre-established residual-based detail enhancement network model to output an enhanced image; The image enhancement module is used to input the enhanced image into a pre-established underwater image enhancement network model for optimization training, output the trained underwater image enhancement network model, obtain the underwater image data set to be tested, input the underwater image data set to be tested into the trained underwater image enhancement network model, and output the underwater image enhancement result.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, a method for enhancing underwater image color constancy according to any one of claims 1 to 6 is adopted.

9. A computer-readable storage medium having a computer program stored therein, characterized in that: When the computer program is loaded and executed by the processor, the method for enhancing underwater image color constancy according to any one of claims 1 to 6 is adopted.

Images