Underwater image enhancement method guided by joint perceptual image metric and contrastive learning

Abstract

The application provides an underwater image enhancement method combining an image perception measure with contrast learning guidance, comprising the following steps: S1, acquiring a first underwater image dataset with pseudo reference labels, a second underwater image dataset without labels and a pseudo reference image dataset; S2, inputting the first underwater image dataset into an HR-Net network to obtain a first enhanced image dataset, introducing a structural similarity index to obtain a first loss function, and training the HR-Net network based on the first loss function to obtain a first enhancement network; S3, inputting the second underwater image dataset into the first enhancement network to obtain a second enhanced image dataset, combining the pseudo reference image dataset to obtain a second loss function, and training the first enhancement network based on the second loss function to obtain a second enhancement network; and S4, inputting an image to be enhanced into the second enhancement network to obtain an enhanced image. The application can break through the limitation of insufficient quality of pseudo reference labels and improve the underwater image enhancement effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of underwater image enhancement, in particular to an underwater image enhancement method combining image perception measure and contrast learning guidance. BACKGROUND

[0002] Autonomous underwater vehicle (AUV) has a wide range of application fields, such as marine archaeology, ocean exploration and underwater detection, etc. Underwater image enhancement (UIE) is an important technology that can improve the visual quality of the original underwater image, so that the autonomous underwater vehicle can see more clearly and reliably perform tasks. However, underwater image enhancement is a challenging task because the original underwater image has some inherent serious degradation problems, such as fogging effect, contrast reduction, color deviation and insufficient brightness, etc.

[0003] In the challenging task of underwater image enhancement, a large number of underwater image enhancement methods based on deep learning have made significant progress. These methods are mainly divided into three categories: supervised, semi-supervised and unsupervised. Compared with unsupervised methods, supervised and semi-supervised methods are usually more superior in performance. However, the current supervised and semi-supervised underwater image enhancement methods mostly train the network through artificially selected or algorithmically generated pseudo-reference labels (Pseudo-reference). Although this can achieve a certain degree of effective training, these pseudo-reference labels have obvious defects, as they often do not have perfect visual quality, which severely limits the upper limit of the performance of such training-efficient underwater image enhancement networks. From the numerical comparison results of the two methods, the influence of this limitation on network performance can be clearly seen. Figure 1 SUMMARY

[0004] The technical problem to be solved by the present application is to break through the limitation of the quality of pseudo-reference labels and improve the effect of underwater image enhancement. In order to overcome the defects of the above prior art (or related art), the present application provides an underwater image enhancement method combining image perception measure and contrast learning guidance.

[0005] The present application provides an underwater image enhancement method combining image perception measure and contrast learning guidance, comprising the following steps:

[0006] Step S1, obtaining a first underwater image dataset with pseudo-reference labels, a second underwater image dataset without labels and a pseudo-reference image dataset provided by a Semi-UIR network;

[0007] ​Step S2, inputting the first underwater image dataset into the HR-Net network to obtain a first enhanced image dataset, introducing a structural similarity index, combining the first enhanced image dataset and the first underwater image dataset to obtain a first loss function, and pre-training the HR-Net network based on the first loss function to obtain a first enhancement network;

[0008] Step S3, inputting the second underwater image dataset into the first enhancement network to obtain a second enhanced image dataset, obtaining a second loss function according to the second enhanced image dataset and the pseudo reference image dataset, and pre-training the first enhancement network based on the second loss function to obtain a second enhancement network;

[0009] Step S4, acquiring and inputting a to-be-enhanced image into the second enhancement network to obtain an enhanced image.

[0010] Compared with the prior art, the underwater image enhancement method guided by the joint image perception measure and the contrast learning has the following advantages:

[0011] In the present application, the acquisition of each image dataset is performed through step S1, the first enhancement network is trained and generated through step S2 by introducing the structural similarity index, the second enhancement network is further generated by further training the first enhancement network through step S3 by introducing the pseudo reference image dataset, and the underwater image enhancement is performed through step S4 by applying the second enhancement network. The first loss function is constructed from the image level and the area level before the network training process to guide the HR-Net network to surpass the pseudo reference image, improve the underwater image enhancement effect, and the second loss function is constructed from the depth level, and the enhancement effect of the low-quality far-area image block is recovered by means of the guidance of the high-quality near-area image block in the pseudo reference image, thereby breaking through the limitation of insufficient quality of the pseudo reference label, guiding the first enhancement network to surpass the pseudo reference image, and further improving the underwater image enhancement effect.

[0012] In a possible implementation, the step S2 includes:

[0013] Step S21, inputting a first original underwater image in the first underwater image dataset into the HR-Net network in sequence to obtain a corresponding first enhanced image to constitute the first enhanced image dataset;

[0014] Step S22, for each pixel point in each of the first enhanced images, a corresponding structural similarity index is calculated in a 13x13 image block range around the pixel point in combination with a reference image;

[0015] Step S23, obtaining the first loss function according to each structure similarity index, configuring a learning rate of 10e-4, a training round number of 200, and an optimizer of Adam, and pre-training the HR-Net network based on the first loss function to obtain the first enhancement network.

[0016] In a possible implementation, in the step S22, the structure similarity index is obtained by the following calculation formula:

[0017] ;

[0018] Wherein,

[0019] denotes the structure similarity index corresponding to the i th pixel point;

[0020] denotes the mean value of the image block centered on the i th pixel point in the reference image;

[0021] denotes the mean value of the image block centered on the i th pixel point in the first enhanced image;

[0022] denotes a first preset parameter;

[0023] denotes the covariance of the image block centered on the i th pixel point in the reference image and the first enhanced image;

[0024] denotes the standard deviation of the image block centered on the i th pixel point in the reference image;

[0025] denotes the standard deviation of the image block centered on the i th pixel point in the first enhanced image;

[0026] denotes a second preset parameter.

[0027] In a possible implementation, in the step S23, the first loss function is obtained by the following calculation formula:

[0028] ;

[0029] Wherein,

[0030] denotes the first loss function;​​​​​​

[0031] denotes a total number of pixel points in the first enhanced image;

[0032] denotes a structure similarity index corresponding to the i-th pixel point.

[0033] In a possible implementation, the step S3 comprises:

[0034] Step S31, sequentially inputting a second original underwater image in the second underwater image dataset into the first enhancement network to obtain a corresponding second enhanced image, and constructing the second enhanced image dataset by using the second enhanced image.

[0035] Step S32, inputting each second enhanced image and a pseudo reference image in the pseudo reference image dataset into a VGG-19 network to obtain a corresponding output feature, and obtaining a content preservation loss according to each output feature.

[0036] Step S33, constructing a quality score surpassing loss based on an image perception measure and a deep perception surpassing loss based on contrast learning guidance according to the second enhanced image dataset and the pseudo reference image dataset, and obtaining a pseudo reference surpassing loss according to the quality score surpassing loss based on the image perception measure and the deep perception surpassing loss based on the contrast learning guidance.

[0037] Step S34, obtaining the second loss function according to the content preservation loss and the pseudo reference surpassing loss, configuring a learning rate as 10e-4, a training round number as 200, and an optimizer as Adam, and pre-training the first enhancement network based on the second loss function to obtain the second enhancement network.

[0038] In a possible implementation, in the step S32, the content preservation loss is obtained by using the following calculation formula:

[0039]

[0040]

[0041] denotes the content preservation loss;

[0042] denotes a weight corresponding to the output feature;

[0043] denotes the output feature corresponding to the i-th layer of the VGG-19 network for the second enhanced image;

[0044] ​​​​ the pseudo reference image in the VGG-19 network at the i-th layer. the output feature corresponding to the i-th layer.

[0045] In a possible implementation, the step S33 comprises:

[0046] Step S331, inputting the second enhanced image dataset and the pseudo reference image set into the URanker network in sequence to obtain a first global quality score and a second global quality score, and obtaining a global comparison loss according to the first global quality score and the second global quality score;

[0047] Step S332, inputting the second enhanced image dataset and the pseudo reference image set into the RQSD-Net network to obtain a local region comparison result, constructing a local comparison loss according to the local region comparison result, and obtaining the quality score surpassing loss based on the image perception measure according to the global comparison loss and the local comparison loss;

[0048] Step S333, generating an image block mask for dividing near distance image blocks and far distance image blocks in the second original underwater image, extracting far distance image blocks in the second enhanced image and near distance image blocks and far distance image blocks in the pseudo reference image through the image block mask, and then calculating a contrast loss function between each of the far distance image blocks and each of the near distance image blocks to obtain the depth perception surpassing loss guided by contrast learning;

[0049] Step S334, obtaining the pseudo reference surpassing loss according to the quality score surpassing loss based on the image perception measure and the depth perception surpassing loss guided by contrast learning.

[0050] In a possible implementation, in the step S331, the global comparison loss is obtained through the following calculation formula:

[0051] ;

[0052] wherein,

[0053] denotes the global comparison loss;

[0054] denotes a sigmoid activation function;

[0055] denotes the first global quality score;

[0056] denotes the second global quality score;

[0057] represents the URanker network;

[0058] represents the second enhanced image;

[0059] represents the pseudo-reference image.

[0060] In a possible implementation, the local comparison loss in the step S332 is obtained by the following calculation formula:

[0061]

[0062] wherein,

[0063] represents the local comparison loss;

[0064] represents the number of rows of the local area comparison result;

[0065] represents the number of columns of the local area comparison result;

[0066] represents the local area comparison result;

[0067] represents the horizontal coordinate of a pixel point;

[0068] represents the vertical coordinate of a pixel point;

[0069] represents the RQSD-Net network;

[0070] represents the second enhanced image;

[0071] represents the pseudo-reference image.

[0072] In a possible implementation, the step S333 includes:

[0073] In step S3331, the second original underwater image is input into a depth prediction network to generate an underwater image depth map, and a division strategy is performed on the underwater image depth map to obtain the image block mask.

[0074] In step S3332, the second enhanced image and the image block mask are combined to extract a long-distance image block in the second enhanced image, and the pseudo-reference image and the image block mask are combined to extract a long-distance image block and a short-distance image block in the pseudo-reference image.​

[0075] Step S3333, taking the long-distance image block of the second enhanced image as an anchor point, taking the long-distance image block of the pseudo reference image as a negative sample, and taking the short-distance image block of the pseudo reference image as a positive sample to obtain the depth perception surpassing loss guided by contrast learning. BRIEF DESCRIPTION OF DRAWINGS

[0076] Figure 1 An average URanker score diagram of different underwater image enhancement methods in the background art on the UIEB-T90 dataset;

[0077] Figure 2 A step flowchart of the present application;

[0078] Figure 3 A training framework diagram of the first enhancement network of the present application;

[0079] Figure 4 A construction framework diagram of the quality score surpassing loss based on image perception measure of the present application;

[0080] Figure 5 A construction framework diagram of the depth perception surpassing loss guided by contrast learning of the present application;

[0081] Figure 6 An acquisition process diagram of the image block mask of the present application;

[0082] Figure 7 A decision diagram of the long-distance and short-distance image blocks of the present application;

[0083] Figure 8 An image comparison result diagram of the second original underwater image with different haze effects in the near distance and the long distance of the present application;

[0084] Figure 9 An image comparison result diagram of the second original underwater image mainly in yellow, green and blue tones of the present application;

[0085] Figure 10 An image comparison result diagram of the second original underwater image with different degrees of low light of the present application;

[0086] Figure 11 A quantitative comparison result diagram of the five sub-datasets under the UIQM, UCIQE, URANKER and RUIQA indicators of the present application;

[0087] Figure 12 A quantitative comparison result diagram of the 13 kinds of SOTA methods in the parameter quantity, floating point operation number and running time of the present application. DETAILED DESCRIPTION

[0088] First, those skilled in the art should understand that these embodiments are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can adjust them as needed in order to adapt to specific application occasions.

[0089] The present application will be further described in detail below in conjunction with the drawings and specific embodiments.

[0090] Reference Figure 2 , the embodiments of the present application disclose an underwater image enhancement method combining image perception measure and contrast learning guidance, comprising:

[0091] Step S1, obtaining a first underwater image dataset with pseudo-reference labels, a second underwater image dataset without labels, and a pseudo-reference image set provided by a Semi-UIR network;

[0092] Step S2, inputting the first underwater image dataset into an HR-Net network to obtain a first enhanced image dataset, introducing a structural similarity index and combining the first enhanced image dataset and the first underwater image dataset to obtain a first loss function, and pre-training the HR-Net network based on the first loss function to obtain a first enhancement network;

[0093] Step S3, inputting the second underwater image dataset into the first enhancement network to obtain a second enhanced image dataset, obtaining a second loss function according to the second enhanced image dataset and the pseudo-reference image set, and pre-training the first enhancement network based on the second loss function to obtain a second enhancement network;

[0094] Step S4, obtaining and inputting an image to be enhanced into the second enhancement network to obtain an enhanced image.

[0095] In the embodiments of the present application, a UIEB dataset is obtained, which includes 890 pairs of training subsets with pseudo-reference labels and 60 challenging test subsets without pseudo-reference labels, and the first underwater image dataset in step S1 is 800 pairs of datasets randomly selected from the 890 pairs of training subsets.

[0096] In the embodiments of the present application, an EUVP dataset is obtained, which includes about 20k paired and unpaired underwater images, and the second underwater image dataset in step S1 is 3205 images randomly selected from the unpaired underwater images.

[0097] In the embodiments of the present application, the HR-Net network used in step S2 is derived from the lightweight network of the second stage in the CCL-Net network, and the specific training process is as follows:

[0098] ;

[0099] in, This represents the first raw underwater image from the first underwater image dataset. This refers to the HR-Net network. This represents the first enhanced image generated by the HR-Net network;

[0100] During the training of the HR-Net network, the Structural Similarity Index (SSIM) is used to calculate the first loss function. The SSIM is calculated using grayscale images for each pixel. The structural similarity index is calculated within a 13×13 image patch surrounding each pixel, and is expressed as follows:

[0101] ;

[0102] ;

[0103] in, Indicates the first 1 pixel This represents the total number of pixels. and Represented by pixels The central image patch in the reference image and the first enhanced image The mean of the middle, and Represented by pixels The central image patch in the reference image and the first enhanced image The standard deviation in Represented by pixels The central image patch in the reference image and the first enhanced image Covariance between Take 0.02, Take 0.03; during the calculation process, when it is necessary to calculate the first... When calculating the structural similarity index of individual pixels, it can be... right By making substitutions, the following formula is formed:

[0104] .

[0105] In this embodiment of the application, step S2 includes:

[0106] Step S21, sequentially input the first original underwater image in the first underwater image data set into the HR-Net network to obtain the corresponding first enhanced image to constitute the first enhanced image data set;

[0107] Step S22, for each pixel point in each first enhanced image, the reference image is combined to calculate the corresponding structural similarity index in the 13x13 image block range around the pixel point;

[0108] Step S23, according to each structural similarity index, a first loss function is obtained, the learning rate is set to 10e-4, the training round number is set to 200, the optimizer is set to Adam, and the HR-Net network is pre-trained based on the first loss function to obtain the first enhanced network.

[0109] In the embodiment of the application, the training framework for training the first enhanced network in step S3 is as shown in Figure 3 Figure 3 The any UIE network in the upper half of the middle part represents the HR-Net network; Figure 3 The pseudo-reference generation network in the lower half of the middle part uses the pre-trained Semi-UIR network to provide the pseudo-reference image that is surpassed, and the specific training process can be represented as:

[0110] ;

[0111] Among them, represents the second original underwater image from the second underwater image data set; represents the first enhanced network; represents the second enhanced image generated by the first enhanced network; represents the pre-trained Semi-UIR network, and the parameters are frozen during the training process and are not changed; represents the pseudo-reference image generated by the Semi-UIR network;

[0112] The second loss function used in the training process is composed of two parts, namely the content preservation loss and the pseudo-reference surpassing loss , and the expression is:

[0113] ;

[0114] Among them, and are the weight coefficients of the two losses, which are set to 0.3 and 1.0 respectively;

[0115] The content preservation loss is the second enhanced image and the pseudo-reference image ​The perceptual loss of the pre-trained VGG-19 network used between them is expressed as:

[0116]

[0117] wherein, , represents the output features of the pre-trained VGG-19 network at the i-th layer, and the output features of the first, third, fourth, ninth and thirteenth layers are selected here, is the corresponding weight, is sequentially set to 1 / 32, 1 / 16, 1 / 8, 1 / 4 and 1;

[0118] Pseudo-reference transcendence loss The pseudo-reference transcendence loss is composed of two parts, i.e., a quality score transcendence loss based on image perceptual measure and a deep perceptual transcendence loss based on contrast learning guidance , and is expressed as:

[0119]

[0120] wherein, and are the weight coefficients of the two losses, and are set to 1.0 and 0.2, respectively.

[0121] In the embodiments of the present application, step S3 comprises:

[0122] Step S31, sequentially input the second original underwater image in the second underwater image data set into the first enhancement network to obtain the corresponding second enhanced image to form a second enhanced image data set;

[0123] Step S32, input each second enhanced image and the pseudo-reference image in the pseudo-reference image set into the VGG-19 network to obtain the corresponding output features, and obtain the content preservation loss according to each output feature;

[0124] Step S33, based on the second enhanced image data set and the pseudo-reference image set, construct the quality score transcendence loss based on image perceptual measure and the deep perceptual transcendence loss based on contrast learning guidance, and obtain the pseudo-reference transcendence loss according to the quality score transcendence loss based on image perceptual measure and the deep perceptual transcendence loss based on contrast learning guidance;

[0125] Step S34, obtain the second loss function according to the content preservation loss and the pseudo-reference transcendence loss, configure the learning rate to be 10e-4, the training round number to be 200, and the optimizer to be Adam, and pre-train the first enhancement network based on the second loss function to obtain the second enhancement network.

[0126] In the embodiments of the present application, step S33 comprises:​​​

[0127] In step S331, the second enhanced image dataset and the pseudo reference image set are sequentially input into the URanker network to obtain a first global quality score and a second global quality score, and a global comparison loss is obtained according to the first global quality score and the second global quality score.

[0128] In step S332, the second enhanced image dataset and the pseudo reference image set are input into the RQSD-Net network to obtain a local region comparison result, a local comparison loss is constructed according to the local region comparison result, and a quality score surpassing loss based on an image perception measure is obtained according to the global comparison loss and the local comparison loss.

[0129] In step S333, an image block mask for dividing near distance image blocks and far distance image blocks is generated in the second original underwater image, far distance image blocks in the second enhanced image and near distance image blocks and far distance image blocks in the pseudo reference image are extracted through the image block mask, and then a contrast loss function between each far distance image block and each near distance image block is calculated to obtain a depth perception surpassing loss guided by contrast learning.

[0130] In step S334, a pseudo reference surpassing loss is obtained according to the quality score surpassing loss based on the image perception measure and the depth perception surpassing loss guided by contrast learning.

[0131] In the embodiment of the present application, step S333 comprises:

[0132] In step S3331, the second original underwater image is input into a depth prediction network to generate an underwater image depth map, and a division strategy is performed on the underwater image depth map to obtain an image block mask.

[0133] In step S3332, the second enhanced image and the image block mask are combined to extract far distance image blocks in the second enhanced image, and the pseudo reference image and the image block mask are combined to extract near distance image blocks and far distance image blocks of the pseudo reference image.

[0134] In step S3333, the far distance image blocks of the second enhanced image are taken as anchor points, the far distance image blocks of the pseudo reference image are taken as negative samples, and the near distance image blocks of the pseudo reference image are taken as positive samples to obtain the depth perception surpassing loss guided by contrast learning.

[0135] In the embodiment of the present application, the framework for constructing the quality score surpassing loss based on the image perception measure is as shown in Figure 4 The quality score surpassing loss based on the image perception measure is composed of two parts, i.e., a global comparison loss and a local comparison loss , and the expression is:

[0136] ;

[0137] in, and These are the weighting coefficients for the two losses, set to 0.03 and 0.1 respectively;

[0138] For global comparison loss ,like Figure 4 As shown on the left, the second enhanced image and pseudo reference image The inputs are fed into the same pre-trained Global Underwater Image Quality Assessment (UWIQA) network. In this process, after passing through the network, global quality scores for the two images are generated. and Used for building The expression is:

[0139] ;

[0140] in, This represents the sigmoid activation function. It uses a pre-trained URanker network. By comparing the second enhanced image and pseudo reference image The global quality score guides the first enhancement network to generate an enhanced image that surpasses the pseudo-reference image in global quality;

[0141] For local comparison loss ,like Figure 4 As shown on the right, the second enhanced image and pseudo reference image They are then fed together into another pre-trained Local Underwater Image Quality Assessment (UWIQA) network. After passing through this network, a local region comparison result of size 16×16 will be generated. Used for building The expression is:

[0142] ;

[0143] Where R and C are the results of local region comparisons. The number of rows and columns, The pre-trained RQSD-Net network is used. By comparing the second enhanced image and pseudo reference image Based on the quality scores of each local region, the first enhancement network is guided to generate an enhanced image that surpasses the quality of the pseudo-reference image in the local regions of the image.

[0144] In this embodiment of the application, the construction of the deep perception transcendence loss based on contrastive learning guidance mainly includes three steps, such as... Figure 5 As shown, the first step requires obtaining the second original underwater image. Divide the image into near and far distance patches and generate a near and far distance patch mask. This process is denoted as NDPD; the second step is to use the image patch mask generated above. The second enhanced image Long-distance tiles and pseudo-reference images The near and far distance image patches are extracted; this process is denoted as... The third step is to calculate the contrast loss function between image patches, which is denoted as... Therefore, the expression for the deep perception transcendence loss based on contrastive learning is:

[0145] .

[0146] In this embodiment, the first step is to obtain the image block mask. The process is as follows Figure 6 As shown, the second original underwater image is first... Input into a depth prediction network (DPN) In the process, underwater image depth maps are generated. Then, the underwater image depth map Execute partitioning strategy Finally, image patch masks for near and far distances are obtained. The expression is:

[0147] ;

[0148] in, It uses a pre-trained UDepth network and a partitioning strategy. The following is a detailed description:

[0149] For partitioning strategies First, the underwater depth map with a resolution of 256×256 is... The image is divided into 16 patches in 4 rows and 4 columns, each patch being 64×64 pixels. Then, two thresholds are defined to determine whether a patch belongs to a near-field or far-field patch: the far-field threshold for depth map pixels. The ratio threshold of distant regions in a single image patch Specifically, first calculate the depth value in an image patch that is greater than... Number of pixels The calculation formula is:

[0150] ;

[0151] in, Represents the depth map of underwater images The horizontal and vertical pixel coordinates of 256×256 This represents the 4×4 horizontal and vertical coordinates of the image patch. This indicates the size of an image patch, which is 64×64. Set to 0.65, where 1 in the formula represents the current depth map pixel. For pixels that are far away, 0 indicates a pixel in the current depth map. Pixels that are close together will be the final result. This indicates the current image patch. The number of all distant pixels, then the image block is determined. Whether it belongs to a distant image patch, the result of 16 image patches is used as... The calculation formula is:

[0152] ;

[0153] in, Represents image blocks The total number of pixels is 64 × 64 = 4096. It is a predefined threshold, which is the number of distant pixels that a distant image patch should have. When set to 0.6, the final result of the formula is 0, which indicates that the current image block belongs to a distant image block, and 1, which indicates that the current image block belongs to a close-range image block.

[0154] In this embodiment of the application, the second step is to use the image patch mask generated above. The second enhanced image Long-distance image patches and pseudo-reference images The near and far image patches are extracted, which is what was mentioned earlier. The process, for Figure 5 The upper horizontal portion of the image is enhanced by a second image. and image patch mask After combining, the second enhanced image can be extracted. Long-distance image patches ;for Figure 5 The lower horizontal part of the image is obtained by using a pseudo-reference image. and image patch mask After combining, the near and far image patches of the pseudo-reference image can be extracted. and The specific expression is:

[0155] ;

[0156] in, and They represent Figure 5 The extraction process of the horizontal portions in the upper and lower parts;

[0157] It is important to note here the number of distant image patches. and the number of close-up image patches The values ​​are not always equal, which poses a challenge to the next step of constructing the contrastive loss. To ensure the balance of the contrastive loss calculation, certain decisions need to be made, such as... Figure 7 As shown, the following discussion will cover two cases: and ;

[0158] Scenario 1: When This means that the distant image patches of the second enhanced image Close-range image patches compared to pseudo-reference images There are many, thus missing some close-range pseudo-reference image patches. At this point, a number of image patches with missing values ​​can be randomly selected from close-range image patches generated from the same image or the same training batch;

[0159] Scenario 2: When This means that the distant image patches of the second enhanced image Close-range image patches compared to pseudo-reference images The number of images is small, resulting in redundant near-field image blocks in some pseudo-reference images. These redundant image blocks can be removed.

[0160] In this embodiment of the application, the third step is to calculate the contrast loss between image patches, such as... Figure 5 As shown on the right, image patch , and Using anchor points, negative samples, and positive samples respectively, a deep perception-based transcendence loss guided by contrastive learning is implemented. The expression is:

[0161] ;

[0162] ;

[0163] in, Indicates the index of the image patch. Indicates comparative loss, Represents the distant image patch of the second enhanced image Quantity, , and These represent the anchor point, negative sample, and positive sample, respectively, for the contrast loss. The first one representing the pre-trained VGG-19 network In this embodiment, the 2nd, 3rd, and 5th max pooling layers are selected. Indicates the first The output features of the max pooling layer Indicates the first The contrastive loss weights for each max-pooling layer feature. Set them to 0.3, 0.1, and 0.6 in sequence.

[0164] In this embodiment of the application, experiments were conducted on the method of the present invention to further verify the feasibility and effectiveness of the present invention;

[0165] For the test dataset, four widely recognized and publicly available real-world UIE benchmark datasets were selected: UIEB, EUVP, SQUID, and RUIE. Specifically, five subsets were selected: UIEB-T90, UIEB-C60, EUVP-515, SQUID-T16, and RUIE-T78.

[0166] For evaluation metrics, four non-reference UIQA metrics were selected, including two traditional UIQA metrics and two deep learning-based UIQA metrics. The two traditional metrics are UIQM and UCIQE, and the two deep learning-based metrics are URanker and RUIQA. The higher the score of all these metrics, the better the image quality.

[0167] In terms of comparison methods, this invention was compared with 10 state-of-the-art UIE methods, including two traditional methods and eight deep learning-based methods. The two traditional methods are MLLE and CBLA, and the eight deep learning-based methods are UWCNN, UColor, TACL, NU2Net, CLUIE, Semi-UIR, HCLR, and CCLNet. The comparative experiments were conducted using the source code and pre-trained models provided by the original authors.

[0168] The comparative experiment consisted of three parts: the first part was a qualitative visual comparison experiment, the second part was a quantitative indicator comparison experiment, and the third part was a comparison experiment on model size and running time.

[0169] For the qualitative visual comparison experiment, according to the three typical underwater degradation environments of haze effect, color deviation and low light environment, the image enhancement results of 11 methods including the method of the present application are compared, and the corresponding results are shown in Figure 8 、 Figure 9 、 Figure 10

[0170] For the quantitative index comparison experiment, according to the above-mentioned four evaluation indexes, the quantitative measurement results of 11 methods including the method of the present application are compared, and the specific results are shown in Figure 11 , the top three are marked as red, green and blue respectively;

[0171] For the comparison experiment of model size and running time, according to the parameters (Parameters), floating point operations (FLOPs) and running time (Running Time) as indexes, the results of 13 methods including the method of the present application are compared, and the specific results are shown in Figure 12 , the values of the top three are marked as red, green and blue respectively;

[0172] From the above experimental results, it can be seen that the method of the present application has better image enhancement effect.

[0173] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "in the present embodiment", "specific examples" or "some examples" means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, different embodiments or examples described in the present specification and the features of different embodiments or examples can be combined and combined by those skilled in the art without contradiction.

[0174] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.​

Claims

1. An underwater image enhancement method guided by joint image perception measurement and contrastive learning, characterized in that, Includes the following steps: Step S1: Obtain the first underwater image dataset with pseudo-reference labels, the second underwater image dataset without labels, and the pseudo-reference image dataset provided by the Semi-UIR network; Step S2: Input the first underwater image dataset into the HR-Net network to obtain the first enhanced image dataset, introduce the structural similarity index and combine the first enhanced image dataset and the first underwater image dataset to obtain the first loss function, and pre-train the HR-Net network based on the first loss function to obtain the first enhanced network; Step S3: Input the second underwater image dataset into the first augmentation network to obtain the second augmented image dataset; obtain the second loss function based on the second augmented image dataset and the pseudo reference image dataset; and pre-train the first augmentation network based on the second loss function to obtain the second augmentation network. Step S4: Obtain and input the image to be enhanced into the second enhancement network to obtain the enhanced image.

2. The underwater image enhancement method guided by joint image perception metric and contrastive learning according to claim 1, characterized in that, Step S2 includes: Step S21: The first original underwater images in the first underwater image dataset are sequentially input into the HR-Net network to obtain the corresponding first enhanced images, which constitute the first enhanced image dataset. Step S22: For each pixel in each of the first enhanced images, the corresponding structural similarity index is calculated within a 13x13 image block around the pixel in conjunction with the reference image. Step S23: Obtain the first loss function based on each of the structural similarity indices, configure the learning rate as 10e-4, the number of training rounds as 200, the optimizer as Adam, and pre-train the HR-Net network based on the first loss function to obtain the first augmented network.

3. The underwater image enhancement method guided by joint image perception metric and contrastive learning according to claim 2, characterized in that, In step S22, the structural similarity index is obtained using the following formula: ； in, Indicates the first The structural similarity index corresponding to each pixel; Indicates the first The mean of an image patch centered at a pixel in the reference image; Indicates the first The mean of an image patch centered at a pixel in the first enhanced image; Indicates the first preset parameter; Indicates the first The covariance of an image patch centered at a pixel in the reference image and the first enhanced image; Indicates the first The standard deviation of an image patch centered at a pixel in the reference image; Indicates the first The standard deviation of an image patch centered at a pixel in the first enhanced image; This indicates the second preset parameter.

4. The underwater image enhancement method guided by joint image perception metric and contrastive learning according to claim 2, characterized in that, In step S23, the first loss function is obtained using the following calculation formula: ； in, Represents the first loss function; This represents the total number of pixels in the first enhanced image; Indicates the first The structural similarity index corresponding to each pixel.

5. The underwater image enhancement method guided by joint image perception metric and contrastive learning according to claim 1, characterized in that, Step S3 includes: Step S31: The second original underwater images in the second underwater image dataset are sequentially input into the first enhancement network to obtain the corresponding second enhanced images, which constitute the second enhanced image dataset; Step S32: Input each of the second enhanced images and the pseudo-reference images in the pseudo-reference image dataset into the VGG-19 network to obtain the corresponding output features, and obtain the content preservation loss based on each of the output features; Step S33: Based on the second enhanced image dataset and the pseudo-reference image dataset, construct the quality score transcendence loss based on image perception metric and the depth perception transcendence loss guided by contrastive learning, and obtain the pseudo-reference transcendence loss according to the quality score transcendence loss based on image perception metric and the depth perception transcendence loss guided by contrastive learning. Step S34: Obtain the second loss function based on the content preservation loss and the pseudo-reference transcendence loss, configure the learning rate as 10e-4, the number of training epochs as 200, the optimizer as Adam, and pre-train the first augmentation network based on the second loss function to obtain the second augmentation network.

6. The underwater image enhancement method guided by joint image perception metric and contrastive learning according to claim 5, characterized in that, In step S32, the content retention loss is calculated using the following formula: ； in, This indicates that the content remains lost; This represents the weight corresponding to the output feature; This indicates that the second enhanced image is in the VGG-19 network. The output features corresponding to the layer; This indicates that the pseudo-reference image is in the VGG-19 network at the [missing information]. The output features corresponding to the layer.

7. The underwater image enhancement method guided by joint image perception metric and contrastive learning according to claim 5, characterized in that, Step S33 includes: Step S331: Input the second enhanced image dataset and the pseudo reference image dataset into the URanker network in sequence to obtain the first global quality score and the second global quality score, and obtain the global comparison loss based on the first global quality score and the second global quality score; Step S332: Input the second enhanced image dataset and the pseudo reference image dataset into the RQSD-Net network to obtain local region comparison results. Construct a local comparison loss based on the local region comparison results. Obtain the quality score transcendence loss based on the image perception metric based on the global comparison loss and the local comparison loss. Step S333: Generate an image patch mask in the second original underwater image to divide the near-field image patch and the far-field image patch. Extract the far-field image patch in the second enhanced image and the near-field image patch and the far-field image patch in the pseudo-reference image through the image patch mask. Then calculate the contrast loss function between each far-field image patch and each near-field image patch to obtain the depth perception transcendence loss based on contrast learning guidance. Step S334: Obtain the pseudo-reference transcendence loss based on the quality score transcendence loss based on image perception metric and the depth perception transcendence loss guided by contrastive learning.

8. The underwater image enhancement method guided by joint image perception metric and contrastive learning according to claim 7, characterized in that, In step S331, the global comparison loss is obtained using the following formula: ； in, This represents the global comparison loss; This represents the sigmoid activation function; This represents the first global quality score; This represents the second global quality score; This refers to the URanker network; This represents the second enhanced image; This refers to the pseudo-reference image.

9. The underwater image enhancement method guided by joint image perception metric and contrastive learning according to claim 7, characterized in that, In step S332, the local comparison loss is obtained using the following formula: ； in, This represents the local comparison loss; This indicates the row number of the comparison results for the local region; This indicates the number of columns representing the comparison results of the local region; This indicates the comparison result of the local region; Represents the horizontal coordinates of a pixel; Represents the vertical coordinates of a pixel; This refers to the RQSD-Net network; This represents the second enhanced image; This refers to the pseudo-reference image.

10. The underwater image enhancement method guided by joint image perception metric and contrastive learning according to claim 7, characterized in that, Step S333 includes: Step S3331: Input the second original underwater image into the depth prediction network to generate an underwater image depth map, and perform a partitioning strategy on the underwater image depth map to obtain the image block mask; Step S3332: Combine the second enhanced image and the image block mask to extract the far-range image block in the second enhanced image, and combine the pseudo-reference image and the image block mask to extract the near-range image block and the far-range image block of the pseudo-reference image; Step S3333: Using the far-distance image patch of the second enhanced image as the anchor point, the far-distance image patch of the pseudo-reference image as the negative sample, and the near-distance image patch of the pseudo-reference image as the positive sample, the depth perception transcendence loss guided by contrastive learning is obtained.

Images