Underwater polarization image enhancement method
By designing a domain adaptive module based on cyclic consistency generation adversarial network and an image enhancement module with semi-supervised learning strategy, the problem of poor stability and accuracy caused by domain differences in the open water is solved, and a more efficient image enhancement effect is achieved.
Patent Information
- Application Number
- CN202510306501.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-14
- Publication Date
- 2025-07-18
AI Technical Summary
In open waters, domain differences exist between databases of underwater polarized images, resulting in poor image enhancement stability, poor accuracy and poor generalization.
A domain adaptive module based on cyclic consistency generation adversarial network is designed, and an image enhancement module combined with a semi-supervised learning strategy is designed to use consistency constraints to eliminate domain differences and improve network performance.
It improves the stability and accuracy of the enhanced polarization image of underwater, enhances the generalization ability of the method, and reduces the influence of refraction, absorption and scattering of underwater imaging.
Smart Images
Figure CN120339097A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular, to an underwater polarization image enhancement method. Background Art
[0002] Underwater optical images play an important role in the exploration of marine resources. However, due to the refraction, absorption, and scattering of light by suspended particles in water, problems such as low contrast, blurred details, and color distortion will occur in the original underwater images; polarization imaging can effectively filter out scattered light and has unique advantages compared with traditional optical imaging underwater; with the development of deep learning, methods based on supervised learning have achieved good results in underwater image enhancement. However, in open waters, it is difficult to obtain clear labels for underwater polarization images, and due to different acquisition devices and waters, there are domain differences between each database, resulting in poor generalization ability of traditional methods.
[0003] Aiming at the problems existing in traditional methods, an underwater polarization image enhancement method is proposed; firstly, by designing a domain adaptation module based on a cyclic-consistent generative adversarial network to eliminate the domain difference between the underwater non-polarization database and the underwater polarization database; then, an image enhancement module using a semi-supervised learning strategy is proposed, and the physical prior of underwater images is added for constraint to solve the training problem of unlabeled images; finally, consistency constraints are used to ensure the stability of training, further improve the network performance, and underwater polarization image data in the coastal waters are collected for image enhancement verification. Summary of the Invention
[0004] The embodiments of the present invention provide an underwater polarization image enhancement method to at least solve the technical problems in the prior art that there are domain differences in different databases in open waters, resulting in poor stability, accuracy, and generalization in image enhancement of underwater polarization images.
[0005] According to one aspect of an embodiment of the present invention, an underwater polarization image enhancement method is provided. The method may include: obtaining a training data set, where the training data set includes a first data set and a second data set, the first data set is a labeled non-polarized underwater image data set, the first data set includes a first initial data set and a first label data set, and the second data set is an unlabeled polarized underwater image data set; constructing an underwater polarization image enhancement network, where the underwater polarization image enhancement network includes: a domain adaptation module and an image enhancement module, the domain adaptation module includes a first conversion network and a second conversion network, a first discriminator and a second discriminator, and the image enhancement module includes a first enhancement network and a second enhancement network; jointly inputting the first initial data set and the second data set into the domain adaptation module for training, and obtaining a successfully trained domain adaptation module for the first time, a fake image of the first initial data set, and a fake image of the second data set by minimizing the loss function of the domain adaptation module during the training process, where the loss function of the domain adaptation module includes: a target adversarial loss function, a target cycle consistency loss, and a target identity mapping loss; jointly inputting the fake image of the first initial data set and the second data set into the second enhancement network for training, and at the same time jointly inputting the fake image of the second data set and the first initial data set into the first enhancement network for training, and obtaining a successfully trained image enhancement module for the first time, a clear image corresponding to the fake image of the first initial data set, a clear image corresponding to the fake image of the second data set, an enhanced image of the second data set, and an enhanced image of the first initial data set by minimizing the loss function of the image enhancement module during the training process, where the loss function of the image enhancement module is a target image enhancement loss; re-inputting the first initial data set and the second data set into the successfully trained domain adaptation module for the first time and the successfully trained image enhancement module for the first time for retraining, and obtaining a fine-tuned underwater polarization image enhancement network, where the fine-tuned underwater polarization image enhancement network includes: a second successfully trained domain adaptation module and a second successfully trained image enhancement module, and the loss function of the fine-tuned underwater polarization image enhancement network is a joint loss; where the joint loss includes: the loss function of the domain adaptation module, the target image enhancement loss, and a target consistency loss; obtaining a test data set, where the test data set is an unlabeled polarized underwater image data set; inputting the test data set into the second successfully trained image enhancement module, and obtaining a clear image corresponding to the test data set.
[0006] Optionally, the step of jointly inputting the first initial data set and the second data set into the domain adaptation module for training, and obtaining a successfully trained domain adaptation module, fake graphs of the first initial data set, and fake graphs of the second data set by minimizing the loss function of the domain adaptation module during the training process includes: inputting the first initial data set into the first conversion network for training to obtain fake graphs of the first initial data set, and during the training process, processing the fake graphs of the first initial data set and the second data set through the first discriminator to obtain a first adversarial loss function; inputting the second data set into the second conversion network for training to obtain fake graphs of the second data set, and during the training process, processing the fake graphs of the second data set and the first initial data set through the second discriminator to obtain a second adversarial loss function; during the training processes of the first conversion network and the second conversion network, obtaining a first cycle loss based on the first initial data set, the first conversion network, and the second conversion network; obtaining a second cycle loss based on the second data set, the second conversion network, and the first conversion network; obtaining a first identity mapping loss based on the first initial data set and the second conversion network; obtaining a second identity mapping loss based on the second data set and the first conversion network; obtaining a target adversarial loss function based on the first adversarial loss function and the second adversarial loss function; obtaining a target cycle consistency loss based on the first cycle loss and the second cycle loss; obtaining a target identity mapping loss based on the first identity mapping loss and the second identity mapping loss; and obtaining the loss function of the domain adaptation module based on the target adversarial loss function, the target cycle consistency loss, and the target identity mapping loss.
[0007] Optionally, the step of obtaining a target adversarial loss function based on the first adversarial loss function and the second adversarial loss function includes: determining the sum of the first adversarial loss function and the second adversarial loss function as the target adversarial loss function.
[0008] Optionally, the step of obtaining a target cycle consistency loss based on the first cycle loss and the second cycle loss includes: determining the sum of the first cycle loss and the second cycle loss as the target cycle consistency loss.
[0009] Optionally, the step of obtaining a target identity mapping loss based on the first identity mapping loss and the second identity mapping loss includes: determining the sum of the first identity mapping loss and the second identity mapping loss as the target identity mapping loss.
[0010] Optionally, the false graphs of the first initial data set and the second data set are jointly input into the second enhancement network for training, and at the same time, the false graphs of the second data set and the first initial data set are jointly input into the first enhancement network for training. During the training process, by minimizing the loss function of the image enhancement module, the image enhancement module that succeeds in the first training, the clear graphs corresponding to the false graphs of the first initial data set, the clear graphs corresponding to the false graphs of the second data set, the enhanced graphs of the second data set, and the enhanced graphs of the first initial data set are obtained, including: jointly inputting the false graphs of the second data set and the first initial data set into the first enhancement network for training to obtain the clear graphs corresponding to the false graphs of the second data set and the enhanced graphs of the first initial data set. During the training process, based on the enhanced graphs of the first initial data set and the first label data set, the first mean square error loss is obtained, and based on the clear graphs corresponding to the false graphs of the second data set, the horizontal gradient operator, and the vertical gradient operator, the first total variation loss is obtained; jointly inputting the false graphs of the first initial data set and the second data set into the second enhancement network for training to obtain the clear graphs corresponding to the false graphs of the first initial data set and the enhanced graphs of the second data set. During the training process, based on the clear graphs corresponding to the false graphs of the first initial data set and the first label data set, the second mean square error loss is obtained, and based on the enhanced graphs of the second data set, the horizontal gradient operator, and the vertical gradient operator, the second total variation loss is obtained; based on the first mean square error loss, the second mean square error loss, the first total variation loss, and the second total variation loss, the target image enhancement loss is obtained.
[0011] Optionally, the obtaining of the target image enhancement loss based on the first mean square error loss, the second mean square error loss, the first total variation loss, and the second total variation loss includes: determining the product of the sum of the first mean square error loss and the second mean square error loss and the first proportionality coefficient as the target mean square error loss; determining the product of the sum of the first total variation loss and the second total variation loss and the second proportionality coefficient as the target total variation loss; determining the sum of the target mean square error loss and the target total variation loss as the target image enhancement loss.
[0012] Optionally, the determination process of the target consistency loss is as follows: during the second training process of the domain adaptation module and the image enhancement module, based on the clear graphs corresponding to the false graphs of the first initial data set, the enhanced graphs of the first initial data set, the clear graphs corresponding to the false graphs of the second data set, and the enhanced graphs of the second data set, the initial consistency loss is obtained; during the second training process of the domain adaptation module and the image enhancement module, based on the features of the first initial data set extracted by the first enhancement network and the features of the false graphs of the first initial data set, the features of the second data set extracted by the second enhancement network and the features of the false graphs of the second data set, the target feature consistency constraint loss is obtained; based on the initial consistency loss and the target feature consistency constraint loss, the target consistency loss is obtained.
[0013] Optionally, the expression of the combined loss is as follows:
[0014] where is the combined loss, is the loss function of the domain adaptation module, is the target image enhancement loss, is the target consistency loss.
[0015] Advantages of the present invention: The present invention proposes an underwater polarization image enhancement method. First, by designing a domain adaptation module based on a cycle-consistent generative adversarial network, the domain difference between the underwater non-polarization database and the underwater polarization database is eliminated. Then, an image enhancement module adopting a semi-supervised learning strategy is proposed, and the physical prior of underwater images is added for constraint to solve the training problem of unlabeled images. Finally, consistency constraints are used to ensure the stability of training and further improve the network performance. This method is not affected by the refraction, absorption, and scattering effects of underwater imaging, and at the same time improves the generalization ability of the underwater image enhancement method. Description of the Drawings
[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings: Figure 1 is a flowchart of an underwater polarization image enhancement method according to an embodiment of the present invention; Figure 2 is a network structure diagram of an underwater polarization image enhancement method according to an embodiment of the present invention; Figure 3 is a schematic structural diagram of a first enhancement network and a second enhancement network according to an embodiment of the present invention; Figure 4 is an enhanced result diagram of a test data set according to an embodiment of the present invention. Detailed Embodiments
[0017] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
[0018] It should be noted that the terms "first", "second", etc. in the description, claims and the above-mentioned drawings of the present invention are used to distinguish similar objects and are used to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.
[0019] Embodiment 1 According to an embodiment of the present invention, a method for enhancing underwater polarization images is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system including at least one set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.
[0020] Figure 1 is a flowchart of a method for enhancing underwater polarization images according to an embodiment of the present invention, as Figure 1 shown, the method may include the following steps: Step S101, obtain a training data set, where the training data set includes a first data set and a second data set. The first data set is a labeled non-polarized underwater image data set, and the first data set includes a first initial data set and a first label data set. The second data set is an unlabeled polarized underwater image data set.
[0021] In the technical solution provided in step S101 of the present invention, Figure 2 is a network structure diagram of a method for enhancing underwater polarization images according to an embodiment of the present invention, as Figure 1 shown, the first data set includes: the first initial data set and the first label data set (that is, the true label corresponding to the first initial data set, also called the source domain). Among them, the first initial data set is a labeled non-polarized image data set, and the second data set is an unlabeled polarized underwater image data set (also called the target domain). Among them, is the number of images in the data set of domain A, is the number of images in the data set of domain B.
[0022] Step S102: Construct an underwater polarization image enhancement network, which includes a domain adaptation module and an image enhancement module. The domain adaptation module includes a first conversion network, a second conversion network, a first discriminator, and a second discriminator. The image enhancement module includes a first enhancement network and a second enhancement network.
[0023] In the technical solution provided in step S102 of the present invention, as Figure 2 shown, the domain adaptation module includes a first conversion network ( ), a second conversion network ( ), a first discriminator ( ), a second discriminator ( ). The image enhancement module consists of two parts, the underwater image enhancement network in domain A and the underwater image enhancement network in domain B . And the network adopts a U-Net structure. And have the same structure. Figure 3 is the structural schematic diagram of the first enhancement network and the second enhancement network according to the embodiment of the present invention. As Figure 3 shown, the first enhancement network and the second enhancement network are composed of an encoder part consisting of 4 Encoders and a decoder part consisting of 4 Decoders.
[0024] Step S103: Input the first initial dataset and the second dataset into the domain adaptation module for training. During the training process, by minimizing the loss function of the domain adaptation module, a successfully trained domain adaptation module for the first time, a fake map of the first initial dataset, and a fake map of the second dataset are obtained. Among them, the loss function of the domain adaptation module includes: a target adversarial loss function, a target cycle consistency loss, and a target identity mapping loss.
[0025] In the technical solution provided in step S103 of the present invention, input the first initial dataset into the first conversion network of the domain adaptation module for training. At the same time, input the second dataset into the second conversion network of the domain adaptation module for training, and obtain a successfully trained domain adaptation module for the first time, a fake map of the first initial dataset and a fake map of the second dataset; the purpose of the loss function of the domain adaptation module is to learn a bidirectional domain transformation network and , so that it can eliminate the differences between the source domain and the target domain .
[0026] Step S104: Input the fake images of the first initial dataset and the second dataset into the second enhancement network for training. At the same time, input the fake images of the second dataset and the first initial dataset into the first enhancement network for training. During the training process, by minimizing the loss function of the image enhancement module, obtain the image enhancement module that is successfully trained for the first time, the clear images corresponding to the fake images of the first initial dataset, the clear images corresponding to the fake images of the second dataset, the enhanced images of the second dataset, and the enhanced images of the first initial dataset. Among them, the loss function of the image enhancement module is the target image enhancement loss.
[0027] In the technical solution provided in step S104 of the present invention, as Figure 2 shown, input the fake images of the first initial dataset and the second dataset into the second enhancement network ( ), and at the same time, input the fake images of the second dataset and the first initial dataset into the first enhancement network ( ), and during the training process, by minimizing the loss function of the image enhancement module, obtain the image enhancement module that is successfully trained for the first time, the clear images corresponding to the fake images of the first initial dataset and the clear images corresponding to the fake images of the second dataset , the enhanced images of the second dataset and the enhanced images of the first initial dataset .
[0028] Step S105: Input the first initial dataset and the second dataset into the domain adaptation module that is successfully trained for the first time and the image enhancement module that is successfully trained for the first time for retraining, and obtain the fine-tuned underwater polarization image enhancement network. Among them, the fine-tuned underwater polarization image enhancement network includes: the domain adaptation module that is successfully trained for the second time and the image enhancement module that is successfully trained for the second time. The loss function of the fine-tuned underwater polarization image enhancement network is the joint loss; among them, the joint loss includes: the loss function of the domain adaptation module, the target image enhancement loss, and the target consistency loss.
[0029] In the technical solution provided in step S105 of the present invention, input the first initial dataset and the second dataset into the domain adaptation module that is successfully trained for the first time and the image enhancement module that is successfully trained for the first time for retraining, and obtain the fine-tuned underwater polarization image enhancement network.
[0030] Step S106: Obtain a test dataset. Among them, the test dataset is an unlabeled polarized underwater image dataset.
[0031] In the technical solution provided in step S106 of the present invention, obtain a test dataset.
[0032] Step S107: Input the test data set into the image enhancement module that has been successfully trained for the second time to obtain the clear images corresponding to the test data set.
[0033] In the technical solution provided in step S107 of the present invention, the test data set is input into the image enhancement module ( ) that has been successfully trained for the second time to obtain the clear images corresponding to the test data set.
[0034] The above method of this embodiment will be further introduced below.
[0035] As an optional embodiment, in step S103, the first initial data set and the second data set are jointly input into the domain adaptation module for training. During the training process, by minimizing the loss function of the domain adaptation module, the domain adaptation module that has been successfully trained for the first time, the fake images of the first initial data set, and the fake images of the second data set are obtained, including: inputting the first initial data set into the first conversion network for training to obtain the fake images of the first initial data set. During the training process, the first discriminator processes the fake images of the first initial data set and the second data set to obtain the first adversarial loss function; inputting the second data set into the second conversion network for training to obtain the fake images of the second data set. During the training process, the second discriminator processes the fake images of the second data set and the first initial data set to obtain the second adversarial loss function; during the training processes of the first conversion network and the second conversion network, based on the first initial data set, the first conversion network, and the second conversion network, the first cycle loss is obtained; based on the second data set, the second conversion network, and the first conversion network, the second cycle loss is obtained; based on the first initial data set and the second conversion network, the first identity mapping loss is obtained; based on the second data set and the first conversion network, the second identity mapping loss is obtained; based on the first adversarial loss function and the second adversarial loss function, the target adversarial loss function is obtained; based on the first cycle loss and the second cycle loss, the target cycle consistency loss is obtained; based on the first identity mapping loss and the second identity mapping loss, the target identity mapping loss is obtained; based on the target adversarial loss function, the target cycle consistency loss, and the target identity mapping loss, the loss function of the domain adaptation module is obtained.
[0036] In this embodiment, as Figure 2 shown, taking as an example, the goal is to make and indistinguishable. Therefore, the discriminator is used to train and according to the minmax optimization method, and processes and Perform constraints, and the adversarial loss function is as follows:
[0037] Among them, is the discriminator network used to determine whether the data belongs to the true distribution of domain B, , respectively represent the expectations randomly sampled from the true data distributions of domains A and B, is the first adversarial loss function.
[0038] For , the false graphs of the second dataset and the first initial dataset are processed by the second discriminator to obtain the second adversarial loss function:
[0039] Among them, is the second adversarial loss function, is the discriminator network used to determine whether the data belongs to the true distribution of domain A.
[0040] In addition, to ensure the stability of training, a cycle consistency loss is adopted to make the images in domain A or domain B output basically the same as the original after two domain transformations, that is: And ; the expression of the target cycle consistency loss is:
[0041] Among them, is the first cycle loss, is the second cycle loss, is the mean absolute error.
[0042] To ensure that the scene content other than the style is not damaged, an identity mapping loss is added to make the images in domain A or domain B consistent with the original after being transformed from other domains to this domain, that is: Figure 1 That is: And ; the expression of the target identity mapping loss is:
[0043] Among them, is the first identity mapping loss, is the second identity mapping loss.
[0044] Calculate the target adversarial loss function, the target cycle consistency loss, and the target identity mapping loss to obtain the loss function of the domain adaptation module.
[0045] As an alternative embodiment, obtaining the target adversarial loss function based on the first adversarial loss function and the second adversarial loss function includes: determining the sum of the first adversarial loss function and the second adversarial loss function as the target adversarial loss function.
[0046] In this embodiment, the first adversarial loss function and the second adversarial loss function are added together to obtain the target adversarial loss function.
[0047] As an alternative embodiment, obtaining the target cycle consistency loss based on the first cycle loss and the second cycle loss includes: determining the sum of the first cycle loss and the second cycle loss as the target cycle consistency loss.
[0048] In this embodiment, the first cycle loss and the second cycle loss are added together to obtain the target cycle consistency loss.
[0049] As an alternative embodiment, obtaining the target identity mapping loss based on the first identity mapping loss and the second identity mapping loss includes: determining the sum of the first identity mapping loss and the second identity mapping loss as the target identity mapping loss.
[0050] In this embodiment, the first identity mapping loss and the second identity mapping loss are added together to obtain the target identity mapping loss.
[0051] The expression of the loss function of the domain adaptation module is:
[0052] where is the third proportionality coefficient, is the fourth proportionality coefficient.
[0053] As an alternative embodiment, in step S104, the false graphs of the first initial data set and the second data set are jointly input into the second enhancement network for training, and at the same time, the false graphs of the second data set and the first initial data set are jointly input into the first enhancement network for training. During the training process, by minimizing the loss function of the image enhancement module, an image enhancement module that is successfully trained for the first time, clear graphs corresponding to the false graphs of the first initial data set and the second data set, enhanced graphs of the second data set, and enhanced graphs of the first initial data set are obtained, including: jointly inputting the false graphs of the second data set and the first initial data set into the first enhancement network for training to obtain clear graphs corresponding to the false graphs of the second data set and enhanced graphs of the first initial data set. During the training process, based on the enhanced graphs of the first initial data set and the first label data set, a first mean square error loss is obtained, and based on the clear graphs corresponding to the false graphs of the second data set, the horizontal gradient operator, and the vertical gradient operator, a first total variation loss is obtained; jointly inputting the false graphs of the first initial data set and the second data set into the second enhancement network for training to obtain clear graphs corresponding to the false graphs of the first initial data set and enhanced graphs of the second data set. During the training process, based on the clear graphs corresponding to the false graphs of the first initial data set and the first label data set, a second mean square error loss is obtained, and based on the enhanced graphs of the second data set, the horizontal gradient operator, and the vertical gradient operator, a second total variation loss is obtained; based on the first mean square error loss, the second mean square error loss, the first total variation loss, and the second total variation loss, a target image enhancement loss is obtained.
[0054] In this embodiment, for the image enhancement loss: using the domain transformation module, the source domain image is transformed to the target domain through the "style transfer" method, and vice versa, thus creating a new data set and ; therefore, the image enhancement network can be trained in two domains in a semi-supervised manner and ; taking domain B as an example, using the labeled data set to perform supervised training on , the enhanced graphs of the first initial data set ( ) are close to the first label data set ( ), and the expression of the first mean square error loss is:
[0055] where is the first mean square error loss, is the mean square error.
[0056] In the unsupervised branch, the total variation loss constrains the clear graphs corresponding to the false graphs of the second data set , and the prior formula is as follows: The total variation loss is the predicted image on -regularized gradient prior, the expression of the first total variation loss is:
[0057] wherein, is the first total variation loss, is the horizontal gradient operator, is the vertical gradient operator.
[0058] In summary, for domain B, the image enhancement loss is composed of the above two parts; similarly, in domain A, there are also the second mean square error loss and the second total variation loss as follows: The expression of the second mean square error loss is:
[0059] wherein, is the second mean square error loss.
[0060] The expression of the second total variation loss is:
[0061] wherein, is the second total variation loss.
[0062] Therefore, the image enhancement loss of the image sharpening module part is as follows:
[0063] wherein, is the first proportionality coefficient, is the second proportionality coefficient.
[0064] As an alternative embodiment, based on the first mean square error loss, the second mean square error loss, the first total variation loss and the second total variation loss, the target image enhancement loss is obtained, including: multiplying the sum of the first mean square error loss and the second mean square error loss by the first proportionality coefficient to determine the target mean square error loss; multiplying the sum of the first total variation loss and the second total variation loss by the second proportionality coefficient to determine the target total variation loss; adding the target mean square error loss and the target total variation loss to determine the target image enhancement loss.
[0065] In this embodiment, multiplying the sum of the first mean square error loss and the second mean square error loss by the first proportionality coefficient to obtain the target mean square error loss; multiplying the sum of the first total variation loss and the second total variation loss by the second proportionality coefficient to obtain the target total variation loss; adding the target mean square error loss and the target total variation loss to obtain the target image enhancement loss.
[0066] As an alternative embodiment, in step S105, the process of determining the target consistency loss is as follows: During the second training of the domain adaptation module and the image enhancement module, based on the clear images corresponding to the fake images in the first initial dataset, the enhanced images of the first initial dataset, the clear images corresponding to the fake images in the second dataset, and the enhanced images of the second dataset, an initial consistency loss is obtained; During the second training of the domain adaptation module and the image enhancement module, based on the features of the first initial dataset extracted by the first enhancement network and the features of the fake images in the first initial dataset, the features of the second dataset extracted by the second enhancement network and the features of the fake images in the second dataset, a target feature consistency constraint loss is obtained; Based on the initial consistency loss and the target feature consistency constraint loss, the target consistency loss is obtained.
[0067] In this embodiment, for the initial consistency loss: For two images in different domains of the same scene, the predicted clear images should be consistent, that is, satisfy and , Therefore, the initial consistency loss is added, and the expression of the initial consistency loss is:
[0068] where is the initial consistency loss.
[0069] In addition, the method also uses the adversarial loss to increase the feature consistency constraint and improve the stability of the domain conversion network and the image enhancement network; The expression of the target feature consistency constraint loss is:
[0070] where is the target feature consistency constraint loss, and are the feature extraction parts of the image enhancement networks in domains A and B respectively, and are the discriminators that make the features in domains A and B consistent respectively.
[0071] The expression of the target consistency loss is:
[0072] where is the target consistency loss, is the fifth proportionality coefficient.
[0073] As an alternative embodiment, the expression of the joint loss is:
[0074] where is the joint loss, is the loss function of the domain adaptation module, is the target image enhancement loss, is the target consistency loss.
[0075] In this embodiment, the sum of the loss function of the domain adaptation module, the target image enhancement loss, and the target consistency loss is determined as the joint loss.
[0076] Experimental part: Training details: The training set data of domains A and B are used for training. The pictures are downsampled to 256×256 and then randomly cropped to 224×224, and the data is augmented by random flipping. After normalizing it to [-1, 1], it is input into the network. The training of the network is divided into 3 steps, specifically as follows: Step 1: Train the domain adaptation module alone. After 100 iterations, obtain and , and the loss function of this step is ; Step 2: Use the and obtained in Step 1 to train the image enhancement networks of the two domains respectively. After 100 iterations, obtain and . During this process, and are not trained, and the loss function of this step is ; Step 3: Use the , , and obtained in the previous two steps to jointly train the entire framework. After 300 iterations, obtain the final various network models, and the loss function of this process is ; The Adam optimizer is used for training, and the settings of each parameter and learning rate are: , , ; The settings of each proportional coefficient are , , , , ; The training is completed using Pytorch on an NVIDIA RTX2080Ti GPU, and the batch size is set to 2.
[0077] After the test set data finishes training the domain adaptation module and the image enhancement module, a fine-tuned underwater polarization image enhancement network is obtained. Among them, the fine-tuned underwater polarization image enhancement network includes: the domain adaptation module that is successfully trained for the second time and the image enhancement module that is successfully trained for the second time.
[0078] Input the test data set into the image enhancement module that was successfully trained for the second time to obtain the clear image corresponding to the test data set; Figure 4 It is the enhanced result graph of the test data set according to the embodiment of the present invention. As Figure 4 shown, input the input image into the image enhancement module that was successfully trained for the second time to obtain the enhanced image corresponding to the input image.
[0079] In the embodiments of the present invention, by obtaining a training data set, where the training data set includes a first data set and a second data set, the first data set is a labeled non-polarized underwater image data set, the first data set includes a first initial data set and a first label data set, and the second data set is an unlabeled polarized underwater image data set; constructing an underwater polarization image enhancement network, the underwater polarization image enhancement network includes: a domain adaptation module and an image enhancement module, the domain adaptation module includes a first conversion network and a second conversion network, a first discriminator and a second discriminator, and the image enhancement module includes a first enhancement network and a second enhancement network; inputting the first initial data set and the second data set into the domain adaptation module for training together, and during the training process, by minimizing the loss function of the domain adaptation module, obtaining a successfully trained domain adaptation module for the first time, a fake image of the first initial data set, and a fake image of the second data set, where the loss function of the domain adaptation module includes: a target adversarial loss function, a target cycle consistency loss, and a target identity mapping loss; inputting the fake image of the first initial data set and the second data set into the second enhancement network for training together, and at the same time inputting the fake image of the second data set and the first initial data set into the first enhancement network for training, and during the training process, by minimizing the loss function of the image enhancement module, obtaining a successfully trained image enhancement module for the first time, a clear image corresponding to the fake image of the first initial data set, a clear image corresponding to the fake image of the second data set, an enhanced image of the second data set, and an enhanced image of the first initial data set, where the loss function of the image enhancement module is a target image enhancement loss; inputting the first initial data set and the second data set into the successfully trained domain adaptation module for the first time and the successfully trained image enhancement module for the first time for retraining, obtaining a fine-tuned underwater polarization image enhancement network, where the fine-tuned underwater polarization image enhancement network includes: a successfully trained domain adaptation module for the second time and a successfully trained image enhancement module for the second time, and the loss function of the fine-tuned underwater polarization image enhancement network is a joint loss; where the joint loss includes: the loss function of the domain adaptation module, the target image enhancement loss, and a target consistency loss; obtaining a test data set, where the test data set is an unlabeled polarized underwater image data set; inputting the test data set into the successfully trained image enhancement module for the second time, obtaining a clear image corresponding to the test data set, solving the technical problem in the prior art that there are domain differences in different databases in open waters, resulting in poor stability, accuracy, and generalization in image enhancement of underwater polarization images, and achieving the technical effect of improving the stability, accuracy, and generalization of image enhancement of underwater polarization images by proposing an underwater polarization image enhancement method.
[0080] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0081] In the above embodiments of the present invention, the descriptions of the various embodiments each have their own emphasis. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.
[0082] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of units or modules can be in an electrical or other form.
[0083] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0084] In addition, the functional units in the various embodiments of the present invention can be integrated into a first processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.
[0085] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention.
Claims
1. An underwater polarization image enhancement method, characterized in that, Including: Obtain a training data set, where the training data set includes a first data set and a second data set. The first data set is a labeled non-polarized underwater image data set, which includes a first initial data set and a first label data set. The second data set is an unlabeled polarized underwater image data set; Construct an underwater polarization image enhancement network, which includes: a domain adaptation module and an image enhancement module. The domain adaptation module includes a first conversion network and a second conversion network, a first discriminator and a second discriminator. The image enhancement module includes a first enhancement network and a second enhancement network; Input the first initial data set and the second data set into the domain adaptation module for training. During the training process, by minimizing the loss function of the domain adaptation module, obtain a domain adaptation module that is successfully trained for the first time, a fake image of the first initial data set, and a fake image of the second data set. Among them, the loss function of the domain adaptation module includes: an objective adversarial loss function, an objective cycle consistency loss, and an objective identity mapping loss; Input the fake image of the first initial data set and the second data set into the second enhancement network for training. At the same time, input the fake image of the second data set and the first initial data set into the first enhancement network for training. During the training process, by minimizing the loss function of the image enhancement module, obtain an image enhancement module that is successfully trained for the first time, a clear image corresponding to the fake image of the first initial data set, a clear image corresponding to the fake image of the second data set, an enhanced image of the second data set, and an enhanced image of the first initial data set. Among them, the loss function of the image enhancement module is the objective image enhancement loss; Input the first initial data set and the second data set into the domain adaptation module that is successfully trained for the first time and the image enhancement module that is successfully trained for the first time for retraining, and obtain a fine-tuned underwater polarization image enhancement network. Among them, the fine-tuned underwater polarization image enhancement network includes: a domain adaptation module that is successfully trained for the second time and an image enhancement module that is successfully trained for the second time. The loss function of the fine-tuned underwater polarization image enhancement network is a joint loss; among them, the joint loss includes: the loss function of the domain adaptation module, the objective image enhancement loss, and the objective consistency loss; Obtain a test data set, where the test data set is an unlabeled polarized underwater image data set; Input the test data set into the image enhancement module that is successfully trained for the second time to obtain a clear image corresponding to the test data set.
2. The method according to claim 1, wherein The step of inputting the first initial data set and the second data set into the domain adaptation module for training, and obtaining a domain adaptation module that is successfully trained for the first time, a fake image of the first initial data set, and a fake image of the second data set by minimizing the loss function of the domain adaptation module during the training process includes: Input the first initial data set into the first conversion network for training to obtain a fake image of the first initial data set. During the training process, process the fake image of the first initial data set and the second data set through the first discriminator to obtain a first adversarial loss function; Input the second data set into the second conversion network for training to obtain the fake images of the second data set. During the training process, the second discriminator processes the fake images of the second data set and the first initial data set to obtain the second adversarial loss function; During the training process of the first conversion network and the second conversion network, based on the first initial data set, the first conversion network, and the second conversion network, obtain the first cycle loss; based on the second data set, the second conversion network, and the first conversion network, obtain the second cycle loss; based on the first initial data set and the second conversion network, obtain the first identity mapping loss; based on the second data set and the first conversion network, obtain the second identity mapping loss; Based on the first adversarial loss function and the second adversarial loss function, obtain the target adversarial loss function; Based on the first cycle loss and the second cycle loss, obtain the target cycle consistency loss; Based on the first identity mapping loss and the second identity mapping loss, obtain the target identity mapping loss; Based on the target adversarial loss function, the target cycle consistency loss, and the target identity mapping loss, obtain the loss function of the domain adaptation module.
3. The method according to claim 2, wherein The obtaining the target adversarial loss function based on the first adversarial loss function and the second adversarial loss function includes: Determine the sum of the first adversarial loss function and the second adversarial loss function as the target adversarial loss function.
4. The method according to claim 2, wherein The obtaining the target cycle consistency loss based on the first cycle loss and the second cycle loss includes: Determine the sum of the first cycle loss and the second cycle loss as the target cycle consistency loss.
5. The method according to claim 2, wherein The obtaining the target identity mapping loss based on the first identity mapping loss and the second identity mapping loss includes: Determine the sum of the first identity mapping loss and the second identity mapping loss as the target identity mapping loss.
6. The method according to claim 2, wherein The jointly inputting the fake images of the first initial data set and the second data set into the second enhancement network for training, and at the same time jointly inputting the fake images of the second data set and the first initial data set into the first enhancement network for training. During the training process, by minimizing the loss function of the image enhancement module, obtain the successfully trained image enhancement module for the first time, the clear images corresponding to the fake images of the first initial data set and the clear images corresponding to the fake images of the second data set, the enhanced images of the second data set, and the enhanced images of the first initial data set, including: Jointly input the fake images of the second data set and the first initial data set into the first enhancement network for training to obtain the clear images corresponding to the fake images of the second data set and the enhanced images of the first initial data set. During the training process, based on the enhanced images of the first initial data set and the first label data set, obtain the first mean square error loss, and based on the clear images corresponding to the fake images of the second data set, the horizontal gradient operator, and the vertical gradient operator, obtain the first total variation loss; The fake images of the first initial dataset and the second dataset are jointly input into the second enhancement network for training to obtain the clear images corresponding to the fake images of the first initial dataset and the enhanced images of the second dataset. During the training process, based on the clear images corresponding to the fake images of the first initial dataset and the first label dataset, a second mean square error loss is obtained. Based on the enhanced images of the second dataset, the horizontal gradient operator, and the vertical gradient operator, a second total variation loss is obtained; Based on the first mean square error loss, the second mean square error loss, the first total variation loss, and the second total variation loss, an objective image enhancement loss is obtained.
7. The method according to claim 6, wherein The obtaining of the objective image enhancement loss based on the first mean square error loss, the second mean square error loss, the first total variation loss, and the second total variation loss includes: Determining the product of the sum of the first mean square error loss and the second mean square error loss and the first proportionality coefficient as the objective mean square error loss; Determining the product of the sum of the first total variation loss and the second total variation loss and the second proportionality coefficient as the objective total variation loss; Determining the sum of the objective mean square error loss and the objective total variation loss as the objective image enhancement loss.
8. The method according to claim 1, wherein The determination process of the objective consistency loss is as follows: During the second training process of the domain adaptation module and the image enhancement module, based on the clear images corresponding to the fake images of the first initial dataset, the enhanced images of the first initial dataset, the clear images corresponding to the fake images of the second dataset, and the enhanced images of the second dataset, an initial consistency loss is obtained; During the second training process of the domain adaptation module and the image enhancement module, based on the features of the first initial dataset extracted by the first enhancement network and the features of the fake images of the first initial dataset, the features of the second dataset extracted by the second enhancement network and the features of the fake images of the second dataset, an objective feature consistency constraint loss is obtained; Based on the initial consistency loss and the objective feature consistency constraint loss, an objective consistency loss is obtained.
9. The method according to claim 8, wherein The expression of the joint loss is: Among them, is the combined loss, is the loss function of the domain adaptation module, is the target image enhancement loss, is the target consistency loss.
10. A computer program product, characterized in that Including computer-executable instructions that, when executed, are used to implement the method described in claim 1.
Citation Information
Cited By
Image enhancement method, electronic equipment, storage medium and program product
CN121073789A