Underwater image enhancement model training method and enhancement method, equipment and medium

Abstract

The invention relates to an underwater image enhancement model training method, an underwater image enhancement method, equipment and a medium. The model comprises an underwater content encoder, an underwater distortion encoder, an air content encoder, a synthetic image generator, an enhanced image generator, a segmentation encoding and decoding network, a segmentation decoder and a semantic perception comparison module. The model training method comprises the steps of training image preprocessing, forward cross conversion, backward cross conversion, semantic perception and double-task guidance to obtain related data used for calculating model loss, and training an underwater image enhancement model by using the model loss. According to the enhancement method, the underwater image to be enhanced is enhanced by using the model trained by the training method. Compared with the prior art, the model training method based on deentanglement representation provided by the invention enables the trained model to concentrate on enhancing features beneficial to machine vision tasks so as to achieve the purpose of improving underwater machine vision task performance.

Description

Technical Field

[0001] The present invention relates to the field of underwater optical image processing, and in particular to an underwater image enhancement model training method and an enhancement method, device and medium. Background Art

[0002] Due to the uniqueness of the underwater environment, the practical application of underwater machine vision tasks in the marine environment is subject to certain limitations. First, factors such as light attenuation, scattering, and dispersion in the underwater environment cause underwater images to often have problems such as color cast, low contrast, and blur. This distortion makes the boundaries of targets in the image unclear and details lost, making it difficult for machine vision algorithms to accurately identify targets or extract features, such as algorithms for target detection, tracking, and recognition. Second, factors such as water flow, particulate matter, and biological tissue in the underwater environment may introduce noise and interference into the image, further reducing the accuracy and reliability of machine vision tasks.

[0003] In recent years, a large number of underwater image enhancement methods have been proposed to solve the problem of underwater image distortion. These methods can generally be divided into three categories: pixel-based methods, physical model-based methods, and deep learning-based methods. Pixel-based methods mainly improve the visual quality of images by modifying pixel values, but because they cannot capture higher-level semantic information, their ability to restore details of complex scenes is limited. Physical model-based methods take into account the physical characteristics of underwater optical transmission and scattering, but when the required model parameters are estimated inaccurately, the reliability of the method will be affected, which will indirectly affect the development of underwater detection technology. Deep learning-based methods enhance images by learning a large amount of underwater image data. However, it is difficult and expensive to obtain large-scale annotated underwater data. In addition, due to the complexity and variability of the underwater environment, deep learning models may exhibit poor generalization capabilities. However, these three types of methods mainly focus on optimizing the quality of human visual perception, lack targeted consideration of the characteristics and requirements of machine vision tasks, resulting in limited improvement in the performance of downstream machine vision tasks. The specific reasons are analyzed as follows:

[0004] Inconsistent data features: Underwater image enhancement algorithms are usually designed based on existing underwater data. However, data features vary greatly in different underwater environments, lighting conditions, and water quality. When the features of the training data are inconsistent with the target task data, the enhancement algorithm may not be able to effectively adapt to the features of the target task data, and thus fail to improve the performance of the target task.

[0005] Loss of key information: During underwater image enhancement, operations such as denoising and image smoothing may lead to the loss of key information required for certain tasks, thus affecting the accuracy of machine vision tasks.

[0006] Mismatched enhancement goals: Human vision-driven enhancement methods aim to improve image quality and visibility, such as denoising, contrast enhancement, and color correction. However, the goals of machine vision tasks are not completely consistent with those of human visual perception. If the enhancement method only focuses on image quality and ignores task-related features, the enhanced results may not directly provide substantial help for machine vision tasks.

[0007] With the deepening of the exploration of underwater vision, in addition to improving the visual quality of underwater images, the impact of enhancement methods on machine vision tasks such as target detection has attracted more and more attention. Some methods merge the hybrid synthesis model, enhancement model and detection perceptron into a cycle-consistent adversarial network, and use two cycle-consistent paths to achieve the conversion of unpaired images between underwater and aerial domains. The detection perceptron provides feedback information in the form of gradients to guide the enhancement model to generate visually pleasing or detection-friendly images at the patch level. Another method proposes a bidirectional constrained closed-loop adversarial enhancement module, which alleviates the need for paired data in an unsupervised manner and retains more information features. In addition, in order to make the enhanced image more realistic, a contrastive learning strategy is adopted in the training stage. Finally, a task-aware feedback module is embedded in the enhancement process to incorporate the gradient information of the detector to guide the enhancement in a direction that is conducive to detection. Although these methods have improved the visual effect and target detection ability of underwater image enhancement to a certain extent, these methods mainly focus on assisting machine vision tasks through visual quality improvement, ignoring the close integration of task requirements and image enhancement.

[0008] Therefore, it is a problem that needs to be solved to provide a method with strong generalization ability to meet the performance requirements of multiple tasks when facing complex and changeable underwater environments, as well as a model suitable for this method. Summary of the invention

[0009] The purpose of the present invention is to provide an underwater image enhancement model training method and enhancement method, device and medium in order to overcome the problem that the general underwater image enhancement method in the above-mentioned prior art cannot effectively improve the performance of underwater machine vision tasks.

[0010] The purpose of the present invention can be achieved by the following technical solutions:

[0011] According to a first aspect of the present invention, a method for training an underwater image enhancement model is provided, wherein the underwater image enhancement model comprises an underwater content encoder, an underwater distortion encoder, an aerial content encoder, a synthetic image generator, an enhanced image generator, a segmentation encoding and decoding network, a segmentation decoder, and a semantic perception contrast module;

[0012] The training method includes:

[0013] Preprocessing of training images: randomly combining the underwater data set and the aerial data set to form an unpaired data set, and preprocessing the unpaired data set to obtain an input image pair; the input image pair includes an underwater image and an aerial image;

[0014] Forward cross conversion: using an underwater content encoder, an underwater distortion encoder, an aerial content encoder, a synthetic image generator, and an enhanced image generator to perform forward cross conversion on the input image pair based on the disentangled representation, extracting first features from the underwater image and the aerial image respectively, and generating an enhanced image and a synthetic image based on the first features; the first features include underwater content features, underwater distortion features, and aerial content features;

[0015] Backward cross conversion: using an underwater content encoder, an underwater distortion encoder, an aerial content encoder, a synthetic image generator, and an enhanced image generator to perform backward cross conversion on the enhanced image and the synthetic image based on the disentangled representation to output a reconstructed image, wherein the reconstructed image includes an aerial reconstructed image and an underwater reconstructed image;

[0016] Semantic perception: Input the input image pair, enhanced image and synthesized image into the semantic perception contrast module to extract the second feature, and form the second feature into two triplets according to a preset rule, and process the triplets to output the contrast loss;

[0017] Dual-task guidance: the underwater content features are input into the segmentation decoder to output feature-level results, and the enhanced image is input into the segmentation encoding and decoding network to output image-level results;

[0018] Obtain model loss: Obtain model loss based on input image, reconstructed image, first feature, enhanced image, synthesized image, contrast loss, image-level results, and feature-level results;

[0019] Model training: The underwater image enhancement model is trained using model loss.

[0020] As a preferred technical solution, the network structures of the underwater content encoder, the aerial content encoder and the underwater distortion encoder are the same, and dense residual blocks are used to replace ordinary convolutions in the network structure. The network structure includes a convolutional normalized ReLu activation layer and multiple dense residual blocks, and a single dense residual block is connected by multiple convolutional ReLu activation layers in a channel splicing manner; the enhanced image generator and the synthetic image generator have the same structure, both including a convolutional normalized ReLu activation layer, a convolutional Tanh activation layer and multiple dense residual blocks.

[0021] As a preferred technical solution, the specific steps of the forward cross conversion include:

[0022] The underwater image is input into an underwater content encoder to output underwater content features, and the aerial image is input into an aerial content encoder to output aerial content features;

[0023] Inputting the underwater content features into an enhanced image generator to output an enhanced image;

[0024] The underwater image is input into an underwater distortion encoder, and undergoes at least four acquisitions during the encoding process, and the first acquisition result and underwater distortion characteristics are output;

[0025] The first acquisition result is added to the aerial content feature pixels, the pixel addition result, the underwater distortion feature and the aerial content feature are input into a synthetic image generator, and a synthetic image is output.

[0026] As a preferred technical solution, the specific steps of the backward cross conversion include:

[0027] The synthetic image is input into the underwater content encoder to output the synthetic content features, and the enhanced image is input into the aerial content encoder to output the enhanced content features;

[0028] Inputting the synthetic content features into an enhanced image generator to output an aerial reconstructed image;

[0029] The synthesized image is input into an underwater distortion encoder, and undergoes at least four acquisitions during the encoding process, and outputs the second acquisition result and the synthesized distortion feature;

[0030] The second acquisition result is added to the enhanced content feature pixels, the pixel addition result, the synthetic distortion feature and the enhanced content feature are input into a synthetic image generator, and an underwater reconstructed image is output.

[0031] As a preferred technical solution, the second feature includes enhanced image features, synthetic image features, underwater image features and aerial image features;

[0032] The preset rule refers to using a contrastive learning strategy to divide the second feature into enhanced triplets and synthetic triplets, specifically: the enhanced triplets use the enhanced image features as anchor points, the input image underwater features as positive samples, and the input image aerial features as negative samples; the synthetic triplets use the synthetic image features as anchor points, the input image aerial features as positive samples, and the input image underwater features as negative samples.

[0033] As a preferred technical solution, the model loss is obtained by weighted summation of cycle consistency loss, identity mapping loss, adversarial loss, contrast loss and task loss.

[0034] As a preferred technical solution, the expression of the cycle consistency loss is:

[0035]

[0036] Among them, L cc represents the cycle consistency loss, represents the underwater reconstructed image, X represents the underwater image in the input image, represents the aerial reconstructed image, and Y represents the aerial image in the input image;

[0037] The expression of the identity mapping loss is:

[0038]

[0039] Among them, L idt represents the identity mapping loss, is the first distortion feature, is the aerial content feature, G syn is a synthetic image generator;

[0040] The expression of the adversarial loss is:

[0041] L adv = -logD(X)-log(1-D(X Y ))

[0042] Among them, L adv represents the adversarial loss, X Y represents the synthetic image, D(·) represents the discriminator;

[0043] The expression of the contrast loss is:

[0044]

[0045] Among them, L con represents the contrast loss, G(Y X ) represents enhanced image features, G(X) represents underwater image features, G(Y) represents aerial image features, G(X Y ) represents the synthetic image features;

[0046] The expression of the task loss is:

[0047]

[0048] Among them, L task represents the task loss, CE(·) represents the cross entropy loss of the semantic segmentation task, is the task-level result, mask represents the semantic label, N seg (X Y ) represents the image-level results.

[0049] According to a second aspect of the present invention, an underwater image enhancement method is provided. The underwater image enhancement method inputs an underwater image to be enhanced into an underwater image enhancement model trained by the above method for enhancement processing, and specifically comprises the following steps:

[0050] The underwater image to be enhanced is input to the underwater content encoder to output underwater content features;

[0051] The enhanced image generator receives underwater content features and outputs an enhanced underwater image.

[0052] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0053] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described when executed by a processor.

[0054] Compared with the prior art, the present invention has the following advantages:

[0055] 1) The present invention trains the enhancement model from cycle consistency loss, identity mapping loss, adversarial loss, contrast loss and task loss respectively to make the enhancement model more suitable for machine vision tasks, wherein training with cycle consistency loss ensures the effective transmission of information in the enhancement and synthesis process; training with identity mapping loss improves the quality of the converted image and stabilizes the training process; training with contrast loss fully retains the content structure information related to the task; training with adversarial loss and task loss avoids information loss or distortion in the training process;

[0056] 2) The present invention introduces a semantic perception contrast module, which effectively avoids the loss of semantic information in the process of underwater image enhancement, especially in scenarios where common operations such as denoising and contrast enhancement may cause information loss. The module can retain key details related to the task and ensure that the enhanced image still has sufficient semantic information in a complex underwater environment;

[0057] 3) The present invention designs task guidance branches for the image and feature levels respectively, so that the underwater content encoder extracts features that are friendly to machine vision tasks, and further optimizes the enhanced image generator to facilitate the generation of images that are beneficial to the task, thereby improving the performance of underwater machine vision tasks;

[0058] 4) The present invention introduces disentangled representation learning to clearly distinguish between content features that are beneficial to the task and underwater distortion features that are not beneficial to the task, so that image enhancement can more effectively focus on retaining and enhancing task-related features, which not only improves the interpretability and generalization ability of the enhancement effect of the enhancement model, but also ensures that the enhanced image can better serve subsequent machine vision tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a framework diagram of the method of the present invention;

[0060] Figure 2 It is the design diagram of the forward cross conversion of the present invention;

[0061] Figure 3 It is the semantic perception design diagram of the present invention;

[0062] Figure 4 It is the dual-task guidance design diagram of the present invention;

[0063] Figure 5 Comparison of visualization results of the target detection task of the present invention;

[0064] Figure 6 Comparison of the semantic perception visualization results of the present invention;

[0065] Figure 7 Comparison of visualization results for the salient object detection task of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0067] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0068] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0069] Embodiment 1:

[0070] This embodiment provides a method for training an underwater image enhancement model for machine vision tasks. Due to the lack of a real underwater paired data set, the training method adopts an unsupervised enhancement method, which uses a cycle-consistent mapping to map unpaired underwater real images to undistorted aerial images. However, ordinary mapping methods may not accurately capture the intrinsic characteristics of the data and lack interpretability, making it difficult for the training model to generalize to tasks with different data characteristics in complex underwater environments. To overcome this limitation, the training method introduces disentangled representations to learn the potential features and representations of underwater images. These potential representations can capture important information and semantic content in the image, showing a higher level of abstraction and expression capabilities. In order to solve the problem of loss of key semantic information during the enhancement process, the training method proposes a semantic-aware feature extractor to extract high-resolution semantic-aware content features that are not related to underwater distortion. By introducing contrast loss during the training process, the network retains important details related to the task as much as possible. In order to more closely integrate the relationship between enhancement and machine vision tasks, the training method also introduces task-guided branches from both feature and image perspectives. At the feature level, it promotes the underwater content encoder to extract more features that are beneficial to machine vision tasks; at the image level, it further guides the enhancer to generate task-friendly images.

[0071] The underwater image enhancement model mentioned above includes underwater content encoder, underwater distortion encoder, aerial content encoder, synthetic image generator, enhanced image generator, segmentation encoding and decoding network, segmentation decoder and semantic perception comparison module. For the specific network structure, see Figure 1 .

[0072] The training method includes:

[0073] S1. Training image preprocessing:

[0074] S11, underwater dataset SUIM consists of 1634 underwater images with semantic labels, corresponding to which 1634 images with a resolution of 512*512 are randomly selected and cropped from the aerial high-resolution dataset Flickr2K;

[0075] S12. Randomly combine the underwater data set and the spatial data set to form an unpaired data group, preprocess the unpaired data group and adjust the size to 256*256 to obtain an input image pair, wherein the input image pair includes an underwater image and an aerial image, and the input image pair is used as the input of the enhanced image.

[0076] S2, forward cross conversion:

[0077] Since data sets in different underwater environments usually exhibit different data characteristics, it is difficult for general underwater image enhancement algorithms to be effectively generalized to different underwater scenes. In order to improve the interpretability and generalization ability of the network, this paper introduces disentangled representation learning. Specifically, the network details of the forward cross conversion are as follows: Figure 2 As shown, for underwater images, the underwater content encoder is used and underwater distortion encoder To separate task-friendly content features from task-unfriendly distortion features, the underwater content encoder Responsible for extracting underwater content features, while the underwater distortion encoder It is used to extract underwater distortion features; for images in the aerial domain, since there is no underwater distortion, only an aerial content encoder is required To extract its content features, the specific steps include:

[0078] S21, inputting the underwater image into the underwater content encoder Output underwater content features and feed aerial images into the aerial content encoder Outputting aerial content features;

[0079] S22, inputting underwater content features into enhanced image generator G enh The output enhanced image is expressed as:

[0080]

[0081] Y X represents the enhanced image, and X represents the underwater image;

[0082] S23, inputting the underwater image into the underwater distortion encoder And in the encoding process, at least four acquisitions are performed, and the first acquisition result and underwater distortion characteristics are output;

[0083] S24, adding the first acquisition result to the aerial content feature pixel, and inputting the pixel addition result, underwater distortion feature and aerial content feature into the synthetic image generator G syn In the output, the synthetic image is expressed as:

[0084]

[0085] X Y represents the synthetic image and Y represents the aerial image.

[0086] Moreover, the network structures of the underwater content encoder, the aerial content encoder and the underwater distortion encoder are the same. A U-Net-based encoder-decoder structure is adopted to use dense residual blocks instead of ordinary convolutions in the network structure. The network structure includes a convolutional normalized ReLu activation layer and multiple dense residual blocks. A single dense residual block is connected by multiple convolutional ReLu activation layers in a channel-wise splicing manner; the structures of the enhanced image generator and the synthetic image generator are the same, both of which include a convolutional normalized ReLu activation layer, a convolutional Tanh activation layer and multiple dense residual blocks.

[0087] S3, backward cross conversion:

[0088] The purpose of the forward cross transform is to generate an enhanced image Y X And the composite image X Y , and the purpose of the backward cross conversion is to use the conversion results and the original image to provide pixel-level constraints for network training. The specific steps include:

[0089] S31, inputting the synthesized image into an underwater content encoder to output a synthesized content feature, and inputting the enhanced image into an aerial content encoder to output an enhanced content feature;

[0090] S32, inputting the synthetic content feature into the enhanced image generator to output the aerial reconstructed image, the expression of which is:

[0091]

[0092] represents the aerial reconstructed image, X Y represents a composite image;

[0093] S33, inputting the synthesized image into an underwater distortion encoder, and undergoing at least four acquisitions during the encoding process, and outputting the second acquisition result and the synthesized distortion feature;

[0094] S34, adding the second acquisition result and the enhanced content feature pixel, inputting the pixel addition result, the synthetic distortion feature and the enhanced content feature into the synthetic image generator, and outputting the underwater reconstructed image, which is expressed as:

[0095]

[0096] represents the underwater reconstructed image, Y X Represents an enhanced image.

[0097] S4, semantic perception:

[0098] Underwater image enhancement involves a series of processing methods, such as noise removal, image smoothing and contrast enhancement, etc. However, these processing methods may introduce some errors or change the characteristics of the original image when performing local repair; although these methods can improve the visual quality of underwater images, some important detail information related to machine vision tasks may be lost; in order to minimize the loss of detail information, the idea of ​​contrastive learning is introduced into the proposed model during the training of the enhancement model; accordingly, the present invention proposes a semantic-aware contrast module, which guides the semantic-aware feature extractor to extract task-related semantic information through contrast loss, ensuring that the enhanced underwater image maintains the integrity of the original structural details while reducing the distortion caused by underwater conditions. The structure of the semantic-aware contrast module is as follows Figure 3 As shown, the specific steps are:

[0099] S41, use the semantic-aware feature extractor G(·) to extract underwater image X, aerial image Y, and enhanced image Y X And the composite image X Y The features of the above features maintain the same resolution as the input image to ensure that rich details are preserved; among them, the enhanced image feature G(Y X ), underwater image features G(X), aerial image features G(Y) and synthetic image features G(X Y ) and is called the second characteristic;

[0100] S42, using a contrastive learning strategy to divide the second feature into an enhanced triplet and a synthetic triplet, specifically: the enhanced triplet uses the enhanced image feature as an anchor point, the input image underwater feature is a positive sample, and the input image aerial feature is a negative sample;

[0101] The synthetic triplet uses the synthetic image features as anchors, the input image aerial features as positive samples, and the input image underwater features as negative samples.

[0102] S43. Move the anchor points closer to positive samples with the same content and away from negative samples with different content, ensuring that the enhancement process does not lose semantic detail information in the original image that is potentially beneficial for machine vision tasks.

[0103] S5, dual-task guidance:

[0104] In order to establish a closer connection between enhancement and machine vision tasks, the present invention sets a task guidance branch at the feature level and the image level respectively; at the feature level, by combining a task-specific constraint mechanism, the underwater content encoder becomes more adept at capturing the key information required for subsequent machine vision tasks and extracting features that are beneficial to the task; at the image level, the enhanced image generator generates a three-channel image that is beneficial to the task. The detailed structure of the dual-task guidance branch is as follows: Figure 4 As shown, the specific steps include:

[0105] S51, inputting underwater content features into a segmentation decoder to output a feature-level result, and inputting an enhanced image into a segmentation encoding and decoding network to output an image-level result;

[0106] S6. Get model loss:

[0107] S61. Calculate cycle consistency loss: Since the proposed unsupervised framework contains two cycles, in order to ensure the effective transmission of information in the enhancement and synthesis process, the present invention introduces cycle consistency loss. This loss function serves as a supervisory signal to guide the enhancement and synthesis network to retain the content of the input image. The cycle consistency loss expression is:

[0108]

[0109] Among them, L cc represents the cycle consistency loss, represents the underwater reconstructed image, X represents the underwater image in the input image, represents the aerial reconstructed image, and Y represents the aerial image in the input image;

[0110] S62. Calculate the identity mapping loss: This training method proposes to apply the identity mapping loss to improve the quality of the converted image and stabilize the training process. The expression is:

[0111]

[0112] Among them, L idt represents the identity mapping loss, is the first distortion feature, is the aerial content feature, G syn is a synthetic image generator;

[0113] S63. Calculate adversarial loss: used to distinguish between real underwater images X and synthetic underwater images X Y , the expression is:

[0114] L adv = -logD(X)-log(1-D(X Y ))

[0115] Among them, L adv represents the adversarial loss, X Y represents the synthetic image, D(·) represents the discriminator;

[0116] S64. Calculate contrast loss: In order to fully retain the content structure information related to the task, the feature extractor G(·) is used to extract semantic perception features, and L1 regularization is used to strengthen the contrast prior. The expression is:

[0117]

[0118] Among them, L con represents the contrast loss, G(Y X ) represents enhanced image features, G(X) represents underwater image features, G(Y) represents aerial image features, G(X Y ) represents the synthetic image features;

[0119] S65. Calculate task loss: In the dual-task guided branch, the underwater semantic segmentation task is jointly trained with the enhanced network. Therefore, the task loss consists of two parts, which are constrained at the feature level and the image level respectively. The expression is:

[0120]

[0121] Among them, L task represents the task loss, CE(·) represents the cross entropy loss of the semantic segmentation task, is the task-level result, mask represents the semantic label, N seg (X Y ) represents the image-level results.

[0122] S66. Calculate model loss: For the joint training process of the enhanced model, the model loss is obtained by weighted summation of cycle consistency loss, identity mapping loss, adversarial loss, contrast loss and task loss, and the expression is:

[0123] L total =λ1L cc +λ2L idt +λ3L adv +λ4L con +λ5L task ,

[0124] Among them, L total represents the total loss, λ1=10, λ2=5, λ3=1, λ4=5 and λ5=0.2 are hyperparameters.

[0125] S7. Model training: Use model loss to train the underwater image enhancement model.

[0126] This embodiment also provides an underwater image enhancement method, which uses the underwater image enhancement model trained by the above training method to achieve the purpose of enhancing the underwater image to be enhanced, and specifically includes the following steps:

[0127] A1: The underwater image to be enhanced is input to the underwater content encoder to output underwater content features;

[0128] A2: The enhanced image generator receives underwater content features and outputs an enhanced underwater image.

[0129] Embodiment 2:

[0130] In order to verify that the model trained by the underwater image enhancement model training method provided by the above embodiment has significant advantages, this embodiment evaluates the trained enhancement model from three tasks in machine vision tasks, including underwater target detection, underwater semantic segmentation and underwater significant target detection; it is compared with some advanced underwater image enhancement methods, and the Adam optimizer used in the enhancement model is set to perform end-to-end training for 200 cycles, and the learning rate in the first 100 cycles is fixed to 2×e -4 , the learning rate gradually decays to 0 in the last 100 cycles.

[0131] For underwater object detection:

[0132] The ChinaMM dataset is used for verification, with a total of 2071 training images and 676 test images, including three types of objects: sea cucumbers, sea urchins and scallops, and mAP@0.5:0.95% is used as an indicator for evaluating the accuracy of target detection. In order to study the impact of underwater image enhancement on the target detection task, in this embodiment, the enhancement results of different models are applied to a variety of detection algorithms, including two one-stage detection networks RetinaNet and SSD, a two-stage detection network Faster-RCNN, and a network FERNet specially designed for underwater target detection. The detection performance comparison results of the four networks are shown in Table 1.

[0133] The experimental results show that the model significantly improves the mAP of the four detectors by 4.6%, 5.7%, 3.8% and 1.6% respectively compared with the original images, while the detection performance of other models shows different degrees of decline in all four detection networks. It can be seen that the enhanced model trained by the underwater image enhancement model training method provided by the present invention has better performance in the underwater target detection task, because the features that are beneficial to the task are focused on during the training process; Figure 5 The detection visualization results of different enhancement algorithms on the RetinaNet network are given; it can be clearly seen from the figure that not only the detection effect for difficult areas is excellent, but the detected targets also show higher confidence.

[0134] Table 1 Comparison of underwater target detection performance

[0135]

[0136] For underwater semantic segmentation:

[0137] The model was verified on two commonly used segmentation network models, DeepLabv3+ and U-Net, and its performance in mIoU and mPA was evaluated. Table 2 shows the segmentation performance comparison of different methods on DeepLabv3+ and U-Net networks. It can be seen that the segmentation performance of this model is excellent on both networks. Compared with the original image, the mIoU on the two segmentation networks is increased by 2.8% and 2.6% respectively;

[0138] Figure 6 A comparison of the segmentation visualization results on the SUIM test set is given. It can be seen from the figure that the segmentation result obtained by this model is most similar to the segmentation label, and it also has a good performance in classifying pixels in the blurred background area. This shows that the enhanced model trained by the underwater image enhancement model training method provided by the present invention can effectively remove the interference of underwater distortion on image semantics.

[0139] Table 2 Comparison of underwater semantic segmentation performance

[0140]

[0141] For underwater salient object detection:

[0142] Performance comparison is performed on the U2NET network. Table 3 gives the quantitative comparison results of the three evaluation indicators on the UFO-120 and USOD10K datasets. It can be seen from the table that this model is always better than other methods in all three indicators, and the performance improvement of most compared methods relative to the original image is relatively limited, or even decreased. On the UFO-120 dataset, compared with the original image, the F-measure of this model is improved by 3.4%, the E-measure is improved by 2.0%, and the MAE is reduced by 0.008. On the USOD10K dataset, the F-measure is improved by 2.5%, the E-measure is improved by 1.7%, and the MAE is reduced by 0.006; it can be seen that the introduced disentanglement representation separates the underwater content and distortion. Even if only the segmentation task loss is introduced during the training process to optimize the enhanced network, it still has a great advantage in the target detection and salient target detection tasks, which effectively verifies that the invented method has a strong generalization ability.

[0143] The visualization results of salient object detection on the UFO-120 dataset are as follows: Figure 7 As shown in the figure, it can be seen that algorithms such as UDCP and Fusion mistakenly identify the background as a salient object. In contrast, the enhanced model trained by the underwater image enhancement model training method provided by the present invention shows more accurate edge detection, and is even subjectively superior to the salient label.

[0144] Table 3 Comparison of underwater salient target detection performance

[0145]

[0146] In order to further verify the effectiveness of this model on multiple machine vision tasks, three key factors are also considered in this embodiment, including: disentangled representation, semantic perception contrast module and dual-task guided branch.

[0147] The baseline truth is a CycleGAN network without any modules added. For the target detection task, the disentangled representation is added to the baseline truth. On the four detection networks, the mAP is improved by 1.6% to 4.7%; the addition of the dual-task guidance branch further improves the performance by 3.1% to 7.9%; the addition of the semantic-aware contrast module improves the detection performance by 2.8% to 7.2%. For the semantic segmentation task, compared with the baseline truth, the addition of the disentangled representation improves the mIoU by 4.8% and 1.4% on the DeepLabv3+ and U-Net networks respectively; the addition of the dual-task guidance branch further improves the mIoU by 2.0% and 2.4%; the addition of the semantic-aware contrast module improves the mIoU by 1.1% and 0.5%. For the salient object detection task, compared with the baseline truth, the model after introducing the disentangled representation improves the F-measure by 2.1% and 0.9% on the two datasets respectively; after adding the dual-task guided branch, the F-measure is improved by 3.1% and 0.6% respectively; the addition of the semantic-aware contrast module improves the F-measure by 4.0% and 0.3% respectively.

[0148] In summary, the enhanced model trained by the underwater image enhancement model training method provided by the above embodiment has significant advantages in a variety of different underwater machine vision tasks, effectively improving the performance of target detection, semantic segmentation and significant target detection tasks in complex underwater environments.

[0149] Embodiment 3:

[0150] The present embodiment provides an electronic device, which includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0151] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0152] The processing unit performs the various methods and processes described above, such as methods S1 to S7 and A1 to A2. For example, in some embodiments, methods S1 to S7 and A1 to A2 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S7 and A1 to A2 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S7 and A1 to A2 in any other appropriate manner (e.g., by means of firmware).

[0153] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0154] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0155] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0156] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for training an underwater image enhancement model, characterized in that: The underwater image enhancement model includes an underwater content encoder, an underwater distortion encoder, an aerial content encoder, a synthetic image generator, an enhanced image generator, a segmentation encoding and decoding network, a segmentation decoder and a semantic perception comparison module; The training method includes: Preprocessing of training images: randomly combining the underwater data set and the aerial data set to form an unpaired data set, and preprocessing the unpaired data set to obtain an input image pair; the input image pair includes an underwater image and an aerial image; Forward cross conversion: using an underwater content encoder, an underwater distortion encoder, an aerial content encoder, a synthetic image generator, and an enhanced image generator to perform forward cross conversion on the input image pair based on the disentangled representation, extracting first features from the underwater image and the aerial image respectively, and generating an enhanced image and a synthetic image based on the first features; the first features include underwater content features, underwater distortion features, and aerial content features; Backward cross conversion: using an underwater content encoder, an underwater distortion encoder, an aerial content encoder, a synthetic image generator, and an enhanced image generator to perform backward cross conversion on the enhanced image and the synthetic image based on the disentangled representation to output a reconstructed image, wherein the reconstructed image includes an aerial reconstructed image and an underwater reconstructed image; Semantic perception: Input the input image pair, enhanced image and synthesized image into the semantic perception contrast module to extract the second feature, and form the second feature into two triplets according to a preset rule, and process the triplets to output the contrast loss; Dual-task guidance: the underwater content features are input into the segmentation decoder to output feature-level results, and the enhanced image is input into the segmentation encoding and decoding network to output image-level results; Obtain model loss: Obtain model loss based on input image, reconstructed image, first feature, enhanced image, synthesized image, contrast loss, image-level results, and feature-level results; Model training: The underwater image enhancement model is trained using model loss.

2. The underwater image enhancement model training method according to claim 1, characterized in that: The network structures of the underwater content encoder, the aerial content encoder and the underwater distortion encoder are the same, and dense residual blocks are used to replace ordinary convolutions in the network structure. The network structure includes a convolutional normalized ReLu activation layer and multiple dense residual blocks, and a single dense residual block is connected by multiple convolutional ReLu activation layers in a channel splicing manner; the structures of the enhanced image generator and the synthetic image generator are the same, both including a convolutional normalized ReLu activation layer, a convolutional Tanh activation layer and multiple dense residual blocks.

3. The underwater image enhancement model training method according to claim 1, characterized in that: The specific steps of the forward cross conversion include: The underwater image is input into an underwater content encoder to output underwater content features, and the aerial image is input into an aerial content encoder to output aerial content features; Inputting the underwater content features into an enhanced image generator to output an enhanced image; The underwater image is input into an underwater distortion encoder, and undergoes at least four acquisitions during the encoding process, and the first acquisition result and underwater distortion characteristics are output; The first acquisition result is added to the aerial content feature pixels, the pixel addition result, the underwater distortion feature and the aerial content feature are input into a synthetic image generator, and a synthetic image is output.

4. The underwater image enhancement model training method according to claim 1, characterized in that: The specific steps of the backward cross conversion include: The synthetic image is input into the underwater content encoder to output the synthetic content features, and the enhanced image is input into the aerial content encoder to output the enhanced content features; Inputting the synthetic content features into an enhanced image generator to output an aerial reconstructed image; The synthesized image is input into an underwater distortion encoder, and undergoes at least four acquisitions during the encoding process, and outputs the second acquisition result and the synthesized distortion feature; The second acquisition result is added to the enhanced content feature pixels, the pixel addition result, the synthetic distortion feature and the enhanced content feature are input into a synthetic image generator, and an underwater reconstructed image is output.

5. The underwater image enhancement model training method according to claim 1, characterized in that: The second features include enhanced image features, synthetic image features, underwater image features and aerial image features; The preset rule refers to using a contrastive learning strategy to divide the second feature into enhanced triplets and synthetic triplets, specifically: the enhanced triplets use the enhanced image features as anchor points, the input image underwater features as positive samples, and the input image aerial features as negative samples; the synthetic triplets use the synthetic image features as anchor points, the input image aerial features as positive samples, and the input image underwater features as negative samples.

6. The underwater image enhancement model training method according to claim 1, characterized in that: The model loss is obtained by weighted summation of cycle consistency loss, identity mapping loss, adversarial loss, contrast loss and task loss.

7. The underwater image enhancement model training method according to claim 6, characterized in that: The expression of the cycle consistency loss is: Among them, L cc represents the cycle consistency loss, represents the underwater reconstructed image, X represents the underwater image in the input image, represents the aerial reconstructed image, and Y represents the aerial image in the input image; The expression of the identity mapping loss is: Among them, L idt represents the identity mapping loss, is the first distortion feature, is the aerial content feature, G syn is a synthetic image generator; The expression of the adversarial loss is: L adv =-logD(X)-log(1-D(X Y )) Among them, L adv represents the adversarial loss, X Y represents the synthetic image, D(·) represents the discriminator; The expression of the contrast loss is: Among them, L con represents the contrast loss, G(Y X ) represents the enhanced image features, G(X) represents the underwater image features, G(Y) represents the aerial image features, G(X Y ) represents the synthetic image features; The expression of the task loss is: Among them, L task represents the task loss, CE(·) represents the cross entropy loss of the semantic segmentation task, is the task-level result, mask represents the semantic label, N seg (X y ) represents the image-level results.

8. An underwater image enhancement method, characterized in that: The underwater image enhancement method inputs the underwater image to be enhanced into the underwater image enhancement model trained by any one of the training methods of claims 1 to 7 for enhancement processing, and specifically comprises the following steps: The underwater image to be enhanced is input to the underwater content encoder to output underwater content features; The enhanced image generator receives underwater content features and outputs an enhanced underwater image.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images