Underwater image enhancement method and device, storage medium and electronic equipment
By combining global and local enhancement branch networks, utilizing image-adaptive 3D lookup tables and lightweight convolutional neural networks, and introducing a cue learning strategy, the performance-efficiency balance problem in underwater image enhancement is solved, achieving highly efficient underwater image enhancement results.
Patent Information
- Application Number
- CN202410987837.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-07-23
AI Technical Summary
Existing technologies cannot effectively balance the performance and computational efficiency of underwater image enhancement, especially in resource-constrained underwater mobile terminals. Deep learning methods are computationally inefficient, while lightweight methods perform poorly when dealing with complex degradation types.
By combining global and local enhancement branch networks, utilizing an image-adaptive 3D lookup table and a lightweight convolutional neural network, and introducing a cue learning strategy, global and local enhancement branches are constructed, and finely enhanced images are obtained by superimposing them.
While maintaining performance, the number of model parameters is reduced, improving adaptability to images with different degradation types and achieving efficient underwater image enhancement, making it suitable for resource-constrained underwater applications.
Smart Images

Figure CN118982471A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an underwater image enhancement method, device, storage medium and electronic device. Background Art
[0002] Underwater optical imaging plays a vital role in many scientific fields such as marine ecology, marine geological science, and underwater resource discovery. However, due to the absorption and scattering effects of light in underwater environments, the acquired images often suffer from various quality degradation problems, such as color shift, reduced contrast, blurred details, and limited visibility. These problems not only affect human visual perception, but also pose a significant obstacle to the performance of machine learning-based algorithms in scene understanding and analysis, especially in resource-constrained underwater mobile terminal applications. Therefore, it is extremely urgent to design lightweight and efficient underwater optical image enhancement methods to meet these challenges. In recent years, underwater image enhancement methods based on deep learning have received widespread attention, but these methods have an obvious contradiction between enhanced performance and computational cost. For example, high-performance methods represented by Ucolor use deep and complex network architectures to achieve high-quality image enhancement effects. However, the model parameter volume and floating-point operation volume of such methods are usually high, resulting in low computational efficiency and difficulty in integration into underwater mobile platforms that require real-time feedback. On the contrary, lightweight methods such as Shallow-UWnet, although they have some control over the number of model parameters and computational cost, often perform poorly when dealing with complex and diverse underwater degradation types and fail to provide satisfactory enhancement effects. These lightweight methods are difficult to match the performance of deep and complex network models due to their limited adaptability to different degradation scenarios.
[0003] Therefore, the existing commonly used optimization schemes cannot effectively balance the enhancement performance and computational efficiency. There is an urgent need for an enhancement method that can not only cope with diverse underwater degradation problems but also operate efficiently under resource-constrained conditions to fill this gap in the existing technology. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide an underwater image enhancement method, device, storage medium and electronic device to solve the problem that the optimization scheme in the prior art cannot effectively balance the enhancement performance and computing efficiency.
[0005] The embodiment of the present application adopts the following technical solution: an underwater image enhancement method, comprising: establishing a training set consisting of a degraded underwater image and a reference image; constructing a global enhancement branch network, and outputting a global enhancement branch result of the degraded underwater image through the global enhancement branch network; constructing a local enhancement branch network, and outputting a local enhancement branch result of the degraded underwater image through the local enhancement branch network; superimposing the global enhancement branch result and the local enhancement branch result to obtain an enhanced image of the degraded underwater image; training the global enhancement branch network and the local enhancement branch network according to the enhanced image and the reference image until convergence; inputting the underwater image to be enhanced into the trained global enhancement branch network and the local enhancement branch network respectively to obtain an enhanced image of the underwater image to be enhanced.
[0006] The embodiment of the present application also provides an underwater image enhancement device, including: a training set establishment module, used to establish a training set consisting of a degraded underwater image and a reference image; a global network construction module, used to construct a global enhancement branch network, and output the global enhancement branch result of the degraded underwater image through the global enhancement branch network; a local network construction module, used to construct a local enhancement branch network, and output the local enhancement branch result of the degraded underwater image through the local enhancement branch network; a superposition processing module, used to superimpose the global enhancement branch result and the local enhancement branch result to obtain an enhanced image of the degraded underwater image; a training module, used to train the global enhancement branch network and the local enhancement branch network according to the enhanced image and the reference image until convergence; an enhancement module, used to input the underwater image to be enhanced into the trained global enhancement branch network and the local enhancement branch network respectively, to obtain an enhanced image of the underwater image to be enhanced.
[0007] An embodiment of the present application further provides a storage medium storing a computer program, which implements the steps of the above-mentioned underwater image enhancement method when executed by a processor.
[0008] An embodiment of the present application also provides an electronic device, comprising at least a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned underwater image enhancement method when executing the computer program on the memory.
[0009] The beneficial effects of the embodiments of the present application are as follows: by combining an image-adaptive three-dimensional lookup table and a lightweight convolutional neural network, a global enhancement branch network and a local enhancement branch network are constructed, thereby reducing the number of model parameters while ensuring performance; at the same time, a prompt learning strategy is introduced in different branches to enhance the adaptability of the model to images of different degraded types, thereby improving the underwater image enhancement effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0011] Figure 1 This is a flow chart of the underwater image enhancement method in the first embodiment of the present application;
[0012] Figure 2 This is a schematic diagram of the structure of the global enhanced branch network in the first embodiment of the present application;
[0013] Figure 3 This is a specific network architecture diagram of the prompt generation module in the first embodiment of the present application;
[0014] Figure 4 This is a specific network architecture diagram of the prompt interaction module in the first embodiment of the present application;
[0015] Figure 5 This is a schematic diagram of the structure of a local enhanced branch network in the first embodiment of the present application;
[0016] Figure 6 Schematic diagram of the structure of the underwater image enhancement device in the second embodiment of the present application. DETAILED DESCRIPTION
[0017] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.
[0018] Underwater optical imaging plays a vital role in many scientific fields such as marine ecology, marine geological science, and underwater resource discovery. However, due to the absorption and scattering effects of light in underwater environments, the acquired images often suffer from various quality degradation problems, such as color shift, reduced contrast, blurred details, and limited visibility. These problems not only affect human visual perception, but also pose a significant obstacle to the performance of machine learning-based algorithms in scene understanding and analysis, especially in resource-constrained underwater mobile terminal applications. Therefore, it is extremely urgent to design lightweight and efficient underwater optical image enhancement methods to meet these challenges. In recent years, underwater image enhancement methods based on deep learning have received widespread attention, but these methods have an obvious contradiction between enhanced performance and computational cost. For example, high-performance methods represented by Ucolor use deep and complex network architectures to achieve high-quality image enhancement effects. However, the model parameter volume and floating-point operation volume of such methods are usually high, resulting in low computational efficiency and difficulty in integration into underwater mobile platforms that require real-time feedback. On the contrary, lightweight methods such as Shallow-UWnet, although they have some control over the number of model parameters and computational cost, often perform poorly when dealing with complex and diverse underwater degradation types and fail to provide satisfactory enhancement effects. These lightweight methods are difficult to match the performance of deep and complex network models due to their limited adaptability to different degradation scenarios.
[0019] In order to solve the problem that the prior art cannot effectively balance the enhancement performance and the computational efficiency, the first embodiment of the present application provides an underwater image enhancement method, and its flow chart is as follows: Figure 1 As shown, it mainly includes steps S10 to S60:
[0020] S10, establishing a training set consisting of degraded underwater images and reference images.
[0021] In this embodiment, the degraded underwater image X∈R H×W×3 It is the underwater optical image directly collected by the underwater acquisition equipment, and the reference image Z∈R H×W×3 To obtain high-quality underwater images after degraded underwater image enhancement, various types of degraded underwater images X and reference images Z are selected to form a training set {X, Z}. When selecting degraded underwater images and reference images, multi-type and multi-size image selection should be performed, but before integrating the training set, all images of different sizes should be normalized.
[0022] S20, constructing a global enhancement branch network, and outputting a global enhancement branch result of the degraded underwater image through the global enhancement branch network.
[0023] Based on the requirements for efficiency and effect of underwater image enhancement, this embodiment adopts a global enhancement branch to represent the global context of the degraded underwater image, and then generates an adaptive three-dimensional lookup table to form a lightweight global enhancement branch network for outputting the global enhancement branch results of the degraded underwater image. Figure 2 The schematic diagram of the global enhanced branch network is shown below. Figure 2 The content shown describes the process of constructing a global enhanced branch network and outputting global enhanced branch results.
[0024] First, bilinear interpolation is performed on the degraded underwater image to obtain a low-resolution degraded underwater image. In order to improve the network's processing of low-resolution images with a high degree of degradation, this embodiment performs bilinear interpolation on the degraded underwater image X in the training set to obtain its corresponding low-resolution version X′∈R 256×256×3 .
[0025] Then the low-resolution degraded underwater image X′ is input into the lookup table backbone network for feature extraction to obtain the global context feature E∈r of the degraded underwater image h×w×M , where the lookup table backbone network consists of multiple convolutional layers, LeakyReLU activation functions and instance normalization layers. Figure 2 The lookup table backbone network shown includes four feature extraction layers formed by a 3×3 convolution layer, a LeakyReLU activation function and an instance normalization layer, and one feature extraction layer formed by a 1×1 convolution layer, a LeakyReLU activation function and an instance normalization layer. After being processed by the above feature extraction layers, the global context feature E of the degraded underwater image is obtained.
[0026] At the same time, the degraded underwater image X is input into the shallow feature extraction network of the global enhancement branch to obtain the shallow feature H∈R of the degraded underwater image H×W×M , where combined Figure 2 The shallow feature extraction network shown includes three 1×1 convolutional layers, that is, the degraded underwater image X is subjected to feature extraction of three 1×1 convolutional layers to obtain the corresponding shallow feature H.
[0027] Then the shallow features H and the randomly initialized hint component P0∈R S×D Input to the prompt generation module to obtain the prompt feature P carrying degradation information g ∈R S×D Specifically, the specific network architecture of the prompt generation module is as follows Figure 3 As shown in Figure 1, it mainly includes global average pooling processing, 1×1 convolution and Softmax function processing, point multiplication processing, etc.; when performing the prompt feature P carrying degradation information gWhen generating , we first perform global average pooling on the spatial dimension of the shallow feature H, and then process the shallow feature H after global average pooling through a 1×1 convolution layer and a Softmax function to generate a weight w∈R based on the input D Finally, the randomly initialized hint component P0 is multiplied by the weight w to obtain the hint feature P carrying the degradation information. g .
[0028] In the hint feature P that carries the degradation information g After being obtained, it and the global context feature E of the degraded underwater image are input into the prompt interaction module for dynamic feature calibration to obtain the degradation-aware global context feature E′. Specifically, Figure 4 The specific network architecture diagram of the prompt interaction module in this embodiment is shown, which mainly includes a cross attention module and a multi-branch gating mechanism. When calculating and outputting the degradation-aware global context feature E′, the global context feature E of the degraded underwater image is firstly processed by 1×1 convolution and 3×3 depth-separable convolution to obtain a compressed feature Then the compressed feature E c and the hint feature p that carries the degradation information g Input to the criss-cross attention module to generate cue-aware features
[0029] In some embodiments, the specific implementation process of the cross-attention module is as follows:
[0030] First, the request component p is encoded using two layers of 1×1 convolution:
[0031] P=W2(W1(E c )); (1)
[0032] Among them, W1(·) and W2(·) represent 1×1 convolution. At the same time, the mapping between key K and value V is learned using 1×1 convolution:
[0033] K=W3(P g ); (2)
[0034] V=W4(P g ); (3)
[0035] Among them, W3(·) and W4(·) both represent 1×1 convolution. Next, a matrix multiplication operation is performed between P and the transposed K to obtain the attention matrix A:
[0036]
[0037] Among them, α represents a learnable parameter to control the distribution of the attention matrix. Finally, the cue-aware feature E′ is generated based on A, V and the residual strategy c:
[0038] E′ c =E c +W5(A·V); (5)
[0039] Among them, W5(·) represents a 1×1 convolution.
[0040] Generate cue-aware features E′ c After that, the multi-branch gating mechanism is used for feature interaction. Specifically, the global context feature E is first evenly divided into N segmentation features along the channel dimension, where N is the total number of gating mechanism branches. Figure 4 The value of N is 4, which means that the global context feature E is segmented to obtain four segmentation features. Then in each branch, the segmentation feature E i and the cue perception feature E′ c Perform adaptive interaction to obtain the channel dimension interaction feature E′ of each branch i , where combined Figure 4 The architecture of the gating mechanism shown, channel dimension interaction feature E′ i The calculation formula is:
[0041]
[0042] Among them, ⊙ represents the element-by-element multiplication operation, represents the GELU nonlinear activation function, W6(·) and W7(·) represent 1×1 convolution. Finally, the channel-dimensional interaction feature E′ of each branch is i Splicing is performed to obtain the degradation-aware global context feature E′, that is, E′=[E′1; E′2; E′3; E′4], where [;] represents the operation of feature splicing along the channel dimension.
[0043] After obtaining the degradation-aware global context feature E′, the degradation-aware global context feature E′ is average-pooled and flattened to obtain Then Input into two fully connected layers to construct a three-dimensional lookup table T∈[0,1] 3×t×t×t As a global enhancement branch network, where t represents the number of elements in the three-dimensional lookup table in each color dimension.
[0044] After the three-dimensional lookup table T is constructed, the global enhancement branch result output of the degraded underwater image based on the three-dimensional lookup table can be realized. In this embodiment, the degraded underwater image adaptive enhancement is performed in two steps to obtain the global enhancement branch result Y1. Specifically, for the RGB value of any pixel in the degraded underwater image X, First, determine the corresponding position (x, y, z) and subscript index (i, j, k) in the three-dimensional lookup table. The calculation formula is:
[0045]
[0046] Where d = λ max / t,λ max is the maximum color value, Indicates floor operation. Then, based on the 8 elements located by the subscript index (i, j, k) in the three-dimensional lookup table, the color value corresponding to the output pixel is calculated by trilinear interpolation.
[0047]
[0048] Q x =xi,Q y =yj,Q z =zk; (10)
[0049] Among them, O c Let the value stored in the three-dimensional lookup table T, c∈{r,g,b}, and r,g,b represent the red, green and blue channels respectively, and Q represents the deviation between the exact position (x,y,z) and index (i,j,k) of the input color value in the lookup table. After the color value corresponding to each pixel in the degraded underwater image X is calculated, the global enhancement branch result Y1 can be formed according to the color value corresponding to each pixel in the degraded underwater image.
[0050] S30, constructing a local enhancement branch network, and outputting a local enhancement branch result of the degraded underwater image through the local enhancement branch network.
[0051] The local enhancement branch network of this embodiment is used to capture details of the input image. At the same time, the local enhancement branch also introduces a hint learning strategy to improve the adaptability to different degradation types. Figure 5 The schematic diagram of the structure of the local enhancement branch network in this embodiment is shown. The degraded underwater image X is first input into the local feature extraction network for local feature F∈R H×W×N Extraction, where the structure of the local feature extraction network is as follows Figure 5 As shown in the dotted box, it mainly includes three layers of 3×3 convolutional layers, global average pooling, 1×1 convolutional layers and ReLU activation function, 1×1 convolutional layers and Sigmoid activation function, and dot multiplication operations.
[0052] Then the local feature F and the randomly initialized hint component P0 are input into the hint generation module to obtain the hint feature P carrying local information. l ∈R S×D, where the structure of the generated module is also as Figure 3 As shown, the process of processing the local feature F and the randomly initialized hint component P0 by the hint generation module is the same as the process of processing the global context feature E and the randomly initialized hint component P0 in step S20, which will not be repeated here. l And the local feature F is input into the prompt interaction module to obtain the degradation-aware local feature F′, where the structure of the prompt interaction module is also as follows Figure 4 As shown, the prompt interaction module is used to prompt the feature P that carries local information l and local features F, and the processing of the global context features E and the prompt features P carrying degradation information by the prompt interaction module in step S20 g The steps are the same and will not be repeated here. After obtaining the degradation-perceived local feature F′, it is processed by a 1×1 convolution layer to obtain the local enhancement branch result Y2 of the degraded underwater image.
[0053] S40, superimposing the global enhancement branch result and the local enhancement branch result to obtain an enhanced image of the degraded underwater image.
[0054] For any degraded underwater image, after being enhanced by the global enhancement branch network and the local enhancement branch network respectively, the global enhancement branch result Y1 and the local enhancement branch result Y2 can be obtained. By superimposing the two, the finely enhanced predicted image Y corresponding to the degraded underwater image can be obtained.
[0055] S50, training the global enhancement branch network and the local enhancement branch network according to the enhanced image and the reference image until convergence.
[0056] During the training process, this embodiment can construct a loss function as the basis for the training and convergence of the two branch networks. For example, the loss function is constructed based on the predicted image Y formed by the final prediction of the global enhancement branch network and the local enhancement branch network and the reference image Z corresponding to the degraded underwater image. as follows:
[0057]
[0058] Based on the loss function between the predicted image and the reference image of each degraded underwater image, the parameters in the two branch networks are continuously optimized and trained until the loss function is minimized and convergence is completed. It is considered that the training of the global enhancement branch network and the local enhancement branch network is completed. It should be noted that the loss function formula shown in the above formula (11) is only one of the ones that can be used in image enhancement training. Other loss function formulas can also be used for actual processing. This embodiment does not impose specific restrictions. In some embodiments, the number of iterations can also be set as a convergence condition. For example, after the network has been iterated 500 times, the model is considered to have converged and the training is completed.
[0059] S60, inputting the underwater image to be enhanced into the trained global enhancement branch network and the local enhancement branch network respectively to obtain an enhanced image of the underwater image to be enhanced.
[0060] For the unenhanced underwater images to be processed, they are respectively input into the trained global enhancement branch network and the local enhancement branch network to obtain the global enhancement branch results and the local enhancement branch results. After integration, a high-precision enhanced image can be obtained.
[0061] In actual implementation, the trained network can also be verified by establishing a test set, which includes degraded underwater images and reference images different from those in the training set. In the testing stage, the test image is input into the trained network to generate an enhanced image.
[0062] To verify the performance of this embodiment, the inventors conducted performance tests based on two public data sets. The evaluation results of the UIEBD test set are shown in Table 1, where “-” indicates that the maximum memory is exceeded. It can be seen from Table 1 that the proposed method is superior to the existing methods in mainstream image quality evaluation indicators. On an RTX 2080Ti GPU, when enhancing underwater images with a resolution of 1920×1080, the proposed method achieved a running rate of 30 frames per second, which indicates that the method can better balance the enhancement effect and computational complexity. Table 2 shows the evaluation performance on the LSUI test set. It can be seen from Table 2 that the present method has good generalization performance.
[0063] Table 1
[0064]
[0065]
[0066] Table 2
[0067]
[0068] The lightweight dual-branch underwater optical image enhancement method based on prompt learning and lookup table proposed in this embodiment aims to improve the quality of underwater images efficiently and robustly. In order to adapt to various types of underwater degradation, prompt learning strategies are introduced in both branches. First, in terms of global enhancement, a lightweight lookup table backbone network is used to learn the global context representation, thereby guiding the generation of color conversion functions. At the same time, a shallow convolutional neural network is used to optimize the randomly initialized prompt features. These generated prompt components carry specific degradation context information. The global context representation is recalibrated through dynamic features to generate a three-dimensional lookup table (3D LUTs) adaptive to the input image. The global enhancement result is obtained by performing a lookup and interpolation operation between the degraded image and the generated 3DLUTs. Secondly, in terms of local enhancement, multiple convolution blocks are stacked to capture the rich texture and details of the input image. Similar to the global enhancement branch, the local enhancement branch uses the generated input condition prompts and feature interaction mechanism to improve the adaptability to different degradation types. Finally, the enhancement results of the global and local branches are fused to obtain a finely enhanced image.
[0069] The lightweight design of this embodiment allows it to control the model parameters to only about 0.03M while maintaining the enhancement performance, and can achieve real-time enhancement of 1920×1080 resolution images on a single RTX 2080Ti GPU. This design significantly reduces the consumption of computing resources and can be applied to various resource-constrained underwater applications. The prompt learning strategy enhances the adaptability of the model to images of different degradation types. The degradation types of underwater scenes are complex and diverse, and how to improve the generalization performance of the model is the key. This embodiment introduces a prompt learning strategy, which uses prompt words to encode the degradation information of the input image, and dynamically calibrates the features through a multi-branch prompt interaction module, thereby enhancing the applicability of the model to images of different degradation types.
[0070] Based on the same inventive concept, the second embodiment of the present application provides an underwater image enhancement device, whose structural schematic diagram is as follows: Figure 6As shown, it mainly includes the following modules coupled in sequence: a training set establishment module 10 is used to establish a training set consisting of a degraded underwater image and a reference image; a global network construction module 20 is used to construct a global enhancement branch network, and output the global enhancement branch result of the degraded underwater image through the global enhancement branch network; a local network construction module 30 is used to construct a local enhancement branch network, and output the local enhancement branch result of the degraded underwater image through the local enhancement branch network; an overlay processing module 40 is used to overlay the global enhancement branch result and the local enhancement branch result to obtain an enhanced image of the degraded underwater image; a training module 50 is used to train the global enhancement branch network and the local enhancement branch network according to the enhanced image and the reference image until convergence; an enhancement module 60 is used to input the underwater image to be enhanced into the trained global enhancement branch network and the local enhancement branch network respectively to obtain an enhanced image of the underwater image to be enhanced.
[0071] In some embodiments, the global network construction module 20 is specifically used to perform bilinear interpolation on the degraded underwater image to obtain a low-resolution degraded underwater image; input the low-resolution degraded underwater image into a lookup table backbone network to obtain global context features of the degraded underwater image; input the degraded underwater image into a shallow feature extraction network to obtain shallow features of the degraded underwater image; input the shallow features and the randomly initialized prompt components into a prompt generation module to obtain prompt features carrying degradation information; input the global context features and the prompt features into a prompt interaction module for dynamic feature calibration to obtain degradation-aware global context features; perform average pooling, flattening, and two-layer fully connected layer processing on the degradation-aware global context features to construct a three-dimensional lookup table as the global enhancement branch network for constructing the global enhancement branch network.
[0072] In some embodiments, the global network construction module 20 is also used to determine the position and subscript index corresponding to the RGB value of any pixel in the degraded underwater image in the three-dimensional lookup table; based on the 8 elements located by the subscript index in the three-dimensional lookup table, calculate the color value corresponding to the pixel through trilinear interpolation; and form the global enhancement branch result according to the color value corresponding to each pixel in the degraded underwater image.
[0073] In some embodiments, the global network construction module 20 is also specifically used to perform global average pooling on the spatial dimension of the shallow features; process the shallow features after global average pooling through a 1×1 convolution layer and a Softmax function to generate input-based weights; and multiply the randomly initialized hint component by the weight to obtain the hint feature carrying the degradation information.
[0074] In some embodiments, the prompt interaction module includes at least a cross-attention module and a multi-branch gating mechanism; the global network construction module 20 is also specifically used to perform 1×1 convolution and 3×3 depth-separable convolution processing on the global context features to obtain compressed features; the compressed features and the prompt features carrying degradation information are input into the cross-attention module to obtain prompt perception features; the global context features are evenly divided into N segmentation features along the channel dimension, where N is the total number of branches of the gating mechanism; in each branch, the segmentation features are adaptively interacted with the prompt perception features to obtain the channel dimension interaction features of each branch; the channel dimension interaction features of each branch are spliced to obtain the prompt features carrying degradation information.
[0075] In some embodiments, the local network construction module 30 is specifically used to input the degraded underwater image into a local feature extraction network to obtain local features; input the local features and the randomly initialized prompt components into a prompt generation module to obtain prompt features carrying local information; input the prompt features carrying local information and the local features into a prompt interaction module to obtain degradation-perceived local features; input the degradation-perceived local features into a 1×1 convolutional layer for processing to obtain local enhancement branch results of the degraded underwater image.
[0076] In some embodiments, the training module 50 is specifically used to establish a loss function based on the enhanced image and the reference image; train the global enhancement branch network and the local enhancement branch network until the function value of the loss function is minimized and convergence is completed.
[0077] This embodiment combines an image-adaptive three-dimensional lookup table and a lightweight convolutional neural network to construct a global enhancement branch network and a local enhancement branch network, thereby reducing the number of model parameters while ensuring performance; at the same time, a prompt learning strategy is introduced in different branches to enhance the adaptability of the model to images of different degradation types, thereby improving the underwater image enhancement effect.
[0078] Based on the same inventive concept, the third embodiment of the present application provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the underwater image enhancement method described in the first embodiment of the present application is implemented.
[0079] Based on the same inventive concept, the fourth embodiment of the present application provides an electronic device, which includes at least a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program on the memory, the underwater image enhancement method described in the first embodiment of the present application is implemented.
[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for underwater image enhancement, characterized in that: include: Establish a training set consisting of degraded underwater images and reference images; Constructing a global enhancement branch network, and outputting a global enhancement branch result of the degraded underwater image through the global enhancement branch network; Constructing a local enhancement branch network, and outputting a local enhancement branch result of the degraded underwater image through the local enhancement branch network; Superimposing the global enhancement branch result and the local enhancement branch result to obtain an enhanced image of the degraded underwater image; Training the global enhancement branch network and the local enhancement branch network according to the enhanced image and the reference image until convergence; The underwater image to be enhanced is respectively input into the trained global enhancement branch network and the local enhancement branch network to obtain an enhanced image of the underwater image to be enhanced.
2. The underwater image enhancement method according to claim 1, characterized in that: The constructing of the global enhanced branch network comprises: Performing bilinear interpolation on the degraded underwater image to obtain a low-resolution degraded underwater image; Inputting the low-resolution degraded underwater image into a lookup table backbone network to obtain global context features of the degraded underwater image; Inputting the degraded underwater image into a shallow feature extraction network to obtain shallow features of the degraded underwater image; Inputting the shallow features and the randomly initialized hint component into a hint generation module to obtain hint features carrying degradation information; Inputting the global context feature and the prompt feature into a prompt interaction module for dynamic feature calibration to obtain a degradation-aware global context feature; The degradation-aware global context features are average-pooled, flattened, and processed with two fully-connected layers, and a three-dimensional lookup table is constructed as the global enhancement branch network.
3. The underwater image enhancement method according to claim 2, characterized in that: Outputting the global enhancement branch result of the degraded underwater image through the global enhancement branch network includes: Determine the position and subscript index corresponding to the RGB value of any pixel in the degraded underwater image in the three-dimensional lookup table; Based on the eight elements located by the subscript index in the three-dimensional lookup table, a color value corresponding to the pixel is calculated by trilinear interpolation; The global enhancement branch result is formed according to the color value corresponding to each pixel in the degraded underwater image.
4. The underwater image enhancement method according to claim 2, characterized in that: The step of inputting the shallow features and the randomly initialized hint component into the hint generation module to obtain hint features carrying degradation information includes: Performing global average pooling on the spatial dimension of the shallow features; The shallow features after global average pooling are processed through a 1×1 convolution layer and a Softmax function to generate input-based weights; The randomly initialized hint component is multiplied by the weight to obtain the hint feature carrying the degradation information.
5. The underwater image enhancement method according to claim 4, characterized in that: The prompt interaction module at least includes a cross attention module and a multi-branch gating mechanism; the shallow features and the randomly initialized prompt component are input into the prompt generation module to obtain the prompt features carrying degradation information, including: Performing 1×1 convolution and 3×3 depth-separable convolution processing on the global context feature to obtain a compressed feature; Inputting the compression feature and the prompt feature carrying degradation information into the cross attention module to obtain a prompt perception feature; The global context feature is evenly divided into N segmentation features along the channel dimension, where N is the total number of gating mechanism branches; In each branch, the segmentation feature is adaptively interacted with the cue perception feature to obtain a channel dimension interaction feature of each branch; The channel dimension interaction features of each branch are concatenated to obtain the prompt features carrying the degradation information.
6. The underwater image enhancement method according to claim 1, characterized in that: The step of constructing a local enhancement branch network and outputting the local enhancement branch result of the degraded underwater image through the local enhancement branch network includes: Inputting the degraded underwater image into a local feature extraction network to obtain local features; Inputting the local features and the randomly initialized prompt component into a prompt generation module to obtain a prompt feature carrying local information; Inputting the prompt feature carrying local information and the local feature into a prompt interaction module to obtain a degradation-aware local feature; The degradation-perceived local features are input into a 1×1 convolutional layer for processing to obtain a local enhancement branch result of the degraded underwater image.
7. The underwater image enhancement method according to any one of claims 1 to 6, characterized in that: The training of the global enhancement branch network and the local enhancement branch network according to the enhanced image and the reference image until convergence comprises: Establishing a loss function according to the enhanced image and the reference image; The global enhancement branch network and the local enhancement branch network are trained until the function value of the loss function is minimized and convergence is completed.
8. An underwater image enhancement device, characterized in that: include: A training set building module, used for building a training set consisting of a degraded underwater image and a reference image; A global network construction module, used to construct a global enhancement branch network, and output a global enhancement branch result of the degraded underwater image through the global enhancement branch network; A local network construction module, used to construct a local enhancement branch network, and output a local enhancement branch result of the degraded underwater image through the local enhancement branch network; A superposition processing module, used for superimposing the global enhancement branch result and the local enhancement branch result to obtain an enhanced image of the degraded underwater image; A training module, used for training the global enhancement branch network and the local enhancement branch network according to the enhanced image and the reference image until convergence; The enhancement module is used to input the underwater image to be enhanced into the trained global enhancement branch network and the local enhancement branch network respectively to obtain an enhanced image of the underwater image to be enhanced.
9. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the underwater image enhancement method according to any one of claims 1 to 7 are implemented.
10. An electronic device comprising at least a memory and a processor, wherein a computer program is stored in the memory, wherein: The processor implements the steps of the underwater image enhancement method according to any one of claims 1 to 7 when executing the computer program on the memory.
Citation Information
Patent Citations
Image enhancement method and device, electronic equipment and computer readable storage medium
CN114742907A
Underwater image enhancement method combining physical prior and deep learning
CN116309232A
Progressive pixel-level adjustment low-illumination image enhancement method
CN117593222A
Image processing method and apparatus, terminal device, and storage medium
WO2023010754A1
Cited By
Image color enhancement method and device based on lookup table estimation and electronic equipment
CN120070290A
Image Color Enhancement Method, Device and Electronic Device Based on Lookup Table Estimation
CN120070290B
Underwater image enhancement method and device based on text-guided degradation generation and curriculum learning, terminal and medium
CN121998850A
