Underwater image dynamic enhancement method based on pyramid network and application

Abstract

The invention relates to the technical field of underwater image enhancement processing, in particular to a pyramid network-based underwater image dynamic enhancement method and application, and the method comprises the steps: decomposing an input underwater image into layered image groups; using a deterministic color mapping network to convert the low-frequency layer input image into a low-frequency layer output image; using a double convolution attention module to obtain enhanced input features; using a packet expansion feedforward network to obtain a pixel-by-pixel mask; performing operation on the enhanced input features and step-by-step pixel masks to obtain high-frequency components; and reconstructing the low-frequency layer output image and the high-frequency component by using a Laplacian pyramid network to obtain an enhanced output underwater image. In conclusion, the underwater image can be dynamically enhanced, the adaptability is high, the method is suitable for various water bodies, color distortion can be effectively corrected, noise introduction is restrained, more image details can be reserved, and the underwater image has better visibility and higher recognition degree.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater image enhancement processing, and in particular to an underwater image dynamic enhancement method based on a pyramid network and an application thereof. Background Art

[0002] In underwater environments, due to the scattering and absorption of light and the influence of suspended particles in the water, underwater images usually have problems such as color distortion, low contrast, and poor clarity. These factors seriously affect the quality of underwater images, which brings difficulties to underwater detection, search, monitoring and other tasks. With the widespread application of underwater detection and monitoring technologies, such as marine archaeology, marine resource development, and marine biological research, higher requirements are placed on the processing and enhancement of underwater images. Improving the visibility and detail restoration of underwater images through image enhancement technology is crucial to improving the success rate of these tasks.

[0003] Existing underwater image enhancement technologies mainly target optical distortion and color distortion, and have developed a series of methods to improve image quality. However, existing technologies still cannot completely overcome the complex optical conditions in underwater environments, and are prone to introducing noise or causing detail loss during image restoration. Although traditional enhancement methods have improved the visual effects of images to a certain extent, they have poor adaptability to different water depths, lighting conditions, and different water compositions, and have failed to achieve real-time enhancement in dynamic scenes.

[0004] At present, the existing underwater image enhancement technology solutions are mainly divided into two categories: image enhancement methods based on physical models and enhancement methods based on image processing algorithms.

[0005] (1) Methods based on physical models: This type of method uses the physical properties of underwater imaging to establish a light propagation model to reversely deduce and correct the degradation of underwater images. Typical physical models include the atmospheric scattering model (DCP) and the color compensation model (CCM), which separate the scattered light and background light to restore the true color and contrast in the image. The advantage of this type of method is its strong physical interpretation, but because the model establishment depends on the optical parameters of the water body, when the water quality changes or the environment is complex, the model exhibits poor adaptability.

[0006] For example, the Chinese patent document with publication number CN110851965A discloses a light source optimization method and optimization system based on a physical model, which constructs an underwater camera model, an underwater light source model, a water effect model, a complete underwater optical imaging model, simulation method parameter setting, image quality evaluation standards, optimization algorithms, and experimental verification.

[0007] (2) Methods based on image processing algorithms: These methods do not rely on the physical model of the underwater environment, and mainly use image enhancement algorithms such as histogram equalization and contrast enhancement to improve the visual quality of images. In recent years, deep learning technology has also been widely used in the field of underwater image enhancement. Based on deep learning models such as convolutional neural networks (CNNs), networks are trained to automatically enhance images. However, deep learning methods rely on a large amount of labeled data and have limited generalization capabilities for complex underwater environments, which can easily lead to performance degradation in complex scenarios.

[0008] For example, a Chinese patent document with publication number CN115034965A discloses a super-resolution underwater image enhancement method and system based on deep learning, which includes: constructing an improved generative adversarial network model based on a generative adversarial network and a deep residual multiplier; inputting an underwater image into the improved generative adversarial network model for training to generate an enhanced super-resolution underwater image. The system includes: a construction module and a training module. By using the present invention, the visual quality of underwater images can be improved while improving the resolution of underwater images.

[0009] It can be seen that the existing technical solutions have the following technical disadvantages: Poor adaptability: Physical model-based methods usually rely on the optical properties of water bodies, such as turbidity, depth, and light direction. Therefore, when the water environment changes, the model has poor adaptability. Different water depths and lighting conditions will lead to unstable enhancement effects and fail to achieve consistent image quality improvement in various complex environments.

[0010] Color distortion problem: Light of certain wavelengths in underwater environments will be absorbed to varying degrees, causing color distortion in the image. Existing methods often have difficulty in effectively correcting color deviations and restoring true natural colors. Even if some methods can perform color compensation, they can easily introduce excessive color enhancement under certain conditions, resulting in unnatural image effects.

[0011] Noise introduction: Due to the poor quality of underwater images, especially in deep water with insufficient light, image enhancement methods may amplify the noise in the image. Some contrast enhancement-based technologies tend to amplify the noise while enhancing image details, resulting in reduced image visibility.

[0012] Loss of details: Some enhancement methods tend to ignore the restoration of details when improving the overall contrast and color saturation of the image, especially for underwater images under the influence of strong scattered light. Existing technologies often fail to retain enough detail information while ensuring image clarity, resulting in a lack of fine structure in the restored image.

[0013] Therefore, it is necessary to improve the prior art to overcome the above technical shortcomings. Summary of the invention

[0014] In order to overcome the above-mentioned shortcomings, the present invention aims to provide a technical solution that can solve the above-mentioned problems.

[0015] The present invention provides an underwater image dynamic enhancement method based on a pyramid network, comprising the following steps: Step 1: First, the initial input underwater image is decomposed into a layered image group with a total of L layers using a Laplacian pyramid network, where the Lth layer is a low-frequency layer input image, and the 1st to L-1th layers are a group of high-frequency layer input image groups; Step 2, using a deterministic color mapping network, color correction and illumination compensation are performed on the low-frequency layer input image to convert it into a low-frequency layer output image; Step 3: Use the dual convolution attention module to extract and enhance the input features of the input image of the nth layer to obtain the enhanced input features D of the input image. n ; Step 4: Use the grouped dilation feedforward network to perform a refinement operation on the output image of the n+1th layer to obtain the pixel-by-pixel mask M after the output image refinement operation. n+1 ; Step 5: perform an operation on the input features of the nth layer input image after enhancement in step 3 and the pixel-by-pixel mask of the n+1th layer output image after thinning in step 4 to obtain the high-frequency component h of the high-frequency layer output image of the nth layer. n ; Step 6: Repeat steps 3, 4 and 5 for L-1 times to obtain a set of high-frequency components HF, HF=[h L-1 ,…,h n+1 ,h n ,…,h2,h1]; Step 7: Use the Laplacian pyramid network to reconstruct the low-frequency layer output image and the high-frequency component set HF to obtain an enhanced output underwater image.

[0016] As a further solution of the present invention: in step 2, the deterministic color mapping network is a convolutional neural network, and its conversion process includes the following steps: Step 2.1, expand the low-frequency layer input image into a two-dimensional matrix; Step 2.2, use the projection matrix P(k, 3) to perform matrix multiplication with the two-dimensional matrix, so as to embed each pixel in the low-frequency layer input image into a k-dimensional vector space; Step 2.3, performing matrix multiplication operation on the embedded k-dimensional vector space and the color mapping matrix T(k, k) to adjust the color; In step 2.4, the projection matrix Q(3, k) is used to perform matrix multiplication with the adjusted vector space, the adjusted vector space is converted back to the RGB color space, and each pixel is reassembled into an image of the original size, thereby outputting a color-enhanced low-frequency layer output image.

[0017] As a further solution of the present invention: In step 3, preferably, the dual convolution attention module specifically includes the following steps: extracting input features of the input image through two convolutional layers, and using the LeakyReLU activation function for nonlinear activation.

[0018] As a further solution of the present invention: in step 4, the refinement operation further includes using at least two block group expansion feedforward networks to form multiple block group expansion feedforward networks, and respectively performing refinement adjustment on the output features of the output image, specifically including the following steps: Step 4.1, using multiple block grouping expansion feedforward networks, cascading them in sequence, and adjusting the output features of the output image step by step, so as to gradually correct the output features; Step 4.2, the corrected features fed to each group expansion feedforward network are also densely connected to the output of the last group expansion feedforward network and connected to form the total output features of the multi-block group expansion feedforward network; Step 4.3, then use the channel attention module to adaptively adjust the weights of the channel dimension of the total output feature to obtain the pixel-by-pixel mask after the refinement operation on the output feature.

[0019] As a further solution of the present invention: in step 4, the group expansion feedforward network specifically includes the following method: Use two 3x3 convolutional layers in series and combine them with the GELU activation function to capture the local spatial features of the output features of the output image; At the same time, two 1x1 convolutional layers are used in series, combined with the GELU activation function, to capture the local spatial features of the output features of the output image; The two sets of local spatial features are added bit by bit, and the result is then residually connected with the output features of the output image; An adaptive hybrid attention module is used to capture and form a 3x3 convolutional layer after capture, which is then residually connected with the output features of the output image to finally obtain a pixel-by-pixel mask after the output features are refined.

[0020] As a further solution of the present invention: in step 5, the input feature after the n-th layer input image enhancement is set to D n ; Set the pixel-by-pixel mask after the thinning operation of the n+1th layer output image to M n+1; After calculation, the high-frequency component h of the high-frequency layer output image of the nth layer is obtained n ; The specific calculation formula is: , in, is the matrix multiplication operation, It is a bit-by-bit addition operation.

[0021] As a further solution of the present invention: in step 4, the pixel-by-pixel mask of the low-frequency layer image of the Lth layer is obtained by the following steps: Use bilinear operation to upsample the low-frequency layer input image and the low-frequency layer output image respectively to improve their resolution, and © connect the low-frequency layer input image and the low-frequency layer output image after the improved resolution to form [low-frequency layer input image, low-frequency layer output image]; The connected [low-frequency layer input image, low-frequency layer output image] is fed into the double convolution attention module to obtain the enhanced input feature D L ; Use the group-expanded feed-forward network to enhance the input feature D L Perform a refinement operation to obtain the pixel-by-pixel mask LF of the Lth low-frequency layer image L .

[0022] The present invention also provides an underwater image dynamic enhancement system based on a pyramid network, the system comprising: A decomposition module is used to decompose the initial input underwater image into a layered image group with a total of L layers using a Laplacian pyramid network, wherein the Lth layer is a low-frequency layer input image, and the 1st to L-1th layers are a group of high-frequency layer input image groups; A conversion module, used to perform color correction and illumination compensation on the low-frequency layer input image using a deterministic color mapping network to convert it into a low-frequency layer output image; An enhancement module is used to extract and enhance input features of the input image of the nth layer using a dual convolutional attention module to obtain enhanced input features of the input image; A refinement module, used to perform a refinement operation on the output features of the output image of the n+1th layer using a group dilation feedforward network, and obtain a pixel-by-pixel mask of the output image after the refinement operation; The operation module is used to perform operation on the enhanced input features and the pixel-by-pixel mask after the refinement operation to obtain the high-frequency component of the high-frequency layer output image of the nth layer; The repetition module is used to repeat the enhancement module, refinement module and operation module. After repeating L-1 times, a set of high-frequency component sets HF is obtained, HF=[h L-1 ,…,h n+1 ,hn ,…,h2,h1]; The reconstruction module is used to reconstruct the low-frequency layer output image and the high-frequency component set HF using a Laplacian pyramid network to obtain an enhanced output underwater image.

[0023] The present invention also provides a device, comprising: Memory for storing computer programs; The processor is used to execute the computer program and implement the above-mentioned underwater image dynamic enhancement method based on pyramid network.

[0024] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer processor, the above-mentioned method for dynamic enhancement of underwater images based on a pyramid network is implemented.

[0025] Compared with the prior art, the present invention has the following beneficial effects: In view of the defects in the prior art, the present invention proposes a more robust and efficient underwater image enhancement method, aiming to achieve the following improvements: Enhanced adaptability: By combining the physical model with the deep learning algorithm, introducing the adaptive color mapping matrix T and the channel attention layer, the present invention can adaptively adjust the enhancement strategy in various underwater environments. Whether it is shallow or deep water, clear or turbid water, it can maintain consistent image enhancement effects.

[0026] Effectively correct color distortion: Through a deterministic color mapping network, the present invention uses an adaptive color correction algorithm to dynamically adjust color compensation for different water depths and lighting conditions, ensuring that the generated images have natural color performance and avoiding unnatural phenomena caused by over-enhancement.

[0027] Suppressing noise introduction: By expanding the feedforward network in groups and introducing an adaptive hybrid attention module, the present invention can effectively filter out noise while enhancing image details, ensuring image clarity and visual quality, especially in low-contrast or poor-light scenes.

[0028] Preserve image details: By combining the input features and output features of the low-frequency layer image, the high-frequency components are refined to achieve the detail enhancement algorithm of the present invention. While enhancing the contrast, it focuses on restoring the image details, ensuring that the texture and structural information in the image is fully preserved, and improving the recognition ability of underwater targets.

[0029] Therefore, through the above improvements, the present invention can provide a method and application of underwater image dynamic enhancement based on a pyramid network, so as to dynamically enhance underwater images, and has strong adaptability, is suitable for a variety of different water bodies, can effectively correct color distortion, suppress noise introduction, and can retain more image details, so that underwater images have better visibility and higher recognition.

[0030] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 It is a schematic block diagram of the process of the present invention; Figure 2 is a schematic block diagram of the process of pixel-by-pixel masking of the L-th low-frequency layer image of the present invention; Figure 3 It is a schematic diagram of the technical principle framework of the present invention; Figure 4 It is a schematic diagram of the framework of the group expansion feedforward network of the present invention; Figure 5 It is a schematic diagram of the Laplace pyramid decomposition of the present invention. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] See also Figures 1 to 5 In an embodiment of the present invention, a method for dynamic enhancement of underwater images based on a pyramid network comprises the following steps: Step 1: First, the initial input underwater image is decomposed into a total of L layers of layered image groups using a Laplacian pyramid network, and the resolution of the layered images from the 1st layer to the Lth layer is gradually reduced, where the Lth layer is a low-frequency layer input image, and the 1st layer to the L-1th layer are a group of high-frequency layer input image groups; Step 2, using a deterministic color mapping network, color correction and illumination compensation are performed on the low-frequency layer input image to convert it into a low-frequency layer output image; Step 3: Use the dual convolution attention module to extract and enhance the input features of the input image of the nth layer to obtain the enhanced input features D of the input image. n ; Step 4: Use the grouped dilation feedforward network to perform a refinement operation on the output image of the n+1th layer to obtain the pixel-by-pixel mask M after the output image refinement operation. n+1 Specifically, the output feature can be a high-frequency component feature of the high-frequency layer output image, or a low-frequency component feature of the low-frequency layer output image.

[0035] Step 5: Perform matrix multiplication and bit-by-bit addition operations on the input features of the nth layer input image after enhancement in step 3 and the pixel-by-pixel mask after the thinning operation of the n+1th layer output image in step 4 to obtain the high-frequency component h of the high-frequency layer output image of the nth layer. n ; Step 6: Repeat steps 3, 4 and 5 for L-1 times to obtain a set of high-frequency components HF, HF=[h L-1 ,…,h n+1 ,h n ,…,h2,h1]; Step 7: Use the Laplacian pyramid network to reconstruct the low-frequency layer output image and the high-frequency component set HF to obtain an enhanced output underwater image.

[0036] Specifically, the present invention proposes an end-to-end framework based on Laplacian pyramid network to reduce computational complexity. Given an input underwater image, it is first decomposed into a Laplacian pyramid to obtain a set of high-frequency components h and a low-frequency layer image. The resolution of the components in h gradually decreases, starting from h×w and reaching the coarsest horizontal layer , while the low-frequency layer image is used Pixel representation. This decomposition is fully reversible, allowing the original image to be reconstructed through successive operations.

[0037] Secondly, the low-frequency layer image can reflect the global properties of the image, including color and brightness. At the same time, other high-frequency components contain edge detail processing of the image. Therefore, the color is mainly represented by the low-frequency component. Then a deterministic color mapping network is introduced to process the low-frequency layer image and convert it into a low-frequency layer output image, thereby adjusting the color and brightness of the underwater image.

[0038] Thirdly, the high-frequency layer image mainly contains small structures and details of the image, and the high-frequency component contains the edge detail processing of the image. Accurately enhancing the high-frequency component is crucial to recovering small structures and fine details in the image. Due to the strong spatial correlation between different frequency components, in order to facilitate the reconstruction of high-frequency components, it is necessary to explore the inter-layer correlation in the Laplacian pyramid.

[0039] Therefore, an efficient and progressive upsampling strategy is adopted to apply a multi-block group dilation feed-forward network to each high-frequency layer, which uses features from lower layers to support refinement in the current processing layer. The mask of the high-frequency components of the L-1 layer is learned, and the mask is gradually expanded to refine the remaining high-frequency components according to the intrinsic characteristics of the Laplacian pyramid.

[0040] After the refinement operation of the multi-block group expansion feedforward network, the corresponding pixel-by-pixel mask can be generated. The pixel-by-pixel mask of each layer is updated through the multi-block group expansion feedforward network. At each level from n=L-1 to n=1, the high-frequency components of each layer are refined and adjusted through the pixel-by-pixel mask, and then a high-frequency component set HF is obtained, HF=[h L-1 ,…,h n+1 ,h n ,…,h2,h1].

[0041] Finally, by reconstructing the low-frequency layer output image and the high-frequency component set HF using the Laplacian pyramid, an enhanced underwater image J can be obtained. The enhanced underwater image can effectively correct color distortion, suppress noise introduction, and retain more image details, so that the underwater image has better visibility and higher recognition.

[0042] In another embodiment of the present invention, preferably, in step 2, the deterministic color mapping network is a convolutional neural network, and its conversion process comprises the following steps: Step 2.1, expand the low-frequency layer input image into a two-dimensional matrix of size (3, h×w); Step 2.2, use the projection matrix P(k, 3) to perform matrix multiplication with the two-dimensional matrix, so as to embed each pixel in the low-frequency layer input image into a k-dimensional vector space; Step 2.3, performing matrix multiplication operation on the embedded k-dimensional vector space and the color mapping matrix T(k, k) to adjust the color; In step 2.4, the projection matrix Q(3, k) is used to perform matrix multiplication with the adjusted vector space, the adjusted vector space is converted back to the RGB color space, and each pixel is reassembled into an image of the original size, thereby outputting a color-enhanced low-frequency layer output image.

[0043] Specifically, given a low-frequency layer input image, color correction and illumination compensation are first performed through a deterministic color mapping network. Convolutional neural network with deterministic color mapping is a technology used for image processing and image conversion. Its core idea is to use a trained convolutional neural network to achieve the mapping of input image color to target color. However, the combination of filters or LUTs with CNNs has obvious limitations in color mapping. The deterministic color mapping network proposed in the present invention is efficient enough to extract high-level features in images without relying on traditional filters or LUTs.

[0044] Among them, P and Q are both learnable matrices shared by all images in the prior art. P maps the input data from the RGB space (3 channels) to a high-dimensional embedding space through linear projection. Q maps the data from the high-dimensional embedding space back to the original RGB space through linear inverse projection. Moreover, the three matrices P, T, and Q can be applied multiple times to further refine the color adjustment. For example, the low-frequency layer output image output in the above step is used as the input image again, and the operation of step 2 is repeated, so as to make more refined multiple adjustments to the low-frequency layer image.

[0045] Secondly, the data can be ® operated before and after the deterministic color mapping network is converted. The ® in the technical solution framework diagram, that is, the reshape operation, refers to the operation of reshaping and dimensional transformation of tensors in the prior art. The ‌reshape operation‌ is an important function in PyTorch for changing the shape of tensors. It allows users to redefine the dimensions of tensors without changing the tensor data. PyTorch is an open source deep learning framework for machine learning and deep learning.

[0046] Again, given the low-frequency layer input image and the input underwater image of the input image, the input underwater image is used to generate an adaptive color mapping matrix T for deterministic color mapping. Preferably, in step 2.3, the color mapping matrix T(k, k) is obtained by the following method: the initial input underwater image is input into the encoder E, the encoder E extracts features of the input underwater image through the MobileNetV3 model, and then outputs the extracted image features as a color mapping matrix T of size (k×k), and then reshapes it into the form of T(k, k) through the ® operation to obtain the color mapping matrix T(k, k). Thus, T(k, k) can be used as a color transformation matrix to adjust the color of the low-frequency layer input image of the underwater image. Among them, MobileNetV3 is a lightweight network model proposed by the Google team in 2019.

[0047] In another embodiment of the present invention, preferably, in step 3, preferably, the dual convolution attention module specifically includes the following steps: extracting input features of the input image through two convolutional layers, and using the LeakyReLU activation function for nonlinear activation.

[0048] Specifically, preferably, the kernel size of the convolution layer of the dual convolution attention module is set to 3. The two convolution layers in this embodiment preferably use 3x3 convolution layers, and each convolution layer uses the LeakyReLU activation function for nonlinear activation, thereby enhancing the representation ability of the features.

[0049] Secondly, preferably, a channel attention layer is embedded between the two convolutional layers of the dual convolutional attention module to perform adaptive weight adjustment on the channel dimension. The channel attention layer is set as the SE attention module in the prior art to adaptively adjust the weight. The dual convolutional attention module extracts local features through the convolutional layer and combines the global information enhancement of the channel attention layer to effectively improve the model's ability to express input data.

[0050] In another embodiment of the present invention, preferably, in step 4, the refinement operation further includes using at least two group-expanded feedforward networks to form a plurality of group-expanded feedforward networks, and respectively performing refinement adjustment on the output features of the output image, specifically including the following steps: Step 4.1, using x group-expanded feedforward networks, cascading them sequentially, and adjusting the output features of the output image step by step, thereby gradually correcting the output features; Step 4.2, the corrected features fed to each group expansion feedforward network are also densely connected to the output of the last group expansion feedforward network and connected to form the total output features of the multi-block group expansion feedforward network; In step 4.3, a channel attention module is then used to adaptively adjust the weights of the channel dimension of the total output feature, thereby obtaining a pixel-by-pixel mask after the output feature is refined. Specifically, feature extraction is optimized by combining multiple small modules of a group-expanded feedforward network, which ensures a balance between computational efficiency and performance.

[0051] Specifically, the © connection refers to simple concatenation, which means concatenating (connecting in series) multiple tensors (usually the outputs of different network branches) in a certain dimension to form a new tensor.

[0052] In another embodiment of the present invention, preferably, in step 4, the group expansion feedforward network specifically includes the following steps: using two series-connected 3x3 convolutional layers, combined with the GELU activation function, to capture the local spatial features of the output features of the output image; simultaneously using two series-connected 1x1 convolutional layers, combined with the GELU activation function, to capture the local spatial features of the output features of the output image; finally, performing a bit-by-bit addition operation on the two sets of local spatial features, and performing a residual connection operation with the output features of the output image; subsequently, using an adaptive hybrid attention module to capture, and forming a captured 3x3 convolutional layer, and then performing a residual connection operation with the output features of the output image, and finally obtaining a pixel-by-pixel mask after the output features are refined.

[0053] Specifically, two 3x3 convolutional layers are connected in series and combined with the GELU activation function to capture rich local spatial features while keeping the amount of computation low. In addition, the introduction of the 1x1 convolutional layer facilitates cross-channel feature interaction, effectively reducing the number of parameters and making the model more lightweight. Then, through the residual link, the original features are retained without losing spatial information. The adaptive hybrid attention module is a feature module in the prior art that can express lightweight features and can more effectively capture the dependencies between channels.

[0054] Among them, residual connection refers to the operation of adding the input of a layer directly to the output of the layer in the neural network. This operation can help the neural network better learn the identity mapping, that is, pass the input directly to the output, thereby avoiding the problem of gradient disappearance and gradient explosion, accelerating the training of the model, and improving the accuracy of the model.

[0055] In another embodiment of the present invention, preferably, in step 4, the pixel-by-pixel mask of the low-frequency layer image of the Lth layer is obtained by using the following steps: using bilinear operation to upsample the low-frequency layer input image and the low-frequency layer output image respectively, thereby improving their resolution to match the resolution of the L-1 layer high-frequency layer image, and © connecting the low-frequency layer input image and the low-frequency layer output image after the improved resolution to form a form of [low-frequency layer input image, low-frequency layer output image]; feeding the connected [low-frequency layer input image, low-frequency layer output image] into the dual convolution attention module to obtain the enhanced input feature D L ; Then use the group expansion feedforward network to enhance the input feature D L Perform a refinement operation to obtain the pixel-by-pixel mask LF of the Lth low-frequency layer image L .

[0056] Specifically, since in step 5, when the output image of the L-1th layer is refined and updated, the pixel-by-pixel mask of the Lth layer is needed, and the Lth layer is a low-frequency layer image, in step 4, the low-frequency layer input image and the low-frequency layer output image can also be enhanced by the corresponding dual convolution attention module and refined by the group dilation feedforward network. In addition, since the pixel-by-pixel mask of the high-frequency component of the high-frequency layer output image of the first layer will not be used, the high-frequency component of the first layer can be refined without operation, thereby saving the amount of calculation.

[0057] In another embodiment of the present invention, preferably, in step 5, the input feature after the n-th layer input image enhancement is set to D n ; Set the pixel-by-pixel mask after the thinning operation of the n+1th layer output image to M n+1 ; After calculation, the high-frequency component h of the high-frequency layer output image of the nth layer is obtained n ; The specific calculation formula is: , in, is the matrix multiplication operation, It is a bit-by-bit addition operation.

[0058] Specifically, is the pixel-by-pixel mask after the refinement operation of the n+1th layer output image, D n is the input feature after the input image of the nth layer is enhanced using the double convolutional attention module. Finally, h n It can also be fine-tuned through an optional 1×1 lightweight convolution block and LeakyReLU activation function to obtain the high-frequency component of the output image of the nth high-frequency layer after fine-tuning. It is equivalent to using a 1×1 lightweight convolution layer to aggregate and compress the adjusted total output features, optimize feature extraction through the combination of multiple small modules, and ensure the balance between computational efficiency and performance by grouping and expanding the feedforward network.

[0059] Secondly, h n Upsample and use it as the new input of the multi-block group expansion feedforward network of the n-1th layer to update the enhanced input feature D of the n-1th layer n-1 Thus, through this step-by-step upsampling method, a set of high-frequency components HF is obtained, HF=[h L-1 ,…,h n+1 ,h n ,…,h2,h1]; to match the high-frequency components of the high-frequency layer output images from the 1st layer to the L-1th layer.

[0060] Again, n in the above calculation formula is the specific number of layers of the layered image in step 1. The calculation formula is illustrated as follows: h L-1 =D L-1 LF L +D L-1 , using the enhanced input features of the L-1th layered image, perform matrix multiplication and bit-by-bit addition operations with the pixel-by-pixel mask of the Lth layer low-frequency layer output image to obtain the high-frequency components of the L-1th layer; …, h n =D n M n+1 +D n , using the enhanced input features of the layered image of the nth layer, perform matrix multiplication and bit-by-bit addition operations with the pixel-by-pixel mask of the n+1th layer to obtain the high-frequency components of the nth layer; …, h2=D2 M3+D2, using the input features of the enhanced second layer of layered images, performs matrix multiplication and bit-by-bit addition operations with the pixel-by-pixel mask of the third layer to obtain the high-frequency components of the second layer; h1=D1 M2+D1 uses the input features of the enhanced first layer of layered images to perform matrix multiplication and bit-by-bit addition operations with the pixel-by-pixel mask of the second layer to obtain the high-frequency components of the first layer.

[0061] In addition, LF L The L-th layer low-frequency layer output image and low-frequency layer input image are enhanced by the dual convolution attention module, and then the pixel-by-pixel mask of the low-frequency component features after the refinement operation is performed using a multi-block group expansion feedforward network. The high-frequency component is further refined by introducing the conversion features of the low-frequency layer image, thereby achieving faithful reconstruction when manipulating the attributes of a specific field.

[0062] In summary, the output of the multi-block group-expanded feed-forward network is considered as a pixel-by-pixel mask of the high-frequency components of the L-1 layer. Figure 5 As shown, for image pairs in both domains, high-frequency components at the same level differ only slightly in global brightness. Therefore, the pixel-wise mask can be interpreted as a global adjustment, which is relatively easier to optimize than mixing frequency images.

[0063] Since the addition of unprocessed high-frequency components may cause the image to degrade again, in order to alleviate this degradation effect, we first enhance the high-frequency components of the L-1 layer through a double convolution attention module to obtain D L-1Then, by introducing the pixel-by-pixel mask LF of the low-frequency component features of the low-frequency layer image L The above calculation formula is used to calculate, so after the refinement operation, the high-frequency component h of the L-1 layer is obtained. L-1 .

[0064] Finally, an optional lightweight convolution block is used for fine-tuning to obtain the refined high-frequency component. It is upsampled and used as the new input of the multi-block grouped dilation feedforward network to update the mask. Through this step-by-step upsampling method, we obtain a set of masks [M L−2 ,…,M n ,…,M2,M1] to match the remaining high-frequency components.

[0065] The same operation is used to refine all high-frequency components, and the result set is obtained together with the low-frequency layer output image to reconstruct image J.

[0066] Finally, the enhanced underwater image J is obtained by applying Laplacian pyramid reconstruction to the refined components. It is worth noting that the dual convolutional attention module and the multi-block grouped dilated feed-forward network module in the L-1 layer do not share parameters with other layers.

[0067] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

Claims

1. A method for dynamic enhancement of underwater images based on a pyramid network, characterized in that: The steps include: Step 1: First, the initial input underwater image is decomposed into a layered image group with a total of L layers using a Laplacian pyramid network, where the Lth layer is a low-frequency layer input image, and the 1st to L-1th layers are a group of high-frequency layer input image groups; Step 2, using a deterministic color mapping network, color correction and illumination compensation are performed on the low-frequency layer input image to convert it into a low-frequency layer output image; Step 3: Use the dual convolution attention module to extract and enhance the input features of the input image of the nth layer to obtain the enhanced input features D of the input image. n ; Step 4: Use the grouped dilation feedforward network to perform a refinement operation on the output image of the n+1th layer to obtain the pixel-by-pixel mask M after the output image refinement operation. n+1 ; Step 5: perform an operation on the input features of the nth layer input image after enhancement in step 3 and the pixel-by-pixel mask of the n+1th layer output image after thinning in step 4 to obtain the high-frequency component h of the high-frequency layer output image of the nth layer. n ; Step 6: Repeat steps 3, 4 and 5 for L-1 times to obtain a set of high-frequency components HF, HF=[h L-1 ,…,h n+1 ,h n ,…,h2,h1]; Step 7: Use the Laplacian pyramid network to reconstruct the low-frequency layer output image and the high-frequency component set HF to obtain an enhanced output underwater image.

2. The method for dynamic underwater image enhancement based on pyramid network according to claim 1, characterized in that: In step 2, the deterministic color mapping network is a convolutional neural network, and its conversion process includes the following steps: Step 2.1, expand the low-frequency layer input image into a two-dimensional matrix; Step 2.2, use the projection matrix P(k, 3) to perform matrix multiplication with the two-dimensional matrix, so as to embed each pixel in the low-frequency layer input image into a k-dimensional vector space; Step 2.3, performing matrix multiplication operation on the embedded k-dimensional vector space and the color mapping matrix T(k, k) to adjust the color; In step 2.4, the projection matrix Q(3, k) is used to perform matrix multiplication with the adjusted vector space, the adjusted vector space is converted back to the RGB color space, and each pixel is reassembled into an image of the original size, thereby outputting a color-enhanced low-frequency layer output image.

3. The method for dynamic underwater image enhancement based on pyramid network according to claim 1, characterized in that: In step 3, preferably, the dual convolution attention module specifically includes the following steps: extracting input features of the input image through two convolutional layers, and using the LeakyReLU activation function for nonlinear activation.

4. The method for dynamic underwater image enhancement based on pyramid network according to claim 1, characterized in that: In step 4, the refinement operation further includes using at least two group-expanded feedforward networks to form multiple group-expanded feedforward networks, and respectively performing refinement adjustment on the output features of the output image, specifically including the following steps: Step 4.1, using multiple block grouping expansion feedforward networks, cascading them in sequence, and adjusting the output features of the output image step by step, so as to gradually correct the output features; In step 4.2, the corrected features fed to each group expansion feedforward network are also densely connected to the output of the last group expansion feedforward network and connected to form the total output features of the multi-block group expansion feedforward network; Step 4.3, then use the channel attention module to adaptively adjust the weights of the channel dimension of the total output feature to obtain the pixel-by-pixel mask after the refinement operation on the output feature.

5. The method for dynamic underwater image enhancement based on pyramid network according to claim 1, characterized in that: In step 4, the group expansion feedforward network specifically includes the following method: Use two 3x3 convolutional layers in series and combine them with the GELU activation function to capture the local spatial features of the output features of the output image; At the same time, two 1x1 convolutional layers are used in series, combined with the GELU activation function, to capture the local spatial features of the output features of the output image; The two sets of local spatial features are added bit by bit, and the result is then residually connected with the output features of the output image; An adaptive hybrid attention module is used to capture and form a 3x3 convolutional layer after capture, which is then residually connected with the output features of the output image to finally obtain a pixel-by-pixel mask after the output features are refined.

6. The method for dynamic underwater image enhancement based on pyramid network according to claim 1, characterized in that: In step 5, the enhanced input feature of the nth layer input image is set to D n ; Set the pixel-by-pixel mask after the thinning operation of the n+1th layer output image to M n+1 ; After calculation, the high-frequency component h of the high-frequency layer output image of the nth layer is obtained n ; The specific calculation formula is: ， in, is the matrix multiplication operation, It is a bit-by-bit addition operation.

7. The method for dynamic underwater image enhancement based on pyramid network according to claim 1, characterized in that: In step 4, the pixel-by-pixel mask of the low-frequency layer image of layer L is obtained by the following steps: Use bilinear operation to upsample the low-frequency layer input image and the low-frequency layer output image respectively to improve their resolution, and © connect the low-frequency layer input image and the low-frequency layer output image after the improved resolution to form [low-frequency layer input image, low-frequency layer output image]; The connected [low-frequency layer input image, low-frequency layer output image] is fed into the double convolution attention module to obtain the enhanced input feature D L ; Use the group-expanded feed-forward network to enhance the input feature D L Perform a refinement operation to obtain the pixel-by-pixel mask LF of the Lth low-frequency layer image L .

8. An underwater image dynamic enhancement system based on a pyramid network, characterized in that: The system comprises: A decomposition module is used to decompose the initial input underwater image into a layered image group with a total of L layers using a Laplacian pyramid network, wherein the Lth layer is a low-frequency layer input image, and the 1st to L-1th layers are a group of high-frequency layer input image groups; A conversion module, used to perform color correction and illumination compensation on the low-frequency layer input image using a deterministic color mapping network to convert it into a low-frequency layer output image; An enhancement module is used to extract and enhance input features of the input image of the nth layer using a dual convolutional attention module to obtain enhanced input features of the input image; A refinement module, used to perform a refinement operation on the output features of the output image of the n+1th layer using a group dilation feedforward network, and obtain a pixel-by-pixel mask of the output image after the refinement operation; The operation module is used to perform operation on the enhanced input features and the pixel-by-pixel mask after the refinement operation to obtain the high-frequency component of the high-frequency layer output image of the nth layer; The repetition module is used to repeat the enhancement module, refinement module and operation module. After repeating L-1 times, a set of high-frequency component sets HF is obtained, HF=[h L-1 ,…,h n+1 ,h n ,…,h2,h1]; The reconstruction module is used to reconstruct the low-frequency layer output image and the high-frequency component set HF using a Laplacian pyramid network to obtain an enhanced output underwater image.

9. A device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program and implement a method for dynamic enhancement of underwater images based on a pyramid network as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer processor, the method for dynamic enhancement of underwater images based on a pyramid network as described in any one of claims 1 to 7 is implemented.

Images