Underwater image enhancement method and system based on regional gradient guided diffusion model

Through the regional gradient-guided diffusion model and the hybrid Swin-ConvNeXt network, combined with the multi-objective loss function, the problems of local details and global consistency in underwater image enhancement are solved, adaptability and computing efficiency are improved, and the underwater image is significantly improved. Visual quality.

CN119067874APending Publication Date: 2024-12-03SUN YAT SEN UNIV
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202411083522.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing underwater image enhancement methods are difficult to maintain local details and global consistency at the same time, lack adaptability, lack effective utilization of image semantic information, and low computational efficiency.

Method used

Using a diffusion model based on region gradient guidance, the image processing network of the adaptive denoising intensity adjustment and fused with the Swin Transformer block and the ConvNeXt block is improved by combining the multi-objective loss function.

Benefits of technology

It significantly improves the visual quality of underwater images, maintains local details and ensures global consistency, and improves adaptability and computing efficiency to different underwater environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067874A_ABST
    Figure CN119067874A_ABST
Patent Text Reader

Abstract

The invention discloses an underwater image enhancement method and system based on a regional gradient guided diffusion model. The method comprises the following steps: acquiring a to-be-enhanced original underwater image; performing adaptive denoising intensity adjustment according to the gradient information of the original underwater image, and performing denoising processing on the original underwater image to obtain a first underwater image; according to an image processing network in which a Swin Transform block and a ConvNeXt block are fused, the first underwater image is processed, and a second underwater image is obtained; and performing multi-target loss calculation on the second underwater image according to a multi-target loss function, and outputting an enhanced final underwater image. The visual quality of the underwater image is remarkably improved, and the method can be widely applied to the technical field of underwater image processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of underwater image processing, and in particular to an underwater image enhancement method and system based on a regional gradient guided diffusion model. Background Art

[0002] Underwater image enhancement is a key technology in the fields of marine science, underwater robots and marine engineering. Due to the complexity of the underwater environment, such as light scattering and absorption, suspended particles, turbidity changes and other factors, underwater images usually have problems such as color distortion, low contrast, and blurred details. These problems seriously affect the visual quality of underwater images and subsequent analysis tasks, such as target detection and scene understanding. Therefore, the development of effective underwater image enhancement technology is of great significance to promote the development of marine science and technology.

[0003] Existing underwater image enhancement methods can be mainly divided into three categories: methods based on physical models, methods based on image processing, and methods based on deep learning. Although existing technologies have made some progress in underwater image enhancement, there are still the following problems: ① It is difficult to maintain local details and global consistency at the same time; ② Insufficient adaptability to different underwater environments; ③ Lack of effective use of image semantic information; ④ The computational efficiency in practical applications needs to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present invention is to propose an underwater image enhancement method and system based on a regional gradient guided diffusion model to significantly improve the visual quality of underwater images.

[0005] To achieve the above object, an embodiment of the present invention provides an underwater image enhancement method based on a regional gradient guided diffusion model, comprising the following steps:

[0006] Acquire an original underwater image to be enhanced;

[0007] Adaptively adjusting the denoising intensity according to the gradient information of the original underwater image, and performing denoising processing on the original underwater image to obtain a first underwater image;

[0008] Processing the first underwater image according to an image processing network that incorporates a Swin Transformer block and a ConvNeXt block to obtain a second underwater image;

[0009] A multi-objective loss calculation is performed on the second underwater image according to the multi-objective loss function, and an enhanced final underwater image is output.

[0010] In some embodiments, the step of adaptively adjusting the denoising intensity according to the gradient information of the original underwater image and denoising the original underwater image to obtain the first underwater image comprises the following steps:

[0011] Calculating gradient information of the original underwater image and the target image to obtain a difference map;

[0012] Performing histogram equalization processing on the difference map to enhance the contrast of gradient information of the difference map;

[0013] Dividing the difference map after equalization processing into four regions, wherein each region corresponds to a different gradient strength;

[0014] Dynamically calculating weights based on the range of the difference map for each region;

[0015] The original noise distribution is adjusted according to the weights, and denoising processing is performed on the original underwater image to obtain a first underwater image.

[0016] In some embodiments, the calculation process of dynamically calculating the weight according to the range of the difference map of each area is:

[0017] m grad ={

[0018] 1-(0-1.5)×δ, when di ff∈[0,0.25)

[0019] 1-(1-1.5)×δ, when di ff∈[0.25,0.5)

[0020] 1-(2-1.5)×δ, when di ff∈[0.5,0.75)

[0021] 1-(3-1.5)×δ, when di ff∈[0.75,1]

[0022] };

[0023] Among them, m grad represents the weight; diff represents the difference map;

[0024] The expression for adjusting the original noise distribution according to the weight is:

[0025] p noiserggm =p noise ×(1+m grad -μ(m grad ))+x0×(μ(m grad )-m grad )

[0026] Among them, p noiserggmis the noise after adjusting the distribution; p noise is the original noise, μ(·) represents the averaging operation, and x0 is the original underwater image.

[0027] In some embodiments, the image processing network includes:

[0028] The Swin Transformer block processes the input feature map according to the sliding window self-attention mechanism and the multi-layer perceptron block, and outputs global context-aware features;

[0029] ConvNeXt block, used to perform depth-wise separable convolution and point-wise convolution on the input feature map according to the GELU activation function, and output local fine features;

[0030] A feature fusion module, used for performing a convolution operation on the global context-aware feature and the local fine feature, and outputting a fused feature;

[0031] The residual connection block is used to perform residual connection processing on the input feature map and the fusion feature to output the final image feature.

[0032] In some embodiments, performing multi-objective loss calculation on the second underwater image according to the multi-objective loss function comprises the following steps:

[0033] performing mean square error loss calculation on the second underwater image;

[0034] Performing gradient loss calculation on the second underwater image;

[0035] Performing color histogram loss calculation on the second underwater image;

[0036] The total loss is calculated based on the result of the mean square error loss calculation, the result of the gradient loss calculation, and the result of the color histogram loss calculation.

[0037] In some embodiments, the calculation formula of the mean square error loss is:

[0038]

[0039] Among them, L MSE represents the mean square error loss; N represents the total number of pixels in the image; x i Represents the value of the i-th pixel in the original image; Represents the value of the i-th pixel in the enhanced image;

[0040] The calculation formula of the gradient loss is:

[0041]

[0042] Among them, L GV represents the gradient loss; σ 2 represents variance; G x represents the gradient operator in the x direction; G y Represents the gradient operator in the y direction;

[0043] The calculation formula of the color histogram loss is:

[0044]

[0045] Among them, L CH represents the color histogram loss; d represents the histogram distance metric; H represents the color histogram calculation.

[0046] Another aspect of the embodiment of the present invention further provides an underwater image enhancement system based on a regional gradient guided diffusion model, comprising:

[0047] The first module is used to obtain the original underwater image to be enhanced;

[0048] The second module is used to perform adaptive denoising intensity adjustment according to the gradient information of the original underwater image, perform denoising processing on the original underwater image, and obtain a first underwater image;

[0049] A third module is used to process the first underwater image according to an image processing network that integrates a Swin Transformer block and a ConvNeXt block to obtain a second underwater image;

[0050] The fourth module is used to perform multi-objective loss calculation on the second underwater image according to the multi-objective loss function, and output an enhanced final underwater image.

[0051] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0052] The memory is used to store programs;

[0053] The processor executes the program to implement the method described above.

[0054] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0055] Another aspect of an embodiment of the present invention further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0056] The embodiments of the present invention include at least the following beneficial effects: the present invention provides an underwater image enhancement method and system based on a regional gradient guided diffusion model, the present invention obtains an original underwater image to be enhanced; performs adaptive denoising intensity adjustment according to the gradient information of the original underwater image, performs denoising processing on the original underwater image, and obtains a first underwater image; processes the first underwater image according to an image processing network that integrates a Swin Transformer block and a ConvNeXt block to obtain a second underwater image; performs multi-objective loss calculation on the second underwater image according to a multi-objective loss function, and outputs an enhanced final underwater image. The present invention significantly improves the visual quality of underwater images. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of the overall steps provided by an embodiment of the present invention;

[0058] Figure 2 It is a schematic diagram of the overall architecture provided by an embodiment of the present invention;

[0059] Figure 3 is a flow chart of a regional gradient guidance mechanism provided by an embodiment of the present invention;

[0060] Figure 4 is a schematic diagram of a hybrid Swin-ConvNeXt network structure provided by an embodiment of the present invention;

[0061] Figure 5 It is a detailed schematic diagram of the composition of the hybrid Swin-ConvNeXt architecture provided by an embodiment of the present invention;

[0062] Figure 6 is a comparison diagram of underwater image enhancement effects provided by an embodiment of the present invention;

[0063] Figure 7 is a comparison diagram of SI FT feature matching effects provided by an embodiment of the present invention;

[0064] Figure 8 It is a performance indicator comparison chart provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0066] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0067] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0069] Before describing the embodiments of the present invention in detail, some related technologies involved in the embodiments of the present invention are first described as follows:

[0070] RGGDM (Region Gradient-Guided Diffus ion Model): Regional gradient guided diffusion model, the core method proposed in this invention.

[0071] DDPM (Denoising Diffusion Probabilistic Models): Denoising diffusion probability model, a generative model based on Markov chain.

[0072] SCUnetV2: The hybrid Swin-ConvNeXt network structure proposed in this invention.

[0073] Swin Transformer: A sliding window based hierarchical Transformer architecture.

[0074] ConvNeXt: An improved convolutional neural network architecture that combines the design concept of Transformer.

[0075] UIQM (Underwater Image Quality Measure): Underwater image quality assessment indicator.

[0076] PSNR (Peak Signa l-to-Noi se Ratio): Peak signal-to-noise ratio, an indicator used to evaluate image quality.

[0077] SSIM: Structural Similarity, a metric used to evaluate image quality.

[0078] SI FT (Scale-Invariant Feature Transform): Scale-invariant feature transformation, a feature description algorithm in computer vision.

[0079] The underwater image enhancement method and system based on the regional gradient guided diffusion model provided by the embodiment of the present invention relate to the technical field of underwater image processing. The underwater image enhancement method based on the regional gradient guided diffusion model provided by the embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements an underwater image enhancement method based on a regional gradient guided diffusion model, etc., but is not limited to the above forms.

[0080] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0081] Reference Figure 1 , Figure 1 A flowchart of an underwater image enhancement method based on a regional gradient guided diffusion model applied to a server or server provided in an embodiment of the present invention, the method may include the following steps:

[0082] Acquire an original underwater image to be enhanced;

[0083] Adaptively adjusting the denoising intensity according to the gradient information of the original underwater image, and performing denoising processing on the original underwater image to obtain a first underwater image;

[0084] Processing the first underwater image according to an image processing network that incorporates a Swin Transformer block and a ConvNeXt block to obtain a second underwater image;

[0085] A multi-objective loss calculation is performed on the second underwater image according to the multi-objective loss function, and an enhanced final underwater image is output.

[0086] In some embodiments, the step of adaptively adjusting the denoising intensity according to the gradient information of the original underwater image and denoising the original underwater image to obtain the first underwater image comprises the following steps:

[0087] Calculating gradient information of the original underwater image and the target image to obtain a difference map;

[0088] Performing histogram equalization processing on the difference map to enhance the contrast of gradient information of the difference map;

[0089] Dividing the difference map after equalization processing into four regions, wherein each region corresponds to a different gradient strength;

[0090] Dynamically calculating weights based on the range of the difference map for each region;

[0091] The original noise distribution is adjusted according to the weights, and denoising processing is performed on the original underwater image to obtain a first underwater image.

[0092] In some embodiments, the calculation process of dynamically calculating the weight according to the range of the difference map of each area is:

[0093] m grad ={

[0094] 1-(0-1.5)×δ, when di ff∈[0,0.25)

[0095] 1-(1-1.5)×δ, when di ff∈[0.25,0.5)

[0096] 1-(2-1.5)×δ, when di ff∈[0.5,0.75)

[0097] 1-(3-1.5)×δ, when di ff∈[0.75,1]

[0098] };

[0099] Among them, m grad represents the weight; diff represents the difference map;

[0100] The expression for adjusting the original noise distribution according to the weight is:

[0101] p noiserggm =p noise ×(1+m grad -μ(m grad ))+x0×(μ(m grad )-m grad )

[0102] Among them, p noiserggm is the noise after adjusting the distribution; p noise is the original noise, μ(·) represents the average operation, and x0 is the original underwater image

[0103] In some embodiments, the image processing network includes:

[0104] The Swin Transformer block processes the input feature map according to the sliding window self-attention mechanism and the multi-layer perceptron block, and outputs global context-aware features;

[0105] ConvNeXt block, used to perform depth-wise separable convolution and point-wise convolution on the input feature map according to the GELU activation function, and output local fine features;

[0106] A feature fusion module, used for performing a convolution operation on the global context-aware feature and the local fine feature, and outputting a fused feature;

[0107] The residual connection block is used to perform residual connection processing on the input feature map and the fusion feature to output the final image feature.

[0108] In some embodiments, performing multi-objective loss calculation on the second underwater image according to the multi-objective loss function comprises the following steps:

[0109] performing mean square error loss calculation on the second underwater image;

[0110] Performing gradient loss calculation on the second underwater image;

[0111] Performing color histogram loss calculation on the second underwater image;

[0112] The total loss is calculated based on the result of the mean square error loss calculation, the result of the gradient loss calculation, and the result of the color histogram loss calculation.

[0113] In some embodiments, the calculation formula of the mean square error loss is:

[0114]

[0115] Among them, L MSE represents the mean square error loss; N represents the total number of pixels in the image; x i Represents the value of the i-th pixel in the original image; Represents the value of the i-th pixel in the enhanced image;

[0116] The calculation formula of the gradient loss is:

[0117]

[0118] Among them, L GV represents the gradient loss; σ 2 represents variance; G x represents the gradient operator in the x direction; G y Represents the gradient operator in the y direction;

[0119] The calculation formula of the color histogram loss is:

[0120]

[0121] Among them, L CH represents the color histogram loss; d represents the histogram distance metric; H represents the color histogram calculation.

[0122] The following describes the specific implementation process of the present invention in detail by taking the implementation process in a specific application scenario as an example:

[0123] like Figure 2 As shown, the present invention proposes an underwater image enhancement method and system based on regional gradient guidance, called regional gradient guided diffusion model (RGGDM). The method mainly includes the following key components: regional gradient guidance mechanism, hybrid Swin-ConvNeXt network structure (SCUnetV2), multi-objective loss function design and overall training and reasoning process.

[0124] Specifically, the method of the present invention may include the following 5 parts:

[0125] ① Input module: receives the underwater image to be enhanced.

[0126] ② Regional gradient guidance module: generates gradient mask to guide the denoising process.

[0127] ③SCUnetV2 network: the core image processing network, combining Swin Transformer and ConvNeXt structures.

[0128] ④Multi-objective loss calculation module: calculates and combines multiple loss functions.

[0129] ⑤Output module: output enhanced underwater images.

[0130] The regional gradient guidance mechanism of the present invention is described in detail below:

[0131] refer to Figure 3 The purpose of the regional gradient guidance mechanism is to adaptively adjust the denoising strength by utilizing the gradient information of the image, thereby achieving global consistency while maintaining local details. The regional gradient guidance mechanism includes the following implementation steps:

[0132] 1. Gradient calculation: Calculate the gradient of the input image x0 and the target image y to obtain the difference map diff.

[0133] 2. Histogram equalization: Perform histogram equalization on diff to enhance the contrast of gradient information.

[0134] 3. Region segmentation: Divide the equalized diff image into four regions, each region corresponding to a different gradient intensity.

[0135] 4. Weight calculation: Introduce a learnable parameter δ, with an initial value of 0.1 and a range of [-0.2, 0.2]. Dynamically calculate the weight m based on the range of diff grad :

[0136] m grad ={

[0137] 1-(0-1.5)×δ, when di ff∈[0,0.25)

[0138] 1-(1-1.5)×δ, when di ff∈[0.25,0.5)

[0139] 1-(2-1.5)×δ, when di ff∈[0.5,0.75)

[0140] 1-(3-1.5)×δ, when di ff∈[0.75,1]

[0141] }

[0142] 5. Noise adjustment: Use the calculated m grad Adjust the original noise distribution:

[0143] p noiserggm =p noise ×(1+m grad -μ(m grad ))+x0×(μ(m grad )-m grad )

[0144] Among them, p noise is the original noise, μ(·) represents the averaging operation, and x0 is the original input image.

[0145] The specific contents of the hybrid Swin-ConvNeXt network structure of the present invention are described below:

[0146] SCUnetV2 is a new hybrid network structure that combines the global modeling capabilities of Swin Transformer and the local feature extraction advantages of ConvNeXt. Figure 4 As shown, Figure 4 It is a schematic diagram of the hybrid Swin-ConvNeXt network structure; Figure 5 This is a detailed diagram of the hybrid Swin-ConvNeXt architecture.

[0147] refer to Figure 4 and Figure 5 The main components of SCUnetV2 of the embodiment of the present invention include:

[0148] 1. Swin Transformer block:

[0149] Input: Feature map F in ∈R^(H×W×C);

[0150] It uses sliding window self-attention mechanism and multi-layer perceptron (MLP) blocks for processing;

[0151] Output: Global context-aware feature F swin ∈R^(H×W×C').

[0152] 2. ConvNeXt block:

[0153] Input: Feature map F in ∈R^(H×W×C);

[0154] The GELU activation function, depth-wise separable convolution and point-wise convolution are used for processing;

[0155] Output: local fine feature F convnext ∈R^(H×W×C”).

[0156] 3. Feature fusion module:

[0157] Input: F swin and F convnext ;

[0158] The 1×1 convolution operation is used for processing;

[0159] Output: fusion feature F fused ∈R^(H×W×C”').

[0160] 4. Residual Connection:

[0161] F out =Ff used +F in

[0162] In summary, the forward propagation process of SCUnetV2 can be expressed as:

[0163] F swin =SwinTransformer(F in )

[0164] F convnext =ConvNeXt(F in )

[0165] F fused =Conv1x1([F swin ,F convnext ])

[0166] F out =F fused +F in

[0167] The contents of the multi-objective loss function of the embodiment of the present invention are described in detail below:

[0168] In order to comprehensively improve the quality of enhanced images, the present invention designs a multi-objective loss function that combines mean square error loss, gradient loss and color histogram loss.

[0169] The calculation process of the multi-objective loss function is as follows:

[0170] 1. Mean Square Error Loss (MSE Loss):

[0171]

[0172] 2. Gradient Loss

[0173]

[0174] Among them, G_x and G_y represent the gradient operators in the x-direction and y-direction respectively, and σ^2 represents the variance.

[0175] 3. Color histogram loss (Color Histogram Loss):

[0176]

[0177] Among them, H(·) represents the color histogram calculation, and d(·,·) represents the histogram distance metric.

[0178] 4. Total loss:

[0179] L ALL =λ CH ×L CH +λ MSE ×L MSE +λ GV ×L GV

[0180] Among them, λ MSE , GV , CH is the weight coefficient of each loss item.

[0181] In summary, the method of the present invention can be widely used in underwater robot vision systems, marine environment monitoring equipment, underwater archaeology and cultural relics protection equipment, marine resource exploration equipment, aquaculture monitoring systems, marine scientific research equipment, underwater photography and film production equipment, marine education and popular science equipment, port and ship safety detection equipment, marine meteorological observation equipment and other fields.

[0182] The present invention has carried out experimental verification of relevant samples based on the proposed underwater image enhancement method, such as Figure 6 , Figure 7 and Figure 8 As shown, Figure 6 The enhancement effects of RGGDM and other representative methods in different underwater scenes are compared. It can be observed that RGGDM effectively improves the overall color and contrast while maintaining local details.

[0183] Figure 7 The comparison of the number of SI FT feature matching points of the original image and the image enhanced by RGGDM is shown. The enhanced image has more matching points, indicating that RGGDM improves the feature identifiability of the image.

[0184] Figure 8 The comparison results between RGGDM and other methods on evaluation indicators such as PSNR, SSIM, and UIQM are shown in the form of bar graphs.

[0185] From the above detailed description, it can be seen that RGGDM is an end-to-end underwater image enhancement framework, which achieves a significant improvement in underwater image quality through an innovative regional gradient guidance mechanism, an efficient hybrid network structure and a comprehensive loss function design. This method not only performs well in subjective visual effects, but also shows obvious advantages in objective evaluation indicators and downstream tasks.

[0186] The proposed underwater image enhancement method based on regional gradient guidance (RGGDM) has significant advantages over the existing best technologies, such as UI E-DANet based on deep learning and UW-DDPM based on diffusion model. First, RGGDM innovatively introduces the regional gradient guidance mechanism, which dynamically adjusts the denoising strength of different regions through learnable parameters, effectively solving the trade-off problem between local detail preservation and global consistency in existing methods. This enables RGGDM to ensure the natural harmony of the overall visual effect while retaining important texture and edge information, greatly improving the quality and usability of enhanced images.

[0187] Secondly, RGGDM adopts a unique hybrid Swin-ConvNeXt network structure (SCUnetV2), which cleverly combines the global modeling capabilities of Transformer and the local feature extraction advantages of CNN. This innovative network design enables the model to better adapt to different types of underwater environments and image degradation patterns, significantly improving the adaptability and robustness of the method in complex and changeable actual underwater scenes. In addition, the multi-objective loss function design introduced by RGGDM comprehensively considers pixel reconstruction, structure preservation and color restoration, further improving the enhancement effect. RGGDM outperforms the existing best methods in both objective evaluation indicators (such as PSNR, SSIM, UIQM) and subjective visual effects. In particular, in downstream tasks such as SI FT feature matching, the average number of matching points in images processed by RGGDM increased by 132.19%, highlighting its excellent performance in practical applications.

[0188] Another aspect of the embodiment of the present invention further provides an underwater image enhancement system based on a regional gradient guided diffusion model, comprising:

[0189] The first module is used to obtain the original underwater image to be enhanced;

[0190] The second module is used to perform adaptive denoising intensity adjustment according to the gradient information of the original underwater image, perform denoising processing on the original underwater image, and obtain a first underwater image;

[0191] A third module is used to process the first underwater image according to an image processing network that integrates a Swin Transformer block and a ConvNeXt block to obtain a second underwater image;

[0192] The fourth module is used to perform multi-objective loss calculation on the second underwater image according to the multi-objective loss function, and output an enhanced final underwater image.

[0193] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0194] The embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned underwater image enhancement method based on the regional gradient guided diffusion model when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0195] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0196] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the underwater image enhancement method based on the regional gradient guided diffusion model.

[0197] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0198] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0199] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0200] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0201] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0202] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0203] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0204] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0205] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0206] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0207] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0208] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0209] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. An underwater image enhancement method based on a regional gradient guided diffusion model, characterized in that: The following steps are involved: Acquire an original underwater image to be enhanced; Adaptively adjusting the denoising intensity according to the gradient information of the original underwater image, and performing denoising processing on the original underwater image to obtain a first underwater image; Processing the first underwater image according to an image processing network that incorporates a Swin Transformer block and a ConvNeXt block to obtain a second underwater image; A multi-objective loss calculation is performed on the second underwater image according to the multi-objective loss function, and an enhanced final underwater image is output.

2. The underwater image enhancement method based on regional gradient guided diffusion model according to claim 1, characterized in that: The step of adaptively adjusting the denoising intensity according to the gradient information of the original underwater image and denoising the original underwater image to obtain a first underwater image comprises the following steps: Calculating gradient information of the original underwater image and the target image to obtain a difference map; Performing histogram equalization processing on the difference map to enhance the contrast of gradient information of the difference map; Dividing the difference map after equalization processing into four regions, wherein each region corresponds to a different gradient strength; Dynamically calculating weights based on the range of the difference map for each region; The original noise distribution is adjusted according to the weights, and denoising processing is performed on the original underwater image to obtain a first underwater image.

3. The underwater image enhancement method based on regional gradient guided diffusion model according to claim 2, characterized in that: The calculation process of dynamically calculating the weight according to the range of the difference map of each area is: m grad ={ 1-(0-1.5)×δ, when diff∈[0,0.25) 1-(1-1.5)×δ, when diff∈[0.25,0.5) 1-(2-1.5)×δ, when diff∈[0.5,0.75) 1-(3-1.5)×δ, when diff∈[0.75,1] }; Among them, m grad represents the weight; diff represents the difference map; The expression for adjusting the original noise distribution according to the weight is: p noiserggm =p noise ×(1+m grad -μ(m grad ))+x0×(μ(m grad )-m grad ) Among them, p noiserggm is the noise after adjusting the distribution; p noise is the original noise, μ(·) represents the averaging operation, and x0 is the original underwater image.

4. The underwater image enhancement method based on regional gradient guided diffusion model according to claim 1, characterized in that: The image processing network comprises: The Swin Transformer block processes the input feature map according to the sliding window self-attention mechanism and the multi-layer perceptron block, and outputs global context-aware features; ConvNeXt block, used to perform depth-wise separable convolution and point-wise convolution on the input feature map according to the GELU activation function, and output local fine features; A feature fusion module, used for performing a convolution operation on the global context-aware feature and the local fine feature, and outputting a fused feature; The residual connection block is used to perform residual connection processing on the input feature map and the fusion feature to output the final image feature.

5. The underwater image enhancement method based on regional gradient guided diffusion model according to claim 1, characterized in that: The performing multi-objective loss calculation on the second underwater image according to the multi-objective loss function comprises the following steps: performing mean square error loss calculation on the second underwater image; Performing gradient loss calculation on the second underwater image; Performing color histogram loss calculation on the second underwater image; The total loss is calculated based on the result of the mean square error loss calculation, the result of the gradient loss calculation, and the result of the color histogram loss calculation.

6. The underwater image enhancement method based on regional gradient guided diffusion model according to claim 5, characterized in that: The calculation formula of the mean square error loss is: Among them, L MSE represents the mean square error loss; N represents the total number of pixels in the image; x i Represents the value of the i-th pixel in the original image; Represents the value of the i-th pixel in the enhanced image; The calculation formula of the gradient loss is: Among them, L GV represents the gradient loss; σ 2 represents variance; G x represents the gradient operator in the x direction; G y Represents the gradient operator in the y direction; The calculation formula of the color histogram loss is: Among them, L CH represents the color histogram loss; d represents the histogram distance metric; H represents the color histogram calculation.

7. An underwater image enhancement system based on a regional gradient guided diffusion model, characterized in that: include: The first module is used to obtain the original underwater image to be enhanced; The second module is used to perform adaptive denoising intensity adjustment according to the gradient information of the original underwater image, perform denoising processing on the original underwater image, and obtain a first underwater image; A third module is used to process the first underwater image according to an image processing network that integrates a Swin Transformer block and a ConvNeXt block to obtain a second underwater image; The fourth module is used to perform multi-objective loss calculation on the second underwater image according to the multi-objective loss function, and output an enhanced final underwater image.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Underwater image adaptive enhancement method, system and device and storage medium

    CN116912115A