An underwater scene sharpening method and device based on collaborative complementary deep learning and a storage medium

Abstract

The application provides an underwater scene clearification method based on collaborative complementary deep learning, comprising the following steps: acquiring an underwater scene dataset, preprocessing the dataset; constructing a collaborative complementary deep learning model; inputting the preprocessed dataset into the constructed collaborative complementary deep learning model, training the constructed collaborative complementary deep learning model, and saving the trained collaborative complementary deep learning model; inputting the underwater scene data to be processed into the trained collaborative complementary deep learning model to realize clearification of the underwater scene; the constructed collaborative complementary deep learning model structure combines the advantages of CNN and Transformer, extracts similar features of double inputs, complements each other, realizes a high-performance and strong-robustness model, and can effectively realize underwater scene clearification.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and relates to a method, device and storage medium for underwater scene sharpening based on collaborative complementary deep learning. Background Technology

[0002] Today, the ocean's value in terms of resources and strategy is becoming increasingly apparent. The ocean possesses abundant resources and enormous development potential. Underwater imaging technology, as a crucial foundation for human exploration of the underwater world, faces numerous challenges. The attenuation of different light waves in the underwater environment leads to color shifts in images. Simultaneously, factors such as particles, bubbles, and water bodies in the water cause image noise and distortion. These factors severely impact underwater activities such as marine archaeology, pipeline inspection, and resource exploration. Therefore, in order to solve problems such as color shifts and noise distortion in underwater images, underwater scene sharpening has increasingly become a research hotspot.

[0003] The goal of underwater scene sharpening technology is to enhance the visual quality of underwater images, facilitating smoother underwater activities. Underwater scene sharpening technology mainly consists of color enhancement and noise reduction. Traditional underwater scene sharpening techniques employ prior estimation or indirect pixel processing to achieve clarity. However, the complexity and diversity of the underwater environment (e.g., different water qualities, depths, lighting conditions) make traditional underwater scene sharpening techniques difficult to adapt to various water conditions, significantly limiting their effectiveness.

[0004] With the development of deep learning (DNN), researchers have introduced deep learning methods into the field of underwater image enhancement. Existing deep learning-based methods typically employ convolutional neural networks (CNN), generative adversarial networks (GAN), or Transformer models, each with its own advantages. These deep learning-based methods enhance images by learning from large amounts of data. Numerous studies have demonstrated that learning-based methods possess better robustness and practicality. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a technical solution: an underwater scene sharpening method based on collaborative complementary deep learning, comprising the following steps:

[0006] Acquire underwater scene dataset and preprocess the dataset;

[0007] Constructing collaborative and complementary deep learning models;

[0008] The preprocessed dataset is input into the constructed collaborative complementary deep learning model, the constructed collaborative complementary deep learning model is trained, and the trained collaborative complementary deep learning model is saved.

[0009] The underwater scene data to be processed is input into the trained collaborative and complementary deep learning model to achieve clarity processing of the underwater scene.

[0010] Furthermore: the collaborative complementary deep learning model includes a first initial part, a second initial part, a final part, a first feature-aware branch, a second feature-aware branch, and a feature collaborative complementary branch;

[0011] The first initial part and the second initial part are used to increase the number of input data channels from 3 to 32.

[0012] The first feature perception branch and the second feature perception branch are used to extract feature information based on the data after the boosting channel is output from the first initial part and the second initial part, respectively; the first initial part and the second initial part have the same structure.

[0013] The first feature perception branch and the second feature perception branch have the same structure;

[0014] The first feature perception branch includes n sequentially connected feature perception modules Xi, i = 1…n; sharing n sets of weights.

[0015] The feature collaborative complementary branch is used to mutually compensate the extracted features;

[0016] The feature synergy and complementarity branch includes n-1 feature synergy and complementarity modules M j j = 1…n-1; the feature cooperative complementarity module includes a feature cooperative module and a feature complementary module;

[0017] The feature complementarity module and the feature coordination module are connected;

[0018] The feature complementarity module M j The input is the feature perception module Xi of the first feature perception segment and the feature perception module Xi of the second feature perception segment;

[0019] The feature complementarity module M j The output is the feature perception module Xi+1 of the first feature perception segment and the feature perception module Xi+1 of the second feature perception segment;

[0020] The last part receives the mutually compensated features output by the feature cooperative complementary branch, which is used to reduce the number of output data channels from 32 to 3.

[0021] Furthermore: the first initial part is provided with a 3×3 convolutional block;

[0022] The end portion is provided with a 1×1 convolution block.

[0023] Furthermore: the feature perception module includes a 3×3 convolutional layer, and a multilayer perceptron (MLP) consisting of a 1×1 convolution, a GELU activation function, and a 1×1 convolutional layer. The expression of the multilayer perceptron (MLP) is as follows.

[0024] f i+1 =3×3Conv(MLP(f i ))=3×3Conv(1×1Conv(GELU(1×1Conv(f i ))))

[0025] Among them, f i f is the output feature of the i-th feature perception module. i+1 Let k×kConv represent the output features of the (i+1)th feature perception module, k×kConv represent the convolutional layer with kernel k, and GELU is the activation function.

[0026] Furthermore: the feature collaboration module is decomposed into three different features by layer normalization and 3×3 deep convolution: query Q, index K, and content V, and then fed into a multi-head collaborative attention and perceptron to obtain attention features A1 and A2. The obtained features are then fed into the feature complementation module, as shown in the following expression:

[0027]

[0028]

[0029] Among them, f1 i , For the two output features of the i-th feature perception module, Scaling factor The multi-head attention feature output by the i-th feature collaboration module is a depthwise convolution with a kernel size of k.

[0030] Furthermore: The feature complementarity module first adds the two outputs of the feature perception module and the feature collaboration module through residuals, then performs channel merging, performs layer normalization and 3×3 depth convolution decomposition into three different features: query Q, index K, and content V), and then feeds them into a multi-head complementary attention and perceptron. The resulting attention feature segments are then added to the feature perception module through residuals, as shown in the following expression:

[0031]

[0032] Where cat is the channel merging operation, split is the channel splitting operation, and W is the channel splitting operation. i Let be the multi-head attention feature of the i-th feature collaboration module.

[0033] A processing apparatus includes: one or more processors and a memory for storing one or more programs; when the programs are executed by the processors, the processing apparatus causes the processing apparatus to perform the method as described in any one of the above.

[0034] A computer-readable storage medium having a computer program stored thereon that, when executed, implements any of the methods described herein.

[0035] The underwater scene sharpening method based on collaborative complementary deep learning provided by this invention has the following advantages:

[0036] The constructed collaborative and complementary deep learning model structure combines the advantages of CNN and Transformer, extracts similar features of the two inputs, and complements them to achieve a high-performance and robust model that can effectively improve the clarity of underwater scenes.

[0037] The constructed feature-aware branch, employing a weight-sharing feature-aware module, can efficiently extract dual-input features;

[0038] The constructed collaborative complementary branch can extract similar features from the input and complement them through similar features.

[0039] The constructed feature collaboration module utilizes a multi-head collaborative attention mechanism to extract similar features from the input.

[0040] The constructed feature complementarity module utilizes a multi-head attention mechanism to complement similar features extracted by the feature perception module and the feature collaboration module. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a structural diagram of the collaborative complementary deep learning model of the present invention;

[0043] Figure 2 This is the feature sensing module of the present invention;

[0044] Figure 3 This is the feature collaboration module of the present invention;

[0045] Figure 4 This is a complementary feature module of the present invention;

[0046] Figure 5This is a result image of the color correction task applied by the present invention.

[0047] Figure 6 This is a diagram showing the results of the present invention used for noise reduction processing. Detailed Implementation

[0048] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] An underwater scene sharpening method based on collaborative complementary deep learning includes the following steps:

[0051] S1: Obtain the underwater scene dataset and preprocess the dataset;

[0052] S2: Construct a collaborative and complementary deep learning model;

[0053] S3: Input the preprocessed dataset into the constructed synergistic complementary deep learning model, train the constructed synergistic complementary deep learning model, and save the trained model;

[0054] S4: Input the underwater scene data to be processed into the collaborative complementary deep learning model to clarify the underwater scene.

[0055] The steps S1 / S2 / S3 / S4 are executed sequentially;

[0056] Furthermore, the preprocessing of the training set is as follows:

[0057] In the training dataset, data is randomly cropped according to a specified size and then randomly flipped. It should be noted that the specified size depends on hardware computing resources and the actual task.

[0058] Figure 1 This is a structural diagram of the collaborative complementary deep learning model of the present invention;

[0059] The synergistic and complementary deep learning model includes a first initial part, a second initial part, a final part, a first feature-aware branch, a second feature-aware branch, and a feature synergistic and complementary branch.

[0060] The first initial part and the second initial part are used to increase the number of input data channels from 3 to 32.

[0061] The first feature perception branch and the second feature perception branch are used to extract feature information based on the data after the boosting channel is output from the first initial part and the second initial part, respectively; the first initial part and the second initial part have the same structure.

[0062] The first feature perception branch and the second feature perception branch have the same structure;

[0063] The first feature perception branch includes n sequentially connected feature perception modules Xi, i = 1…n; sharing n sets of weights.

[0064] The feature collaborative complementary branch is used to mutually compensate the extracted features;

[0065] The feature synergy and complementarity branch includes n-1 feature synergy and complementarity modules M j j = 1…n-1; the feature synergy and complementarity module includes a feature synergy module and a feature complementarity module; that is, n-1 feature synergy modules and n-1 feature complementarity modules.

[0066] The feature complementarity module and the feature coordination module are connected;

[0067] The feature complementarity module M j The input is the feature perception module Xi of the first feature perception segment and the feature perception module Xi of the second feature perception segment;

[0068] The feature complementarity module M j The output is the feature perception module Xi+1 of the first feature perception segment and the feature perception module Xi+1 of the second feature perception segment;

[0069] The last part receives the mutually compensated features output by the feature cooperative complementary branch, which is used to reduce the number of output data channels from 32 to 3.

[0070] Furthermore: the first initial part is configured with a 3×3 convolutional block;

[0071] The end portion is provided with a 1×1 convolutional block;

[0072] The feature perception branch is a dual-input structure used to extract feature information. The output features of the feature perception module are respectively passed to the feature collaborative complementarity model and the next feature perception module. The two branches share n sets of weight-sharing perception modules.

[0073] Figure 2 This is the feature sensing module of the present invention;

[0074] The feature perception module comprises a multilayer perceptron (MLP) consisting of a 3×3 convolutional layer, a 1×1 convolutional layer, a GELU activation function, and a 1×1 convolutional layer.

[0075] f i+1 =3×3conv(MLP(f i ))=3×3conv(1×1conv(GELU(1×1conv(f i ))))

[0076] Among them, f i f is the output feature of the i-th feature perception module. i+1 Let k×kConv represent the output features of the (i+1)th feature perception module, k×kConv represent the convolutional layer with kernel k, and GELU is the activation function.

[0077] Figure 3 This is the feature collaboration module of the present invention;

[0078] The feature collaboration module decomposes the data into three different features (query Q, index K, and content V) using layer normalization and 3×3 deep convolution. These features are then fed into a multi-head collaborative attention and perceptron to obtain attention features A1 and A2. The resulting features are then fed into the feature complementation module, as shown in the following expression:

[0079]

[0080]

[0081] Among them, f1 i , For the two output features of the i-th feature perception module, Scaling factor The multi-head attention feature output by the i-th feature collaboration module; k×kDWConv is a depthwise convolution with kernel size k;

[0082] Figure 4 This is a complementary feature module of the present invention;

[0083] The feature complementation module first adds the two outputs of the feature perception module and the feature collaboration module through residuals, then performs channel merging, performs layer normalization and 3×3 deep convolution to decompose them into three different features (query Q, index K, and content V), and then feeds them into a multi-head complementary attention and perceptron. The resulting attention feature segments are then added to the feature perception module through residuals, as shown in the following expression:

[0084]

[0085] Where cat is the channel merging operation, split is the channel splitting operation, and W is the channel splitting operation. i The multi-head attention feature of the i-th feature collaboration module;

[0086] Underwater scene images typically exhibit similar degradation characteristics, such as color shift and decreased contrast. These common features provide complementary information that can be used to improve image enhancement. Simultaneously, varying degrees of distortion caused by factors such as suspended particles result in differences between images. These differences can be used to locate distorted areas and repair them using both global and common features to achieve underwater scene sharpening.

[0087] In the feature collaboration and complementarity branch, the feature collaboration module is used to perform cross-branch modeling on different branches, aiming to explore the commonalities and differences of features in different underwater images; the feature complementarity module is used to correct and repair common and different features.

[0088] In the feature collaboration module, the self-attention mechanism of Transformer is improved to increase the interaction between the two branches, so as to establish long-distance dependencies between different input features and capture the commonalities and differences between different branches.

[0089] In the feature complementation module, the Transformer is used to capture global features, supplement common features to better correct colors, restore differences to recover details and remove noise.

[0090] Figure 5 This is a result image of the color correction task applied by the present invention.

[0091] Furthermore, taking the color enhancement task as an example, the comparison metrics are shown in the table below:

[0092] Table 1

[0093]

[0094] As can be seen from Table 1, our method performs excellently in the color enhancement task.

[0095] Figure 6 This is a diagram showing the results of the present invention used for noise reduction processing.

[0096] Furthermore, taking the denoising task as an example, the comparison metrics are shown in the table below:

[0097] Table 2

[0098] method PSNR SSIM The method of this application 32.1752 0.93371 BM3D 29.5543 0.85653 NLM 29.5116 0.83565 DnCNN 27.8991 0.91693 IRCNN 28.5378 0.85001

[0099] As can be seen from Table 2, our method performs excellently in denoising tasks.

[0100] Further: Evaluate the actual performance of the method of the present invention. Figure 5 and Figure 6 The results of the present invention and the comparative method in color correction and noise reduction tasks are shown in the accompanying comparison images. It can be seen that the present method achieves better visual results in both tasks.

[0101] The present invention provides a processing device, characterized in that it includes: one or more processors and a memory for storing one or more programs; when the program is executed by the processor, the processing device enables the underwater scene sharpening method based on collaborative complementary deep learning.

[0102] This invention provides a computer-readable storage medium storing a computer program thereon, characterized in that the instructions included in the program code can be used to execute the steps of the underwater scene sharpening method based on collaborative complementary deep learning described in the above-described method embodiments. The aforementioned storage medium includes media capable of storing program code, such as USB flash drives, portable hard drives, optical discs, and random access memory (RAM).

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for underwater scene sharpening based on collaborative complementary deep learning, characterized in that: Includes the following steps: Acquire underwater scene dataset and preprocess the dataset; Constructing collaborative and complementary deep learning models; The preprocessed dataset is input into the constructed collaborative complementary deep learning model, the constructed collaborative complementary deep learning model is trained, and the trained collaborative complementary deep learning model is saved. The underwater scene data to be processed is input into the trained collaborative and complementary deep learning model to achieve the clarity processing of the underwater scene. The synergistic and complementary deep learning model includes a first initial part, a second initial part, a final part, a first feature-aware branch, a second feature-aware branch, and a feature synergistic and complementary branch. The first initial part and the second initial part are used to increase the number of input data channels from 3 to 32; The first feature perception branch and the second feature perception branch are used to extract feature information based on the data after the boosting channel is output from the first initial part and the second initial part, respectively; the first initial part and the second initial part have the same structure. The first feature perception branch and the second feature perception branch have the same structure; The first feature perception branch includes n sequentially connected feature perception modules Xi, i=1…n, which share n sets of weights. The feature collaborative complementary branch is used to mutually compensate the extracted features; The feature synergy and complementarity branch includes n-1 feature synergy and complementarity modules M j , j=1…n-1; the feature cooperative complementarity module includes a feature cooperative module and a feature complementary module; The feature complementarity module and the feature coordination module are connected; The feature complementarity module M j The input is the feature perception module Xi of the first feature perception segment and the feature perception module Xi of the second feature perception segment; The feature complementarity module M j The output is the feature perception module Xi+1 of the first feature perception segment and the feature perception module Xi+1 of the second feature perception segment; The end portion receives the mutually compensated features output by the feature cooperative complementary branch, which is used to reduce the number of output data channels from 32 to 3. The first initial part is configured with a 3×3 convolutional block; The end portion is provided with a 1×1 convolutional block; The feature perception module comprises a multilayer perceptron (MLP) consisting of 3×3 convolutional layers, 1×1 convolutional layers, a GELU activation function, and another 1×1 convolutional layer. The expression for the MLP is as follows: in, The output feature of the i-th feature perception module is... The output feature of the (i+1)th feature perception module This indicates a convolutional layer with kernel k, and GELU is the activation function; The feature collaboration module decomposes the query into three different features through layer normalization and 3×3 deep convolution: query Q, index K, and content V. These features are then fed into a multi-head collaborative attention and perceptron to obtain attention features A1 and A2. The resulting features are then sent to the feature complementation module, as shown in the following expression: in, , For the two output features of the i-th feature perception module, Scaling factor , Let be the multi-head attention feature output by the i-th feature collaboration module, and let be a depthwise convolution with kernel size k. The feature complementation module first adds the two outputs of the feature perception module and the feature collaboration module through residuals, then performs channel merging, performs layer normalization and 3×3 depth convolution to decompose them into three different features: query Q, index K, and content V. These features are then fed into a multi-head complementary attention and perceptron, and the resulting attention feature segments are added to the feature perception module through residuals, as shown in the following expression: in, cat For channel merging operations, split Channel splitting operation, W i Let be the multi-head attention feature of the i-th feature collaboration module.

2. A processing apparatus, characterized in that, include: One or more processors and a memory are used to store one or more programs; when the programs are executed by the processor, the processing device causes the processing device to perform the method as described in claim 1.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed, it implements the method as described in claim 1.

Images