A method and system for underwater image quality enhancement based on spectral super-resolution technology

Abstract

The present invention discloses a method and system for underwater image quality enhancement based on spectral super-resolution technology, the method comprising: obtaining a real image, a color RGB reference image and a hyperspectral reference image taken in an underwater environment; performing spectral super-resolution on the real image based on a spectral super-resolution neural network; compensating for degradation of the hyperspectral reconstructed image based on a color correction module; constructing a multi-task loss function; training and optimizing the spectral super-resolution neural network and the color correction module based on the multi-task loss function, and enhancing the real image to be enhanced to obtain a final color RGB enhanced image. The system comprises a data acquisition module, a spectral super-resolution neural network module, a color correction module, a loss function construction module, a training optimization module and an enhancement module. By using the present invention, comprehensive optimization and enhancement of underwater images can be achieved. The present invention can be widely applied to the technical field of underwater image quality enhancement.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater image quality enhancement, and in particular to an underwater image quality enhancement method and system based on spectral super-resolution technology. Background Art

[0002] With the continuous development of marine resources and the deepening of marine exploration, underwater image quality enhancement technology plays an increasingly important role in underwater detection, marine scientific research, underwater archaeology and other fields. However, the quality of underwater images is affected by many factors, such as water turbidity, light attenuation, noise interference, etc. These factors have caused the quality of underwater images to be seriously degraded and difficult to meet the needs of practical applications. Therefore, how to effectively improve the quality of underwater images has become an urgent problem to be solved.

[0003] The current mainstream underwater image enhancement technology relies on deep learning methods. By learning the mapping relationship between the original underwater image and the corresponding clear image, the image enhancement network is finally trained to improve the quality of underwater images. However, in view of the complexity and heterogeneity of the causes of underwater image color, most existing deep networks only use limited information between the original image and the generated clear reference image, and the process of image color enhancement fails to integrate the interpretability of the correction process. Taking the patent with publication number CN118071656A as an example, it discloses a "Vision Transformer-based underwater image enhancement method". Although this method uses a deep learning model to train data, in practical applications, it does not fully consider the unique properties of the underwater image data itself, such as the heterogeneity of color information, which may lead to certain limitations in the effect of underwater image enhancement. Summary of the invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide a method and system for underwater image quality enhancement based on spectral super-resolution technology, which can achieve comprehensive optimization and enhancement of underwater images by introducing spectral image super-resolution technology.

[0005] The first technical solution adopted by the present invention is: a method for enhancing underwater image quality based on spectral super-resolution technology, comprising the following steps:

[0006] Obtain real images, color RGB reference images, and hyperspectral reference images taken in underwater environments;

[0007] Performing spectral super-resolution on the real image based on a spectral super-resolution neural network to obtain a hyperspectral reconstructed image;

[0008] Compensating for degradation of the hyperspectral reconstructed image based on a color correction module to obtain a color RGB enhanced image;

[0009] Constructing a multi-task loss function based on the spectral reconstruction loss of the hyperspectral reconstructed image compared to the hyperspectral reference image and the difference loss between the color RGB enhanced image and the color RGB reference image;

[0010] Based on a multi-task loss function, the spectral super-resolution neural network and the color correction module are trained and optimized to obtain a trained spectral super-resolution neural network and a color correction module;

[0011] The real image to be enhanced is input into the trained spectral super-resolution neural network and color correction module for enhancement to obtain a final color RGB enhanced image.

[0012] Furthermore, the spectral super-resolution neural network includes a shallow feature extraction module, a feature mapping module and a reconstruction module, wherein:

[0013] The shallow feature extraction module includes a convolution layer and an activation function;

[0014] The feature mapping module includes a plurality of multi-scale feature enhancement modules;

[0015] The reconstruction module includes a convolutional layer, a spatial spectrum optimization module and an activation function;

[0016] The multi-scale feature enhancement module includes a plurality of grouped large-kernel convolution modules, a convolution layer and a channel selective gating module;

[0017] The space spectrum optimization module includes a second-order non-local residual module and a channel selective gating module;

[0018] The grouped large-kernel convolution module includes a large-kernel convolution module and a multi-scale fusion module.

[0019] Furthermore, the step of performing spectral super-resolution on the real image based on the spectral super-resolution neural network to obtain a hyperspectral reconstructed image specifically includes:

[0020] Performing feature extraction on the real image based on the shallow feature extraction module to obtain a first feature image;

[0021] Performing group feature extraction on the first feature image based on the group large kernel convolution module to obtain a plurality of groups of fifth feature images;

[0022] Based on the convolution layer, feature fusion is performed on the plurality of groups of fifth feature images, and the number of feature map channels is compressed to obtain a sixth feature image;

[0023] Performing multi-scale attention splicing on the sixth feature image based on the channel selective gating module to obtain a seventh feature image;

[0024] The seventh characteristic image is reconstructed based on the reconstruction module to obtain a hyperspectral reconstructed image.

[0025] Through this preferred step, the nonlinear mapping generation from the real image to the hyperspectral reconstructed image can be effectively achieved through data-driven, providing additional spectral information supplement for subsequent color image quality enhancement based on multispectral information.

[0026] Further, the step of performing group feature extraction on the first feature image based on the group large kernel convolution module to obtain a plurality of groups of fifth feature images specifically includes:

[0027] Performing tensor grouping on the first feature images to obtain a plurality of groups of second feature images;

[0028] Performing feature expansion on the plurality of groups of second feature images based on a three-stage large-kernel convolution module to obtain a plurality of groups of third feature images;

[0029] Adjusting the plurality of groups of third feature images based on the gating generated by the deep convolution to obtain a plurality of groups of fourth feature images;

[0030] The plurality of groups of fourth feature images are fused based on the multi-scale fusion module to obtain a plurality of groups of fifth feature images.

[0031] This optimization step helps to better utilize multi-level features, expand the receptive field range and thus improve model performance.

[0032] Further, the step of reconstructing the seventh feature image based on the reconstruction module to obtain a hyperspectral reconstructed image specifically includes:

[0033] Performing dimension compression on the seventh feature image based on the convolution layer to obtain an eighth feature image;

[0034] Performing residual connection on the eighth feature image based on the second-order non-local residual module to obtain a ninth feature image;

[0035] Channel weight embedding is performed on the ninth feature image based on the channel selective gating module to obtain a hyperspectral reconstructed image.

[0036] Through this preferred step, the feature map can be compressed to the dimension of the target hyperspectral image, and the spatial dimension and the spectral dimension can be continuously adjusted to finally obtain a more refined hyperspectral image.

[0037] Furthermore, the step of compensating the degradation of the hyperspectral reconstructed image based on the color correction module to obtain a color RGB enhanced image specifically includes:

[0038] Assigning weights to the band information of the hyperspectral reconstructed image based on an adaptively selected color compensation function to obtain an initial weight vector;

[0039] Decomposing the band information data matrix of the hyperspectral reconstructed image into a weight matrix and a prototype matrix based on a non-negative matrix decomposition method;

[0040] The weight matrix is ​​learned based on the deep variational information bottleneck method and the prototype matrix to obtain an optimized weight matrix;

[0041] Optimizing the initial weight vector based on the optimized weight matrix to obtain an optimized weight vector;

[0042] The hyperspectral reconstructed image is compensated for degradation based on the optimized weight vector to obtain a color RGB enhanced image.

[0043] Through this preferred step, more additional fine high-resolution spectral band information can be used to achieve color correction, thereby obtaining an enhanced image with more accurate colors and better visual effects.

[0044] Furthermore, the multi-task loss function is expressed as follows:

[0045]

[0046] Among them, L represents the multi-task loss function; L1 represents the spectral reconstruction loss; L2 represents the difference loss; N represents the total number of pixels; represents the pixel of the pth pixel of the hyperspectral reconstructed image; represents the pixel of the pth pixel of the hyperspectral reference image; τ represents the weighting coefficient; M represents the number of pixels of the hyperspectral single-channel image; Represents the one-dimensional spectral curve of the j-th pixel of the hyperspectral reconstructed image; Represents the one-dimensional spectral curve of the j-th pixel of the hyperspectral reference image; represents the pixel of the i-th pixel of the color RGB enhanced image; represents the pixel of the i-th pixel of the color RGB reference image; λ and α are both hyperparameters; C fixed represents the prototype matrix of the fixed position; H and W both represent the weight matrix after learning; λ reg represents the regularization coefficient; represents the square of the Euclidean norm; represents the square of the Frobenius norm; F represents the Frobenius norm.

[0047] The second technical solution adopted by the present invention is: an underwater image quality enhancement system based on spectral super-resolution technology, comprising:

[0048] A data acquisition module, used to acquire real images, color RGB reference images and hyperspectral reference images taken in underwater environments;

[0049] A spectral super-resolution neural network module is used to perform spectral super-resolution on the real image to obtain a hyperspectral reconstructed image;

[0050] A color correction module, used for compensating the degradation of the hyperspectral reconstructed image to obtain a color RGB enhanced image;

[0051] A loss function construction module, which constructs a multi-task loss function based on the spectral reconstruction loss of the hyperspectral reconstructed image compared to the hyperspectral reference image and the difference loss between the color RGB enhanced image and the color RGB reference image;

[0052] A training optimization module, which performs training optimization on the spectral super-resolution neural network and the color correction module based on a multi-task loss function to obtain trained spectral super-resolution neural network and color correction module;

[0053] The enhancement module is used to input the real image to be enhanced into the trained spectral super-resolution neural network and color correction module for enhancement to obtain a final color RGB enhanced image.

[0054] The beneficial effects of the method and system of the present invention are as follows: the present invention proposes a spectral super-resolution neural network, which has powerful feature learning and mapping capabilities, can effectively realize the nonlinear mapping generation of original images to high-resolution spectral image data through data-driven, and provides additional spectral information supplement for subsequent color image quality enhancement based on multi-spectral information; a color correction module is proposed, which performs preliminary color correction on the image through a deep prototype analysis method, aiming to optimize the band weights and improve the initial color performance of the image. In the link of weight initialization, a color compensation method for underwater image characteristics is integrated, which can accurately assign appropriate initialization weights to each band, further enhancing the pertinence and effectiveness of color correction; the present invention proposes a joint optimization strategy, combines spectral reconstruction loss and difference loss to construct a multi-task loss function, and trains the spectral super-resolution neural network and color correction module; finally, the trained spectral super-resolution neural network and color correction module are closely combined to realize the joint optimization of underwater hyperspectral image data generation and underwater image quality enhancement tasks, and realize comprehensive optimization and enhancement of underwater images, which not only improves the efficiency and accuracy of image processing, but also provides a broader possibility for the application field of underwater images. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flow chart of the steps of an underwater image quality enhancement method based on spectral super-resolution technology of the present invention;

[0056] Figure 2 It is a structural block diagram of an underwater image quality enhancement system based on spectral super-resolution technology of the present invention;

[0057] Figure 3 It is a schematic diagram of an implementation framework of an underwater image quality enhancement method based on spectral super-resolution technology of the present invention;

[0058] Figure 4 It is a spectral super-resolution neural network structure diagram of an underwater image quality enhancement method based on spectral super-resolution technology of the present invention;

[0059] Figure 5 This is a structural diagram of a multi-scale feature enhancement module of an underwater image quality enhancement method based on spectral super-resolution technology of the present invention;

[0060] Figure 6 This is a structural diagram of a grouped large-kernel convolution module of an underwater image quality enhancement method based on spectral super-resolution technology in the present invention. DETAILED DESCRIPTION

[0061] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0062] Reference Figure 1 and Figure 3 The present invention provides a method for enhancing underwater image quality based on spectral super-resolution technology, the method comprising the following steps:

[0063] S1, obtaining a real image, a color RGB reference image and a hyperspectral reference image taken in an underwater environment;

[0064] Specifically, accurately acquiring real image data taken in an actual underwater environment directly determines the accuracy and reliability of subsequent image processing and analysis. In order to verify and evaluate the effect of subsequent processing, it is also necessary to obtain reference image data corresponding to the real image data. These reference image data include color RGB reference images and hyperspectral reference images. Color RGB reference images can be standard images of underwater environments or high-quality images that have been professionally processed. They will serve as a benchmark for measuring the effect of image quality enhancement. Hyperspectral reference images contain more spectral information than traditional RGB images and can more comprehensively reflect the physical and chemical properties of underwater environments. Therefore, when acquiring real image data taken in an actual underwater environment, the corresponding hyperspectral reference images will be collected synchronously so that these rich spectral information can be used in subsequent processing to enhance image quality.

[0065] S2, performing spectral super-resolution on the real image based on a spectral super-resolution neural network to obtain a hyperspectral reconstructed image;

[0066] Specifically, refer to Figure 4 , the spectral super-resolution neural network includes a shallow feature extraction module, a feature mapping module and a reconstruction module, wherein:

[0067] The shallow feature extraction module includes two 3×3 convolutional layers and a PReLU activation function;

[0068] The feature mapping module includes a plurality of multi-scale feature enhancement modules (MSFEM);

[0069] The reconstruction module includes a 3×3 convolutional layer, a spatial spectrum optimization module (SSOM) and a LeakReLU activation function. S2.1. Extracting features from the real image based on the shallow feature extraction module to obtain a first feature image;

[0070] Specifically, in the shallow feature extraction module, a PReLU activation function is added between two 3×3 convolutional layers, and two 3×3 convolutional layers are used to extract features from the real image to obtain a first feature image.

[0071] S2.2, performing group feature extraction on the first feature image based on the group large kernel convolution module to obtain a plurality of groups of fifth feature images;

[0072] Specifically, refer to Figure 6 The grouped large kernel convolution module (GLKCM) includes a large kernel convolution part (LKC) and a multi-scale fusion part (except LKC).

[0073] S2.2.1. performing tensor grouping on the first feature images to obtain a plurality of groups of second feature images;

[0074] Specifically, when the first feature image is input into the grouped large kernel convolution module, assuming that the input first feature image is First, they were divided into two groups: and Then, X2 is divided into n blocks along the channel direction to obtain several sets of second feature images, that is, X2 = [x 21 ,x 22 ,…,x 2n ], where the dimension of each sub-feature map is

[0075] S2.2.2. Perform feature expansion on the plurality of groups of second feature images based on a three-stage large-kernel convolution module to obtain a plurality of groups of third feature images;

[0076] Specifically, the large kernel convolution module divides the K×K convolution into three stages, namely, the depth convolution of size (2d-1)×(2d-1) A size of Depthwise convolution And a point convolution H PW (·).

[0077] If a feature image is given The convolution function expression of the large kernel convolution module is as follows:

[0078]

[0079] S2.2.3. Adjust the plurality of groups of third feature images based on the gate generated by the deep convolution to obtain a plurality of groups of fourth feature images;

[0080] Specifically, although large kernel convolution can expand the perceptual range, there may be blocking artifacts in the generated attention feature map. In order to avoid this blocking effect and learn local information more effectively, the present invention uses a spatial gating dynamic adjustment method for adjustment. The specific process is to adjust the large kernel convolution function L(X) using the gating generated by the deep convolution of size a×a. This method helps to eliminate the "blocking effect" that occurs in the spectral super-resolution process. The block boundaries of the feature map become obvious, and L(X) with a larger receptive field is more sensitive to long-distance dependencies.

[0081] S2.2.4. Based on the multi-scale fusion module, fuse the several groups of fourth feature images to obtain several groups of fifth feature images.

[0082] After the above-mentioned large-kernel convolution module and spatial gating dynamic adjustment method, the fourth feature image is generated, and then a 3×3 convolution layer is used to fuse the features of several groups of fourth feature images to generate an attention feature map of the corresponding scale, namely the fifth feature image.

[0083] S2.3, performing feature fusion on the plurality of groups of fifth feature images based on the convolution layer, and compressing the number of feature map channels to obtain a sixth feature image;

[0084] Specifically, refer to Figure 5 , the multi-scale feature enhancement module (MSFEM) includes 6 grouped large kernel convolution modules (GLKCM), a 3×3 convolution layer and a channel selective gating module (CSGB). The 6 grouped large kernel convolution modules (GLKCM) are divided into three groups of functional units to perform group feature extraction in accordance with step S2.2. The calculation expressions of the three groups of functional units are as follows:

[0085] s1=HG11 (s0)

[0086] s2=H G22 (H G21 (s0))

[0087] s3=H G33 (H G32 (H G31 (s0)))

[0088] Among them, s0 represents the input of the grouped large kernel convolution module; s1 represents the output of the first functional unit; H G11 represents the first grouped large kernel convolution module of the first functional unit; s2 represents the output of the second functional unit; H G21 H represents the first grouped large kernel convolution module of the second functional unit; G22 represents the second grouped large kernel convolution module of the second functional unit; s3 represents the output of the third functional unit; H G31 H represents the first grouped large kernel convolution module of the third functional unit; G32 H represents the second grouped large kernel convolution module of the third functional unit; G33 Represents the third grouped large kernel convolution module of the third functional unit.

[0089] Then, a 3×3 convolutional layer is used to perform a feature fusion operation on the fifth feature image output by the three groups of functional units, and the number of feature map channels is compressed to obtain the sixth feature image.

[0090] S2.4, performing multi-scale attention splicing on the sixth feature image based on the channel selective gating module to obtain a seventh feature image;

[0091] Specifically, by splicing the attention feature maps of each scale in the channel dimension, it is possible to obtain the attention feature map with full-scale information, that is, the seventh feature image.

[0092] S2.5. Reconstruct the seventh feature image based on the reconstruction module to obtain a hyperspectral reconstructed image.

[0093] Specifically, the space spectrum optimization module (SSOM) includes a second-order non-local residual module and a channel selective gating module.

[0094] S2.5.1. Dimensionally compressing the seventh feature image based on the convolution layer to obtain an eighth feature image;

[0095] Specifically, the role of the 3×3 convolutional layer is to compress the attention feature map (i.e., the seventh feature image) obtained in the previous stage to the dimension of the target hyperspectral image to obtain a rough hyperspectral image, i.e., the eighth feature image.

[0096] S2.5.2. Perform residual connection on the eighth feature image based on the second-order non-local residual module to obtain a ninth feature image;

[0097] Specifically, in the second-order non-local residual module, the input feature map is first Decompose in the spatial dimension to obtain 4 sub-feature maps For each sub-feature map B k , first perform a 1×1 convolution operation, and set the number of output channels to Among them, r represents a constant, and the output channel is divided into r parts; thus, two new feature maps M are output k and N k , after reshaping and transposition operations, becomes and Then calculate the feature map M k The covariance of is expressed as follows:

[0098]

[0099] n=hw

[0100] Among them, D k represents covariance; I represents the identity matrix of size n×n; 1 represents the full-1 matrix of size;

[0101] The covariance D k After being processed by the softmax(·) activation function, k Perform local matrix multiplication and apply U k Reshape and transpose Then U k With the initial feature B k They are respectively passed to their respective 1×1 convolutional layers with output channels of C, and the calculation results are residually connected to obtain the ninth feature image.

[0102] S2.5.3. Based on the channel selective gating module, channel weight embedding is performed on the ninth feature image to obtain a hyperspectral reconstructed image.

[0103] Specifically, after the ninth feature image is input into the channel selective gating module, the ninth feature image Z is processed according to the following formula, which is expressed as follows:

[0104]

[0105] in, represents the output of the channel selective gating module with respect to the ninth feature image; F(·) represents the channel selective gating module function; ω represents the embedding weight, which is used to adjust the embedding output; ξ represents the gating weight; ψ represents the bias; ξ and ψ determine the activation of the gate; ω, ξ, and ψ simultaneously determine the behavior of each gating channel of the feature map.

[0106] Assume that the feature map F is given and its representation is as follows:

[0107]

[0108] Among them, F C Represents the Cth channel of the feature map.

[0109] The embedding weights of the feature map F are expressed as follows:

[0110] ω=[ω1,ω2,…,ω C ]

[0111] Where ω represents the embedding weight; ω C Represents the embedding weight of the Cth channel of the feature map.

[0112] The embedding weights ω aggregate the context information in each channel to generate the feature map z c , which is expressed as follows:

[0113]

[0114] Among them, ε represents a very small constant, which is used to avoid the derivative problem at the zero point; Represents the pixel value of the i-th row and j-th column on the C-th channel of the feature map D.

[0115] Then establish a competition mechanism between neurons to maintain the balance between neurons and obtain intermediate features Its expression is as follows:

[0116]

[0117] Among them, C represents the number of feature map channels; Represents a scalar, used for normalization scale to avoid the problem of large number of feature map channels Smaller scale.

[0118] A gating mechanism is established through gating weights ξ and bias ψ to promote competition or cooperation during network model training and adjust intermediate features. The goal is given by the gating weights and biases, which are expressed as follows:

[0119] ξ=[ξ1,ξ2,…ξ C ]

[0120] ψ=[ψ1,ψ2,…,ψ C ]

[0121] Among them, ξ C represents the gating weight of the Cth channel of the feature map; ψ C Represents the bias of the Cth channel of the feature map.

[0122] Each initial channel F is realized with given gating weights and biases C Adjusted by the corresponding gate, its expression is as follows:

[0123]

[0124] in, Represents the final output of each initial channel.

[0125] When the channel gating weight ξ C When the channel is positively activated, the channel selective gating module causes the channel to compete with other channels; when the channel gating weight ξ C When negatively activated, the channel-selective gating module forces the channel to cooperate with other channels.

[0126] S3, compensating for degradation of the hyperspectral reconstructed image based on a color correction module to obtain a color RGB enhanced image;

[0127] Specifically, in order to use more additional fine high-resolution spectral band information to achieve color correction, the 31-channel hyperspectral reconstructed image data information will be degraded to a more representative 3-channel color image, thereby obtaining an enhanced image with more accurate colors and better visual effects. Degrading the hyperspectral reconstructed image to an RGB image requires 31 bands of information, and finally achieving band compensation compression and reconstruction into 3 bands to represent the red, green and blue channels. Therefore, the weight vector can be defined as follows:

[0128] w=[w1,W2…,w 31 ]

[0129] Among them, w1 represents the weight of the first band in the color RGB enhanced image.

[0130] The color correction module uses a set of appropriate weights to make the color RGB enhanced image as close to the ideal color representation as possible visually.

[0131] S3.1, assigning weights to the band information of the hyperspectral reconstructed image based on an adaptively selected color compensation function to obtain an initial weight vector;

[0132] Specifically, in order to guide the weight matrix to reasonably compensate the weight vector, it is necessary to take into account the imaging characteristics of the underwater image. The specific embodiment of the present invention evaluates the imaging characteristics through calculation, and then adaptively selects the color compensation function to initialize the weight vector.

[0133] First, choose the simplest accumulation method to calculate the sum of all pixel values ​​of each channel, and finally determine the compensation order from high to low. If the blue channel has the highest total intensity, first use the blue channel to compensate the green channel, and then use the green channel to compensate the red channel. If the green channel has the highest total intensity, use the green channel to compensate both the blue and red channels. If the red channel has the highest total intensity, use the red channel to compensate the green channel, and then use the green channel to compensate the blue channel. For each input image, after calculating its imaging characteristics, use the adjacent channel compensation formula to compensate the color. For example, use the green channel to compensate the red channel. The compensation function is as follows:

[0134] R'(x)=R(x)=[G(x)-R(x)]·[1-R(x)]·G(x)

[0135] Among them, R'(x) represents the pixel value of the red channel after compensation; R(x) represents the pixel value of the red channel; G(x) represents the pixel value of the green channel.

[0136] The green channel is compensated using the red channel, and the compensation function is as follows:

[0137] G′(x)=G(x)+[R(x)-G(x)]·[1-G(x)‖·R(x)

[0138] Wherein, G'(x) represents the pixel value of the green channel after compensation.

[0139] The green channel is compensated using the blue channel, and the compensation function is as follows:

[0140] G′(x)=G(x)+[B(x)-G(x)]·[1-G(x)]·B(x)

[0141] Among them, B(x) represents the pixel value of the blue channel.

[0142] The blue channel is compensated with the green channel, and the compensation function is as follows:

[0143] B'(x)=B(x)+[G(x)-B(x)]·[1-B(x)]·G(x)

[0144] Wherein, B'(x) represents the pixel value of the blue channel after compensation.

[0145] The factors [1-G(x)], [1-B(x)], and [1-R(x)] in each equation are used to ensure that the compensation operation does not introduce unrealistic colors, where G(x), B(x), and R(x) are the normalized values ​​of the pixel values ​​of the corresponding channels. Each time if the red, green, and blue channel intensity ratios of the compensated RGB image are r:g:b, the initial weights can be set as follows:

[0146]

[0147] Among them, l represents the initialization coefficient, which is used to normalize the initial weight vector; r represents the sum of the pixel values ​​of the red channel; g represents the sum of the pixel values ​​of the green channel; and b represents the sum of the pixel values ​​of the blue channel.

[0148] Such initial weight design helps to incorporate the color characteristics of underwater images and further enhance interpretability.

[0149] S3.2, decomposing the band information data matrix of the hyperspectral reconstructed image into a weight matrix and a prototype matrix based on a non-negative matrix decomposition method;

[0150] Specifically, a loss function L(w) is defined to quantify the difference between the converted color RGB enhanced image and the target, and its expression is as follows:

[0151]

[0152] Among them, RGB tar get Represents the target RGB image; RGB predict Represents the converted color RGB enhanced image.

[0153] The objective function of loss function optimization is expressed as follows:

[0154]

[0155] Among them, w i Represents the weight of the i-th band.

[0156] However, there are constraints in the optimization, which requires ensuring that the weights are normalized and that the weights of each band are non-negative. In order to solve the above optimization problem, a non-negative matrix decomposition method can be used to decompose the band information data matrix of the hyperspectral reconstructed image into a weight matrix and a prototype matrix, which is expressed as follows:

[0157]

[0158] Where X represents the data matrix; A represents the prototype matrix, which contains k prototypes a i ; P represents the weight matrix.

[0159] Each row of the weight matrix satisfies the probability distribution, that is:

[0160]

[0161] Among them, p i,j represents the element in the i-th row and j-th column of the weight matrix P; a i,j Represents the element in the i-th row and j-th column of the prototype matrix A.

[0162] The prototype is regarded as the most important band feature in the hyperspectral image, and the weight matrix P is the weight required for band compensation. By learning these prototypes, the band combination that best represents the RGB color space based on multispectral band compressed sensing can be found.

[0163] S3.3, learning the weight matrix based on the deep variational information bottleneck method and the prototype matrix to obtain an optimized weight matrix;

[0164] Specifically, in order to address the nonlinear representation introduced by the above decomposition process, the deep variational information bottleneck method is then used. The goal of this method is to find a random variable Z that compresses a given random vector X while retaining as much information about the second random vector Y as possible. The expression of its objective function is as follows:

[0165]

[0166] Where λ represents the Lagrange multiplier; θ represents the random variable; and E represents the expected value of the random variable.

[0167] The above prototype concept is introduced in the deep variational information bottleneck framework to learn the nonlinear potential representation of data, that is, the objective function can be converted to the following form:

[0168]

[0169] Among them, Ω(A,P) represents the prototype loss function; A fixed A prototype representing a fixed position; PA fixed represents the prototype position predicted by the learned weight matrix P.

[0170] The training and learning process involves minimizing the objective function, which is usually achieved through stochastic gradient descent (SGD) or its variants (such as Adam optimizer). During the training process, it is necessary to learn the parameters of the encoder and decoder, as well as the weight matrix and prototype matrix, and finally obtain the optimized weight matrix and prototype features. In the optimized weight matrix, the band combination based on multi-spectral band compressed sensing that best represents the RGB color space can be found. Since each channel represents a color, the data of each channel needs to be normalized to the range of 0 to 256 to adapt to the standard format of 8-bit color images.

[0171] S3.4. Optimizing the initial weight vector based on the optimized weight matrix to obtain an optimized weight vector;

[0172] S3.5. Compensate for degradation of the hyperspectral reconstructed image based on the optimized weight vector to obtain a color RGB enhanced image.

[0173] S4, constructing a multi-task loss function based on the spectral reconstruction loss of the hyperspectral reconstructed image compared to the hyperspectral reference image and the difference loss between the color RGB enhanced image and the color RGB reference image;

[0174] Specifically, for the spectral reconstruction stage, pixel-level information and spectral characteristic similarity are integrated when designing the loss function to enhance the robustness of the model to noise and data changes. Specifically, the spectral reconstruction loss function is defined as follows:

[0175]

[0176] Where L1 represents the spectral reconstruction loss; N represents the total number of pixels; represents the pixel of the pth pixel of the hyperspectral reconstructed image; represents the pixel of the pth pixel of the hyperspectral reference image; τ represents the weighting coefficient, which is generally set to 0.1 during training; M represents the number of pixels of the hyperspectral single-channel image; Represents the one-dimensional spectral curve of the j-th pixel of the hyperspectral reconstructed image; Represents the one-dimensional spectral curve of the j-th pixel of the hyperspectral reference image.

[0177] For the color enhancement stage, four parts are mainly considered when designing the loss function. First, the loss between the color RGB enhanced image and the color RGB reference image is measured using the mean square error, which is expressed as follows:

[0178]

[0179] Among them, L recon represents mean square error; represents the pixel of the i-th pixel of the color RGB enhanced image; represents the pixel at the i-th pixel of the color RGB reference image; A number representing the square of the Euclidean norm.

[0180] For the prototype loss generated by deep prototype analysis, its expression is as follows:

[0181]

[0182] Among them, L AT represents the prototype loss; C fixed represents the prototype matrix of the fixed position; H and W both represent the weight matrix after learning; represents the square of the Frobenius norm.

[0183] In addition, in order to prevent overfitting, it is necessary to design a regularization term to control the complexity of the model. The expression is as follows:

[0184]

[0185] Among them, L reg represents the regularization term; λ reg represents the regularization coefficient.

[0186] Therefore, the difference loss in the color enhancement stage is expressed as follows:

[0187]

[0188] Among them, L2 represents the difference loss; λ and α are both hyperparameters.

[0189] Finally, the multi-task loss function is expressed as follows:

[0190]

[0191] S5, training and optimizing the spectral super-resolution neural network and the color correction module based on a multi-task loss function to obtain trained spectral super-resolution neural network and color correction module;

[0192] Specifically, before training the spectral super-resolution neural network based on the multi-task loss function, a transfer learning strategy is adopted, that is, firstly, the network is trained using a rich non-underwater hyperspectral image data set, and the trained network is used as the network initialized for underwater spectral super-resolution to realize the knowledge transfer of spectral reconstruction in the absence of water medium, and then a small amount of underwater hyperspectral data set is used to fine-tune the network. This strategy not only solves the problem that underwater hyperspectral image data is relatively scarce due to the high cost of acquisition, but also significantly reduces the network's dependence on specific underwater hyperspectral image data sets, thereby fully demonstrating the efficiency and practicality of the transfer learning method. Then, based on the principle of minimizing the multi-task loss function, the parameters of the spectral super-resolution neural network and the color correction module are adjusted. Once the value of the total loss function converges to within the pre-set threshold range, then there is sufficient basis to believe that the optimization task has been successfully completed, and the trained spectral super-resolution neural network and color correction module are obtained.

[0193] S6. Inputting the real image to be enhanced into the trained spectral super-resolution neural network and color correction module for enhancement to obtain a final color RGB enhanced image.

[0194] Reference Figure 2 , an underwater image quality enhancement system based on spectral super-resolution technology, comprising:

[0195] A data acquisition module, used to acquire real images, color RGB reference images and hyperspectral reference images taken in underwater environments;

[0196] A spectral super-resolution neural network module is used to perform spectral super-resolution on the real image to obtain a hyperspectral reconstructed image;

[0197] A color correction module, used for compensating the degradation of the hyperspectral reconstructed image to obtain a color RGB enhanced image;

[0198] A loss function construction module, which constructs a multi-task loss function based on the spectral reconstruction loss of the hyperspectral reconstructed image compared to the hyperspectral reference image and the difference loss between the color RGB enhanced image and the color RGB reference image;

[0199] A training optimization module, which performs training optimization on the spectral super-resolution neural network and the color correction module based on a multi-task loss function to obtain trained spectral super-resolution neural network and color correction module;

[0200] The enhancement module is used to input the real image to be enhanced into the trained spectral super-resolution neural network and color correction module for enhancement to obtain a final color RGB enhanced image.

[0201] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0202] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for underwater image quality enhancement based on spectral super-resolution technology, characterized in that: The following steps are involved: Obtain real images, color RGB reference images, and hyperspectral reference images taken in underwater environments; Performing spectral super-resolution on the real image based on a spectral super-resolution neural network to obtain a hyperspectral reconstructed image; Compensating for degradation of the hyperspectral reconstructed image based on a color correction module to obtain a color RGB enhanced image; Constructing a multi-task loss function based on the spectral reconstruction loss of the hyperspectral reconstructed image compared to the hyperspectral reference image and the difference loss between the color RGB enhanced image and the color RGB reference image; Based on a multi-task loss function, the spectral super-resolution neural network and the color correction module are trained and optimized to obtain a trained spectral super-resolution neural network and a color correction module; Inputting the real image to be enhanced into the trained spectral super-resolution neural network and color correction module for enhancement to obtain a final color RGB enhanced image; The spectral super-resolution neural network includes a shallow feature extraction module, a feature mapping module and a reconstruction module, wherein: The shallow feature extraction module includes a convolution layer and an activation function; The feature mapping module includes a plurality of multi-scale feature enhancement modules; The reconstruction module includes a convolution layer, a spatial spectrum optimization module and an activation function; The multi-scale feature enhancement module includes a plurality of grouped large-kernel convolution modules, a convolution layer and a channel selective gating module; The space spectrum optimization module includes a second-order non-local residual module and a channel selective gating module; The grouped large kernel convolution module includes a large kernel convolution module and a multi-scale fusion module; The step of performing spectral super-resolution on the real image based on the spectral super-resolution neural network to obtain a hyperspectral reconstructed image specifically includes: Performing feature extraction on the real image based on the shallow feature extraction module to obtain a first feature image; Performing group feature extraction on the first feature image based on the group large kernel convolution module to obtain a plurality of groups of fifth feature images; Based on the convolution layer, feature fusion is performed on the plurality of groups of fifth feature images, and the number of feature map channels is compressed to obtain a sixth feature image; Performing multi-scale attention splicing on the sixth feature image based on the channel selective gating module to obtain a seventh feature image; Reconstructing the seventh characteristic image based on the reconstruction module to obtain a hyperspectral reconstructed image; The step of compensating the degradation of the hyperspectral reconstructed image based on the color correction module to obtain a color RGB enhanced image specifically includes: Assigning weights to the band information of the hyperspectral reconstructed image based on an adaptively selected color compensation function to obtain an initial weight vector; Decomposing the band information data matrix of the hyperspectral reconstructed image into a weight matrix and a prototype matrix based on a non-negative matrix decomposition method; The weight matrix is ​​learned based on the deep variational information bottleneck method and the prototype matrix to obtain an optimized weight matrix; Optimizing the initial weight vector based on the optimized weight matrix to obtain an optimized weight vector; The hyperspectral reconstructed image is compensated for degradation based on the optimized weight vector to obtain a color RGB enhanced image.

2. According to claim 1, the underwater image quality enhancement method based on spectral super-resolution technology is characterized in that: The step of performing group feature extraction on the first feature image based on the group large kernel convolution module to obtain a plurality of groups of fifth feature images specifically includes: Performing tensor grouping on the first feature images to obtain a plurality of groups of second feature images; Performing feature expansion on the plurality of groups of second feature images based on a three-stage large-kernel convolution module to obtain a plurality of groups of third feature images; Adjusting the plurality of groups of third feature images based on the gating generated by the deep convolution to obtain a plurality of groups of fourth feature images; The plurality of groups of fourth feature images are fused based on the multi-scale fusion module to obtain a plurality of groups of fifth feature images.

3. The underwater image quality enhancement method based on spectral super-resolution technology according to claim 2 is characterized in that: The step of reconstructing the seventh feature image based on the reconstruction module to obtain a hyperspectral reconstructed image specifically includes: Performing dimension compression on the seventh feature image based on the convolution layer to obtain an eighth feature image; Performing residual connection on the eighth feature image based on the second-order non-local residual module to obtain a ninth feature image; Channel weight embedding is performed on the ninth feature image based on the channel selective gating module to obtain a hyperspectral reconstructed image.

4. The underwater image quality enhancement method based on spectral super-resolution technology according to claim 1, characterized in that: The multi-task loss function is expressed as follows: Among them, L represents the multi-task loss function; L1 represents the spectral reconstruction loss; L2 represents the difference loss; N represents the total number of pixels; represents the pixel of the pth pixel of the hyperspectral reconstructed image; represents the pixel of the pth pixel of the hyperspectral reference image; τ represents the weighting coefficient; M represents the number of pixels of the hyperspectral single-channel image; Represents the one-dimensional spectral curve of the j-th pixel of the hyperspectral reconstructed image; Represents the one-dimensional spectral curve of the j-th pixel of the hyperspectral reference image; represents the pixel of the i-th pixel of the color RGB enhanced image; represents the pixel of the i-th pixel of the color RGB reference image; λ and α are both hyperparameters; C fixed represents the prototype matrix of the fixed position; H and W both represent the weight matrix after learning; λ reg represents the regularization coefficient; represents the square of the Euclidean norm; represents the square of the Frobenius norm; F represents the Frobenius norm.

5. An underwater image quality enhancement system based on spectral super-resolution technology according to the method of claim 1, characterized in that: include: A data acquisition module, used to acquire real images, color RGB reference images and hyperspectral reference images taken in underwater environments; A spectral super-resolution neural network module is used to perform spectral super-resolution on the real image to obtain a hyperspectral reconstructed image; A color correction module, used to compensate for degradation of the hyperspectral reconstructed image to obtain a color RGB enhanced image; A loss function construction module, which constructs a multi-task loss function based on the spectral reconstruction loss of the hyperspectral reconstructed image compared to the hyperspectral reference image and the difference loss between the color RGB enhanced image and the color RGB reference image; A training optimization module, which performs training optimization on the spectral super-resolution neural network and the color correction module based on a multi-task loss function to obtain trained spectral super-resolution neural network and color correction module; The enhancement module is used to input the real image to be enhanced into the trained spectral super-resolution neural network and color correction module for enhancement to obtain a final color RGB enhanced image.

Images