Image enhancement method and device based on multi-view input and structure guidance

Abstract

The invention discloses a structure-guided multi-priori input image enhancement method and device. The method comprises the following steps: step 1, collecting low-quality images and corresponding high-quality images to form a data set; dividing the data set into a training set, a verification set and a test set; 2, constructing a multi-view-angle input enhancement network, wherein the multi-view-angle input enhancement network is used for preliminarily enhancing the low-quality image; designing a multi-view-angle enhanced joint loss function Lmve, specifying a training round, training and optimizing the multi-view-angle input enhanced network model, and storing the weight of the optimal model; step 3, constructing a structure guidance enhancement network, wherein the structure guidance enhancement network performs secondary enhancement on the image data based on the structure information extracted by the structure extraction module; and designing a structure guidance enhancement joint loss function Lsge, specifying a training round, training and optimizing the structure guidance enhancement network model, and storing the weight of the optimal model.

Description

Technical Field

[0001] The present invention relates to the field of image enhancement or restoration classification, and in particular to an image enhancement method and device based on multi-view input and structure guidance. Background Art

[0002] Image enhancement technology processes image data to improve its visual quality and information content, thereby improving the results of subsequent processing tasks. Image enhancement technology plays a vital role in the field of image processing and is widely used in many fields such as medical imaging, remote sensing imaging, and video surveillance to optimize image visibility, contrast, and detail performance.

[0003] In the prior art, there are the following problems:

[0004] 1. Traditional image enhancement algorithms usually use various filters and transformation methods to improve image quality. For example, histogram equalization can improve the contrast of an image, while a high-pass filter can enhance the edge details of an image. However, these traditional methods are often limited to rule-based processing and may not be able to effectively handle complex image quality issues such as image quality degradation under noise, blur, or low-light conditions;

[0005] 2. In recent years, deep learning methods have also made significant progress in the field of image enhancement. Deep learning models such as convolutional neural networks (CNN) and Transformer can achieve more advanced image enhancement by learning complex features of image data. However, convolutional neural networks (CNN) mainly focus on local information and have difficulty capturing global context, resulting in poor results when dealing with global structures and long-distance dependencies. Although the existing ViT has a global receptive field, its image block embedding part may cause local detail loss and artifacts or incoherent effects on image block boundaries, affecting the final restoration quality. In addition, the computational complexity of ViT is O(n 2 ), which will lead to high computing and storage requirements, resulting in efficiency issues;

[0006] 3. In deep neural networks, although the deep layers of the network usually extract rich semantic information, this process often leads to the abstraction and loss of important information such as structural features, edges and textures, thus affecting the effect of image enhancement. Summary of the invention

[0007] In view of the above problems, the present invention provides an image enhancement method and device combining multi-view input and structure guidance.

[0008] This method uses multi-view input to comprehensively improve the color accuracy and local contrast of the image. By combining different traditional image enhancement algorithms, it can more effectively deal with various image problems, enhance the details and visibility of the image, and avoid the shortcomings that may be caused by a single input method, thereby improving image quality and visual effects.

[0009] The channel group axial Transformer module proposed by this method has no image block embedding part, which can retain all the details of the original image, avoid information loss or boundary effects that may be caused by image block division, and can more effectively maintain and restore details and boundary information. In addition, the channel group axial Transformer has both global receptive field and linear computational complexity.

[0010] This method uses structural information for secondary guided enhancement to supplement and strengthen the network's extraction of structural information, which helps to make up for the shortcomings of deep networks in processing structural information and ensures that the structure and details of the image are better preserved and enhanced.

[0011] The present invention adopts the following technical solutions:

[0012] An image enhancement method based on multi-view input and structure guidance comprises the following steps:

[0013] S1: Collect low-quality images and corresponding high-quality images to form a data set; divide the data set into a training set, a validation set, and a test set;

[0014] S2: Construct a multi-view input enhancement network, which is used to perform preliminary enhancement on low-quality images; and design a multi-view enhancement joint loss function L mve , specify the training rounds, train and optimize the multi-view input enhancement network model, and save the weights of the optimal model;

[0015] The multi-view input enhancement network includes multi-view input (original input and its corresponding white balance and contrast limited adaptive histogram equalization image), cross-channel attention feature fusion (CCAF, Channel-Cross Attention Fusion) module, channel group axial Transformer (CGAT, Channel-Group Axial Transformer) module; the multi-view enhancement joint loss function L mve Contains L1 loss and perceptual loss;

[0016] S3: Construct a structure-guided enhancement network, which performs secondary enhancement on image data based on the structural information extracted by the structure extraction module; and design a joint loss function L for structure-guided enhancement. sge, specify the training rounds, train and optimize the structure-guided enhanced network model, and save the weights of the optimal model;

[0017] The structure-guided enhancement network includes a structure extraction module and a structure-guided enhancement module (SGEM, StructureGuided Enhancement Module); the structure-guided enhancement joint loss function Ls sge Contains L1 loss, perceptual loss and structural similarity index loss;

[0018] S4: Construct an enhancement network combining multi-view input and structure guidance. This network combines the trained multi-view input enhancement network and the structure guidance enhancement network to obtain the final image enhancement result.

[0019] S5: Test model performance.

[0020] As a preferred solution of the present invention, an image enhancement method based on multi-view input and structure guidance, the multi-view input enhancement network in S2:

[0021] The multi-view input enhancement network consists of two parts: multi-view input feature fusion and an asymmetric encoder-decoder structure network.

[0022] Multi-view input feature fusion part, the multi-view input includes the original input and its corresponding WB and CLAHE images. The three inputs of the multi-view input data are mapped to a new feature space through the same convolution layer. The three groups of feature maps are fused using the cross-channel attention feature fusion module to obtain the feature fusion result FU;

[0023] The encoder part has three layers of networks, and the feature fusion result FU is used as the input of the three layers of the encoder network. Each layer of the network is composed of CGAT modules, and the output of each layer will be element-wise added with the feature fusion result FU. The results are downsampled twice, and each layer of the network obtains three feature maps of different scales. Then, the feature maps of the same scale of different layers are connected in the channel dimension to obtain three groups of feature maps of different scales, which are FS1, FS2 and FS3 in increasing scale.

[0024] The decoder has three layers. Each layer of the decoder consists of a Squeeze-and-Excitation Network and a CGAT module. The encoder's three sets of output feature maps of different scales will be used as the input of the decoder's three-layer network, that is, FS1, FS2 and FS3 are the inputs of the bottom, middle and top layers of the decoder respectively. Each set of feature maps will first pass through the corresponding layer's Squeeze-and-Excitation network to obtain three feature maps FS11, FS21 and FS31. Starting from the bottom layer, FS11 is upsampled after passing through the CGAT module, and its result is connected with FS21 in the channel dimension. The result is put into the middle layer's CGAT module and then upsampled. The result is then connected with FS31 in the channel dimension. After passing through the top layer's CGAT module and convolutional layer, the multi-view input enhancement result OUT is obtained. mve .

[0025] As a preferred solution of the present invention, an image enhancement method based on multi-view input and structure guidance, the cross-channel attention feature fusion module in S2:

[0026] The cross-channel attention feature fusion module is based on the channel self-attention mechanism and can capture the relationship between different channels of multi-input feature maps.

[0027] Specifically, N multi-view input feature maps of shape (C, H, W) are connected in the channel dimension to obtain a feature map S1 of shape (N×C, H, W), where C is the number of channels, and H and W are the height and width of the feature map, respectively. After 1×1 convolution and 3×3 depth convolution, the feature map S1 is reshaped to (N×C, H×W) to obtain the query matrix Q, key matrix K, and value matrix V. The Q matrix and the K matrix are matrix multiplied to obtain a feature matrix of shape (N×C, N×C), and then the feature matrix is ​​matrix multiplied with V. The result is element-wise added to the feature map S1, and finally the feature fusion result is obtained after 1×1 convolution and channel shuffle.

[0028] As a preferred solution of the present invention, an image enhancement method based on multi-view input and structure guidance, the channel group axial Transformer module in S2:

[0029] The channel group axial Transformer module consists of channel shuffling, channel grouping and axial Transformer. The axial Transformer consists of a multi-layer structure, including an instance normalization layer, an axial self-attention layer and a spatial gated feedforward network layer.

[0030] The channel group axial Transformer module generates multiple feature maps after channel shuffling and channel grouping of the input of the module. Each feature map will pass through the axial Transformer respectively, and the results will be connected in the channel dimension to obtain the output feature map.

[0031] The spatial gated feedforward network is a feedforward network designed to utilize local context information and global spatial feature information, and realizes information fusion through three paths. This mechanism consists of three paths: a local context information path, a local context information gating path, and a global spatial feature information gating path. The feature map first undergoes a 1×1 convolution to expand the number of channels to three times the input, and then undergoes a 3×3 depth convolution to be divided into three groups of feature maps F1, F2, and F3. F2 passes through the Gaussian Error Linear Unit (GELU) and then passes through the Sigmoid function to map the value to the (0, 1) interval. The result is element-wise multiplied with F1 to obtain F11. F3 passes through GELU to perform spatial attention calculation. Specifically, the maximum value and average value of each pixel on all channels are calculated respectively, and the results are added to obtain the feature map FS3. FS3 uses the Sigmoid function to obtain the spatial weight. After F11 and FS3 are element-wise multiplied, the result is passed through a 1×1 convolution to make the number of channels consistent with the number of input channels as the output of the CGAT module.

[0032] As a preferred solution of the present invention, an image enhancement method based on multi-view input and structure guidance, the multi-view enhancement joint loss function L in S2 is mve :

[0033] Joint loss function L MVE It mainly consists of two parts: L1 loss and perceptual loss, as shown in formula (1). Among them, L1 loss is used to measure the absolute difference between the predicted value and the actual value. It takes the absolute value of the error of each pixel and then calculates the average of these absolute errors, as shown in formula (2). Perceptual loss is based on the intermediate layer features of the pre-trained deep network (such as VGG) and is used to measure the difference between the generated image and the target image in the feature space rather than the pixel-level difference. It emphasizes the high-level semantics and details of the image, making the generated image visually closer to the real image, as shown in formula (3).

[0034] Loss mve =α1Loss per +α2Loss l1 (1)

[0035]

[0036] Where Loss peris the perceptual loss, Loss l1 is the L1 loss, φ is the VGG16 network pre-trained on the ImageNet dataset, and H and W are the height and width of the feature map, respectively. is the reconstructed image, and I is the true value label. and They are the values ​​of the corresponding pixels of the predicted image and the true value label in the pre-trained VGG16 network, respectively. and I(m,n) are the pixel values ​​of the predicted image and the true value label respectively. Where α1 and α2 are L1 loss and perceptual loss in the joint loss function L mve The weight of .

[0037] As a preferred solution of the present invention, an image enhancement method based on multi-view input and structure guidance, the structure-guided enhancement network in S3:

[0038] The structure-guided enhancement network consists of two parts: a structure extraction module and a U-shaped network with an encoder-decoder structure.

[0039] In the structure extraction module, the module uses the white balance WB (White Balance) and contrast limited adaptive histogram equalization CLAHE (Contrast Limited Adaptive Histogram Equalization) algorithms to process low-quality images, so that the color consistency and contrast of the image are improved to a certain extent, and then uses the edge detection algorithm (such as Canny edge detection) to extract the structural information of the image.

[0040] The U-shaped network has two inputs, namely the structure information extracted by the structure extraction module and the multi-view input enhancement result OUT mve . In the encoder part, there are three layers of network structure, each layer of the network includes a convolution block and a downsampling module, but the bottom layer only contains a structure-guided enhancement module; correspondingly, the decoder part also has a three-layer network structure, each layer of the network includes a structure-guided enhancement module and an upsampling module, and the top layer only contains a structure-guided enhancement module. The decoder will use the structural feature information to guide the enhancement starting from the bottom layer. Specifically, each layer connects the input feature map with the output feature map of the corresponding layer of the encoder in the channel dimension, and the result and the structural information are passed through the structure-guided enhancement module and the upsampling module to obtain the output of the layer. Finally, the top layer of the decoder obtains the structure-guided enhancement result OUT sge .

[0041] As a preferred solution of the present invention, an image enhancement method based on multi-view input and structure guidance, the structure guidance enhancement module in S3:

[0042] The structure-guided enhancement module consists of a multi-layer structure, including a structure-guided self-attention layer, an instance normalization layer, and a spatial gated feedforward network layer, and has two inputs, namely the input feature map and the structural information.

[0043] For the structure-guided self-attention layer, specifically, the input feature map is first normalized by instance normalization to obtain the feature map S, and then the query matrix Q and the key matrix K are obtained by 3×3 deep convolution; the structural feature information is passed through the multi-layer perceptron MLP (Multilayer Perceptron) to obtain two sets of feature maps α and γ, and the numerical matrix V is calculated by the formula S×(1+α)+γ. The Q matrix and the K matrix are matrix multiplied to obtain the feature matrix, and then the feature matrix is ​​matrix multiplied with V, and the result is reshaped to make it consistent with the input feature matrix. Figure 1 After that, it is element-wise added to the input feature map to get the output.

[0044] As a preferred solution of the present invention, an image enhancement method based on multi-view input and structure guidance, the structure-guided enhancement joint loss function L in S3 is sge :

[0045] Joint loss function L sge It mainly consists of three parts: L1 loss, perceptual loss and structural similarity index loss, as shown in formula (4). Among them, L1 loss is used to measure the absolute difference between the predicted value and the actual value. It takes the absolute value of the error of each pixel and then calculates the average of these absolute errors, as shown in formula (2). Perceptual loss is based on the intermediate layer features of the pre-trained deep network (such as VGG) and is used to measure the difference between the generated image and the target image in the feature space rather than the pixel-level difference. It emphasizes the high-level semantics and details of the image, making the generated image visually closer to the real image, as shown in formula (3). Structural similarity index loss is a loss function that evaluates the similarity of images by comparing the brightness, contrast and structural information of the local area of ​​the image. The main goal is to ensure that the generated image is consistent with the reference image in terms of structure, brightness and contrast, thereby improving the quality and visual effect of the image, as shown in formula (6).

[0046] Loss sge =β1Loss per +β2Loss l1 +β3Loss ssim (4)

[0047]

[0048] Where Loss per is the perceptual loss, Loss l1is the L1 loss, Loss ssim is the structural similarity loss, x represents each pixel value, and To predict the mean and standard deviation of the corresponding patch in the image, similarly, μ I (x) and σ I (x) is the mean and standard deviation of the image patch corresponding to the true value label I, and is the cross covariance. Where c1 and c2 are stability constants, β1, β2 and β3 are the weights of L1 loss, perceptual loss and structural similarity index loss in the joint loss function L1.

[0049] As a preferred solution of the present invention, an image enhancement method based on multi-view input and structure guidance, the enhancement network based on multi-view input and structure guidance in S4:

[0050] The enhancement network based on multi-view input and structure guidance consists of two parts, namely the multi-view input enhancement network and the structure guidance enhancement network.

[0051] Specifically, the multi-view input enhancement network and the structure-guided enhancement network use the corresponding optimal model weights saved after their training optimization, and the original input and its corresponding WB and CLAHE images are passed through the multi-view input enhancement network to obtain the multi-view input enhancement result OUT mve , the original input and OUT mve After the structure-guided enhancement network, the structure-guided enhancement result OUT is obtained sge ,OUT sge This is the final image enhancement result.

[0052] Preferably, an image enhancement device based on multi-view input and structure guidance comprises at least one processor and a memory; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above-mentioned image enhancement methods based on multi-view input and structure guidance.

[0053] The beneficial effects of the present invention are:

[0054] (1) The present invention uses multi-view input to provide additional contextual information and constraints. Combined with different viewpoint knowledge, it can specifically help the enhanced model to more accurately restore details, reduce noise and artifacts, and improve image contrast, clarity, and visual effects. In particular, it can enhance the robustness of the model when processing complex scenes or low-quality images.

[0055] (2) The cross-channel attention fusion module (CCAFM) proposed in the present invention uses the self-attention calculation method of the channel dimension for multi-view input, so that multiple features can be better integrated, and finally channel shuffle is performed to improve the feature expression ability and enhance the generalization ability of the model.

[0056] (3) The channel group axial transformer (CGAT) module proposed in the present invention reduces the computational complexity of the self-attention calculation process by grouping the channel dimensions, thereby speeding up the reasoning speed. And through channel shuffle, the possibility of local channel combination is prevented. At the same time, the spatial gated feedforward network (SGFFN) effectively makes up for the lack of attention to the image spatial information during the calculation of the axial self-attention.

[0057] (4) The structure-guided enhancement network (SGEN) proposed in the present invention performs secondary guided enhancement on the image through the structural information of the image. The structural information can enhance the details of the image, especially for areas with low signal-to-noise ratio, to help produce clear and realistic results. Furthermore, edge information helps to distinguish different areas and establish better connections between these areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings of the embodiments are briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present invention, but are not intended to limit the present invention.

[0059] Figure 1 A flowchart of the method proposed by the present invention;

[0060] Figure 2 A network structure diagram of the method proposed in the present invention;

[0061] Figure 3 This is a structural diagram of the channel group axial Transformer module of the method proposed in the present invention;

[0062] Figure 4 This is a structural diagram of the structure-guided enhancement module of the method proposed in the present invention;

[0063] Figure 5 This is a diagram of the spatial gated feedforward network structure of the method proposed in the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0066] like Figure 1 , an image enhancement method based on multi-view input and structure guidance, comprising the following steps:

[0067] S1: Using the UIEB underwater image enhancement benchmark dataset, 800 pairs of images in the dataset are divided into a training set, and the remaining 90 pairs of images are divided into a test set;

[0068] S2: Construct a multi-view input enhancement network, which is used to perform preliminary enhancement on low-quality images; and design a multi-view enhancement joint loss function L mve , specify the training rounds, train the multi-view input enhancement network model for 600 rounds, and save the weights of the optimal model.

[0069] like Figure 2 The multi-view input enhancement network includes a multi-view input (the original input and its corresponding white balance and contrast limited adaptive histogram equalization image), a cross-channel attention feature fusion (CCAF, Channel-CrossAttention Fusion) module, and a channel group axial Transformer (CGAT, Channel-Group AxialTransformer) module; the multi-view enhancement joint loss function L mve Contains L1 loss and perceptual loss;

[0070] The multi-view input enhancement network consists of two parts: multi-view input feature fusion and an asymmetric encoder-decoder structure network.

[0071] Multi-view input feature fusion part, multi-view input is used to provide additional contextual information and constraints to improve image quality. In order to solve the problem of severe color imbalance and low contrast of underwater images, the contrast, color and visual effects of the image are improved by combining the two traditional enhancement algorithms WB and CLAHE. The three inputs with dimensions of 256×256×3 are mapped to a new feature space through the same convolutional layer, and the three sets of feature maps S1, S2, and S3 with shapes of (64, 256, 256) are obtained. S1, S2, and S3 are fused using the cross-channel attention feature fusion module to obtain the feature fusion result FU;

[0072] like Figure 3 In the encoder part, the feature fusion result FU is used as the input of the three-layer network in the encoder. Each layer of the network is composed of channel group axial Transformer. The output of each layer will be element-wise added with the feature maps S1, S2, and S3 of the corresponding layer. The result is downsampled twice, so that each layer obtains three feature maps of different scales. Then, the feature maps of the same scale of different layers are connected in the channel dimension to obtain three groups of feature maps of different scales. They are FS1, FS2, and FS3 in ascending scale, and their shapes are (384, 64, 64), (192, 128, 128), and (192, 256, 256), respectively.

[0073] In the decoder part, each layer of the decoder is composed of a squeeze-and-excitation network and a channel group axial Transformer module. The three feature maps FS1, FS2 and FS3 of the encoder are the inputs of the bottom, middle and top layers of the decoder respectively. Each feature map will first pass through the squeeze-and-excitation network of the corresponding layer to obtain three feature maps FS11, FS21 and FS31. Starting from the bottom layer, FS11 is upsampled after passing through the CGAT module, and the result is connected with FS21 in the channel dimension. The result is put into the middle-layer CGAT module and upsampled, and the result is connected with FS31 in the channel dimension. After passing through the top-layer CGAT module and convolutional layer, the first-stage enhancement result is obtained.

[0074] The cross-channel attention feature fusion module is based on the channel self-attention mechanism, which can capture the relationship between different channels of multi-input feature maps, thereby enhancing the model's ability to express features.

[0075] Specifically, three multi-view input feature maps with shapes of (64, 256, 256) are connected in the channel dimension to obtain a feature map S1 with a shape of (192, 256, 256). After 1×1 convolution and 3×3 depth convolution, the feature map is reshaped to (192, 65536) to obtain Q, K, and V matrices. The Q matrix and the K matrix are matrix multiplied to obtain a feature matrix with a shape of (192, 192). The feature matrix is ​​then matrix multiplied with V, and the result is element-wise added to the feature map S1. Finally, after 1×1 convolution and channel shuffle, the feature fusion result is obtained. The channel self-attention mechanism can dynamically adjust the weights of different channels, so that the model can better handle complex input data, improving the model's expressiveness and performance; the purpose of channel shuffle is to improve the feature expressiveness and enhance the generalization ability of the model.

[0076] The channel group axial self-attention module consists of channel shuffling, channel grouping and axial transformer. The axial transformer consists of a multi-layer structure, including an instance normalization layer, an axial self-attention layer and a spatial gated feedforward network layer. Channel shuffling is intended to avoid locality caused by channel grouping, and channel grouping is intended to reduce the computational complexity of self-attention calculation.

[0077] Specifically, the channel group axial self-attention module will generate multiple feature maps after channel shuffling and channel grouping. Each feature map will perform axial self-attention calculation separately, and the results will be connected in the channel dimension to obtain the output feature map.

[0078] like Figure 5 , the spatial gated feedforward network module is a feedforward network designed to utilize local context information and global spatial feature information, and realizes information fusion through three paths. This mechanism consists of three paths: a local context information path, a local context information gating path, and a global spatial feature information gating path. In the transformer, the traditional feedforward network maps the input features to a higher-dimensional feature space through its stacked fully connected layers, thereby extracting and representing complex patterns and features, but relatively speaking, the traditional feedforward network has no direct perception of the spatial layout and local features in the input data, and for image enhancement, both spatial layout and local feature information are of critical reference significance. The three paths of the spatial gated feedforward network module are designed to focus on the spatial layout and local feature information.

[0079] Specifically, the feature map is divided into three groups of feature maps F1, F2, and F3 after 3×3 deep convolution. F2 is activated by the Gaussian Error Linear Unit (GELU) and then the Sigmoid function is used. The result is element-wise multiplied with F1 to obtain F11. F3 is subjected to spatial attention calculation after GELU. Specifically, the maximum and average values ​​of each pixel in all channels are calculated respectively, and the results are added to obtain the feature map FS3. FS3 uses the Sigmoid function to obtain the spatial weight. After element-wise multiplication of F11 and FS3, the result is used as the output of the CGAT module.

[0080] The multi-view enhancement joint loss function L mve It mainly consists of two parts: L1 loss and perceptual loss, as shown in formula (1). Among them, L1 loss is used to measure the absolute difference between the predicted value and the actual value. It takes the absolute value of the error of each pixel and then calculates the average of these absolute errors, as shown in formula (2). Perceptual loss is based on the intermediate layer features of the pre-trained deep network (such as VGG) and is used to measure the difference between the generated image and the target image in the feature space rather than the pixel-level difference. It emphasizes the high-level semantics and details of the image, making the generated image visually closer to the real image, as shown in formula (3).

[0081] Loss mve =α1Loss per +α2Loss l1 (1)

[0082]

[0083] Where Loss per is the perceptual loss, Loss l1 is the L1 loss, φ is the VGG16 network pre-trained on the ImageNet dataset, and H and W are the height and width of the feature map, respectively. is the reconstructed image, and I is the true value label. Where α1 and α2 are the weights of L1 loss and perceptual loss in the joint loss function L1, which are 0.5 and 1 respectively in this embodiment.

[0084] S3: Construct a structure-guided enhancement network, which performs secondary enhancement on image data based on the structural information extracted by the structure extraction module; and design a joint loss function L for structure-guided enhancement. sge , specify the training rounds, train the structure-guided enhanced network model for 600 rounds, and save the weights of the optimal model.

[0085] The structure-guided enhancement network consists of two parts: a structure extraction module and a U-shaped network of an encoder-decoder structure, including a structure extraction module and a structure-guided enhancement module (SGEM); the structure-guided enhancement joint loss function L sge Contains L1 loss, perceptual loss and structural similarity index loss;

[0086] The structure extraction module uses the white balance (WB) and contrast limited adaptive histogram equalization (CLAHE) algorithms to process low-quality images, thereby improving the color consistency and contrast of the image to a certain extent, and then uses Canny edge detection to extract the structural information of the image.

[0087] The U-shaped network has two inputs, namely the structure information extracted by the structure extraction module and the multi-view input enhancement result OUT mve . In the encoder part, there are three layers of network structure, each layer of the network includes a convolution block and a downsampling module, but the bottom layer only contains a structure-guided enhancement module; correspondingly, the decoder part also has a three-layer network structure, each layer of the network includes a structure-guided enhancement module and an upsampling module, and the top layer only contains a structure-guided enhancement module. The decoder will use the structural feature information to guide the enhancement starting from the bottom layer. Specifically, each layer connects the input feature map with the output feature map of the corresponding layer of the encoder in the channel dimension, and the result and the structural information are passed through the structure-guided enhancement module and the upsampling module to obtain the output of the layer. Finally, the structure-guided enhancement result OUT is obtained from the top layer of the decoder. sge .

[0088] like Figure 4 The structure-guided enhancement module is composed of a multi-layer structure, including a structure-guided self-attention layer, an instance normalization layer, and a spatial gated feedforward network layer, and has two inputs, namely, the input feature map and the structure information. Specifically, the feature map is first normalized by instance normalization to obtain the feature map S, and then the Q and K matrices are obtained by 3×3 deep convolution; the structural feature information is passed through the multi-layer perceptron MLP (Multilayer Perceptron) to obtain two sets of feature maps α and γ, and the V matrix is ​​calculated by the formula S×(1+α)+γ, and the Q, K, and V matrices are output by the self-attention calculation method.

[0089] The structure-guided enhanced joint loss function L sgeIt mainly consists of three parts: L1 loss, perceptual loss and structural similarity index loss, as shown in formula (4). Among them, L1 loss is used to measure the absolute difference between the predicted value and the actual value. It takes the absolute value of the error of each pixel and then calculates the average of these absolute errors, as shown in formula (2). Perceptual loss is based on the intermediate layer features of the pre-trained deep network (such as VGG) and is used to measure the difference between the generated image and the target image in the feature space rather than the pixel-level difference. It emphasizes the high-level semantics and details of the image, making the generated image visually closer to the real image, as shown in formula (3). Structural similarity index loss is a loss function that evaluates the similarity of images by comparing the brightness, contrast and structural information of the local area of ​​the image. The main goal is to ensure that the generated image is consistent with the reference image in terms of structure, brightness and contrast, thereby improving the quality and visual effect of the image, as shown in formula (6).

[0090] Losssge=β1Loss per +β2Loss l1 +β3Loss ssim (4)

[0091]

[0092] Where Loss per is the perceptual loss, Loss l1 is the L1 loss, Loss ssim is the structural similarity loss, x represents each pixel value, and is the mean and standard deviation of the predicted image patch. Similarly, μ I (x) and σ I (x) is the mean and standard deviation of the image patch corresponding to the true value label I, and is the cross covariance. c1 and c2 are stability constants, which are 0.02 and 0.03 respectively in this embodiment, β1, β2 and β3 are the weights of L1 loss, perceptual loss and structural similarity index loss in the joint loss function L1, which are 10, 1 and 1 respectively in this embodiment.

[0093] S4: Construct an enhancement network based on multi-view input and structure guidance. The network consists of two parts: a multi-view input enhancement network and a structure-guided enhancement network.

[0094] Specifically, the multi-view input enhancement network and the structure-guided enhancement network use the corresponding optimal model weights saved after their training optimization. The original input and its corresponding WB and CLAHE images are passed through the multi-view input enhancement network to obtain the multi-view input enhancement result OUT mve , the original input and OUT mve After the structure-guided enhancement network, the structure-guided enhancement result OUT is obtainedsge ,OUT sge This is the final image enhancement result.

[0095] S5: Test model performance.

Claims

1. An image enhancement method based on multi-view input and structure guidance, characterized in that: The following steps are involved: S1: Collect low-quality images and corresponding high-quality images to form a data set; divide the data set into a training set, a validation set, and a test set; S2: Construct a multi-view input enhancement network, which is used to perform preliminary enhancement on low-quality images; And design a multi-view enhanced joint loss function L mve , specify the training rounds, train and optimize the multi-view input enhancement network model, and save the weights of the optimal model; The multi-view input enhancement network includes multi-view input, cross-channel attention feature fusion module, channel group axial Transformer module; the multi-view enhancement joint loss function L mve Contains L1 loss and perceptual loss; S3: Construct a structure-guided enhancement network, which performs secondary enhancement on image data based on the structural information extracted by the structure extraction module; and design a joint loss function L for structure-guided enhancement. sge , specify the training rounds, train and optimize the structure-guided enhanced network model, and save the weights of the optimal model; The structure-guided enhancement network includes a structure extraction module and a structure-guided enhancement module; the structure-guided enhancement joint loss function L sge Contains L1 loss, perceptual loss and structural similarity index loss; S4: Construct an enhancement network combining multi-view input and structure guidance. This network combines the trained multi-view input enhancement network and the structure guidance enhancement network to obtain the final image enhancement result. S5: Test model performance.

2. The image enhancement method based on multi-view input and structure guidance according to claim 1, characterized in that: In S2, the multi-view input enhancement network is: The multi-view input enhancement network consists of two parts: multi-view input feature fusion and an asymmetric encoder-decoder structure network; Multi-view input feature fusion part: The multi-view input includes the original input and its corresponding WB and CLAHE images. The multi-view input data is mapped to a new feature space through the same convolutional layer. The three sets of feature maps are fused using the cross-channel attention feature fusion module to obtain the feature fusion result FU; The encoder part has three layers of networks, and the feature fusion result FU is used as the input of the three-layer network of the encoder; each layer of the network is composed of CGAT modules, and the output of each layer will be element-wise added with the feature fusion result FU, and the results are downsampled twice, so that each layer of the network obtains three feature maps of different scales, and then the feature maps of the same scale of different layers are connected in the channel dimension to obtain three groups of feature maps of different scales, which are FS1, FS2 and FS3 in increasing scale. The decoder part has three layers. Each layer of the decoder consists of a compression activation network and a CGAT module. The three sets of output feature maps of different scales of the encoder will be used as the input of the three-layer network of the decoder, that is, FS1, FS2 and FS3 are the inputs of the bottom layer, middle layer and top layer of the decoder respectively. Each set of feature maps will first pass through the compression activation network of the corresponding layer to obtain three feature maps FS11, FS21 and FS31. Starting from the bottom layer, FS11 is upsampled after passing through the CGAT module, and its result is connected with FS21 in the channel dimension. The result is put into the middle layer CGAT module and then upsampled. The result is connected with FS31 in the channel dimension. After passing through the top layer CGAT module and convolution layer, the multi-view input enhancement result OUT is obtained. mve .

3. The image enhancement method based on multi-view input and structure guidance according to claim 1, characterized in that: In S2, the cross-channel attention feature fusion module is: The cross-channel attention feature fusion module is based on the channel self-attention mechanism, which can capture the relationship between different channels of multiple input feature maps; N multi-view input feature maps of shape "C, H, W" are connected in the channel dimension to obtain a feature map S1 of shape "N×C, H, W", where C is the number of channels, and H and W are the height and width of the feature map respectively; after 1×1 convolution and 3×3 depth convolution, the feature map S1 is reshaped to "N×C, H×W", and the query matrix Q, key matrix K, and numerical matrix V are obtained. The Q matrix and the K matrix are matrix multiplied to obtain a feature matrix of shape "N×C, N×C", and then the feature matrix is ​​matrix multiplied with V, and the result is element-wise added to the feature map S1. Finally, the feature fusion result is obtained after 1×1 convolution and channel shuffling.

4. The image enhancement method based on multi-view input and structure guidance according to claim 1, characterized in that: In S2, the channel group axial Transformer module is: The channel group axial self-attention module consists of channel shuffling, channel grouping, and axial Transformer. The axial Transformer consists of a multi-layer structure, including an instance normalization layer, an axial self-attention layer, and a spatial gated feedforward network layer. The channel group axial self-attention module generates multiple feature maps after shuffling and grouping the input of this module. Each feature map will pass through the axial Transformer respectively, and the results will be connected in the channel dimension to obtain the output feature map; The spatial gated feedforward network is a feedforward network that uses local context information and global spatial feature information to achieve information fusion through three paths; this mechanism consists of three paths: a local context information path, a local context information gating path, and a global spatial feature information gating path; the feature map first undergoes a 1×1 convolution to expand the number of channels to three times the input, and then undergoes a 3×3 depth convolution to be divided into three groups of feature maps F1, F2, and F3. F2 undergoes a Gaussian error linear activation function GELU and then a Sigmoid function to map the value to the (0, 1) interval, and the result is element-wise multiplied with F1 to obtain F11, and F3 undergoes a spatial attention calculation after passing through GELU; The maximum and average values ​​of each pixel in all channels are calculated respectively, and the results are added to obtain the feature map FS3; FS3 uses the Sigmoid function to obtain the spatial weight. After element-wise multiplication of F11 and FS3, the result is subjected to a 1×1 convolution to make the number of channels consistent with the number of input channels and then used as the output of the CGAT module.

5. The image enhancement method based on multi-view input and structure guidance according to claim 1, characterized in that: In S2, the multi-view enhancement joint loss function L mve for: Joint loss function L mve It mainly consists of two parts: L1 loss and perceptual loss, as shown in formula (1); L1 loss is used to measure the absolute difference between the predicted value and the actual value. It takes the absolute value of the error of each pixel and then calculates the average of these absolute errors, as shown in formula (2); perceptual loss is based on the intermediate layer features of the pre-trained deep network and is used to measure the difference between the generated image and the target image in the feature space rather than the pixel-level difference. It emphasizes the high-level semantics and details of the image, making the generated image visually closer to the real image, as shown in formula (3); Loss mve =α1Loss per +α2Loss l1 (1) Where Loss per is the perceptual loss, Loss l1 is the L1 loss, φ is the VGG16 network pre-trained on the ImageNet dataset, H and W are the height and width of the feature map respectively; To reconstruct the image, I is the true value label; and are the values ​​of the corresponding pixels of the predicted image and the ground truth in the pre-trained VGG16 network, respectively; and I(m,n) are the pixel values ​​of the predicted image and the ground truth respectively; where α1 and α2 are the L1 loss and the perceptual loss in the joint loss function L mve The weight of .

6. The image enhancement method based on multi-view input and structure guidance according to claim 1, characterized in that: In S3, the structure-guided enhancement network is: The structure-guided enhancement network consists of two parts: a structure extraction module and a U-shaped network with an encoder-decoder structure; The structure extraction module uses the white balance and contrast-limited adaptive histogram equalization algorithms to process low-quality images, so that the color consistency and contrast of the image are improved to a certain extent, and then uses the edge detection algorithm to extract the structural information of the image; The U-shaped network has two inputs, namely the structure information extracted by the structure extraction module and the multi-view input enhancement result OUT mve ; In the encoder part, there are three layers of network structure, each layer of the network includes a convolution block and a downsampling module, but the bottom layer only contains a structure-guided enhancement module; correspondingly, the decoder part also has a three-layer network structure, each layer of the network includes a structure-guided enhancement module and an upsampling module, and the top layer only contains a structure-guided enhancement module. The decoder will use the structural feature information to guide the enhancement from the bottom layer. Each layer connects the input feature map with the output feature map of the corresponding layer of the encoder in the channel dimension. The result and the structural information are passed through the structure-guided enhancement module and the upsampling module to obtain the output of the layer; finally, the top layer of the decoder obtains the structure-guided enhancement result OUT sge .

7. The image enhancement method based on multi-view input and structure guidance according to claim 1, characterized in that: In S3, the structure-guided enhancement module is: The structure-guided enhancement module consists of a multi-layer structure, including a structure-guided self-attention layer, an instance normalization layer, and a spatial gated feedforward network layer. The structure-guided enhancement module has two inputs, namely the input feature map and the structural information. The structure guides the self-attention layer. Specifically, the input feature map is first normalized by instance to obtain the feature map S, and then the query matrix Q and the key matrix K are obtained by 3×3 deep convolution. The structural feature information is passed through a multi-layer perceptron to obtain two sets of feature maps α and γ, and the numerical matrix V is calculated using the formula S×(1+α)+γ. The Q matrix and the K matrix are matrix multiplied to obtain a feature matrix, which is then matrix multiplied with V. The result is reshaped to make it consistent with the input feature map, and then element-wise addition is performed with the input feature map to obtain the output.

8. The image enhancement method based on multi-view input and structure guidance according to claim 1, characterized in that: In S3, the structure-guided enhanced joint loss function L sge for: Joint loss function L sge It mainly consists of three parts: L1 loss, perceptual loss and structural similarity index loss, as shown in formula (4); among them, L1 loss is used to measure the absolute difference between the predicted value and the actual value. It takes the absolute value of the error of each pixel and then calculates the average of these absolute errors, as shown in formula (2); perceptual loss is based on the intermediate layer features of the pre-trained deep network and is used to measure the difference between the generated image and the target image in the feature space rather than the pixel-level difference. It emphasizes the high-level semantics and details of the image, making the generated image visually closer to the real image, as shown in formula (3); structural similarity index loss is a loss function that evaluates the similarity of images by comparing the brightness, contrast and structural information of the local area of ​​the image. The main goal is to ensure that the generated image is consistent with the reference image in terms of structure, brightness and contrast, thereby improving the quality and visual effect of the image, as shown in formula (6); Loss sge =β1Loss per +β2Loss l1 +β3Loss ssim (4) Where Loss per is the perceptual loss, Loss l1 is the L1 loss, Loss ssim is the structural similarity loss, x represents each pixel value, and To predict the mean and standard deviation of the corresponding patch in the image, μ I (x) and σ I (x) is the mean and standard deviation of the patch corresponding to the true value label I, and is the cross covariance; where c1 and c2 are stability constants, β1, β2 and β3 are the L1 loss, perceptual loss and structural similarity index loss in the joint loss function L sge The weight of .

9. The image enhancement method based on multi-view input and structure guidance according to claim 1, characterized in that: In S4, the enhanced network based on multi-view input and structure guidance is: The enhancement network based on multi-view input and structure guidance consists of two parts: multi-view input enhancement network and structure guidance enhancement network; The multi-view input enhancement network and the structure-guided enhancement network respectively use the corresponding optimal model weights saved after their training optimization. The original input and its corresponding white balance and contrast-constrained adaptive histogram equalization image are passed through the multi-view input enhancement network to obtain the multi-view input enhancement result OUT mve , the original input and OUT mve After the structure-guided enhancement network, the structure-guided enhancement result OUT is obtained sge ,OUT sge This is the final image enhancement result.

10. An image enhancement device based on multi-view input and structure guidance, characterized in that: It includes at least one processor and a memory; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a multi-view input image enhancement method based on structure guidance as described in any one of claims 1 to 9.

Images