A lightweight Transformer image enhancement method based on curve adjustment

Abstract

The present invention discloses a lightweight Transformer image enhancement method based on curve adjustment, comprising the steps of: 1. performing a preprocessing operation on the original input image; 2. inputting the image into a Transformer curve adjustment processing module; 3. pixel-by-pixel adjustment of the first-stage adjustment curve and the original input image; 4. residual connection of the first-stage adjustment residual map and the original input image; 5. pixel-by-pixel adjustment of the second-stage adjustment curve and the first-stage adjustment result map; 6. weighted averaging of the first-stage adjustment residual map and the second-stage adjustment residual map; 7. residual connection of the overall adjustment residual map and the original input image. The present invention can achieve performance superior to that of currently popular methods while having a smaller number of parameters, can realize resolution-independent image enhancement, and exhibits good real-time performance when processing ultra-high-resolution images, with significant effects and easy promotion.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a lightweight Transformer image enhancement method based on curve adjustment. Background Art

[0002] With the widespread adoption of lightweight personal photography equipment, the demand for convenient and fast image enhancement has driven the development of technology in this field, as many non-professionals lack knowledge of image enhancement. Appropriate post-processing of images can significantly enhance their visual appeal and expressiveness, leading to the popularity of one-click image retouching software. For professional photo editors, one-click retouching often falls short, but it can provide preliminary adjustments to image brightness, contrast, and hue, reducing post-processing workload and improving efficiency. Therefore, finding ways to quickly and easily enhance images has become a key research topic.

[0003] In recent years, deep learning-based methods have made significant progress in image enhancement. However, the complex network architecture and computational overhead pose significant challenges to real-time performance. Generally speaking, pixel-level image reconstruction often requires large networks, which inevitably leads to excessively large parameters, resulting in high memory usage and long inference times. Conversely, due to the limited feature extraction capabilities of shallow CNNs, while improving efficiency often fails to balance image enhancement quality, resulting in chromatic aberration or artifacts. CNNs have achieved considerable success in image enhancement due to their local perception and weight sharing. They utilize convolutional kernels to continuously abstract high-level image features. While theoretically, their receptive field can cover the entire image, practical studies have shown that the actual receptive field is significantly smaller than the theoretical one. To achieve a larger receptive field, deeper convolutional layers must be stacked, which significantly increases model size and computational cost. Therefore, designing both high-performance and efficient enhancement methods in the field of image enhancement remains an unresolved issue.

[0004] With the advancement of computing power and the support of big data, learning-based image enhancement methods have achieved astonishing results. Learning-based image processing methods can be broadly categorized into three types: CNN-based image enhancement methods, GAN-based image enhancement methods, and reinforcement learning-based image enhancement methods. CNN-based image enhancement methods achieve image enhancement through overly deep network structures or complex loss function designs, which can lead to poor real-time performance and poor generalization. Image enhancement can also be viewed as a generative task, utilizing GANs to generate high-quality images. GAN-based methods suffer from unstable training processes, and the generated models are generally large, making them difficult to meet real-time requirements. By modeling the image enhancement task as a Markov decision problem, the image enhancement process can be visualized in a white-box manner. Reinforcement learning can white-box the image enhancement process by outputting the usage of enhancement tools. However, the performance of reinforcement learning-based methods is directly limited by the quality of the enhancement tools used.

[0005] The Transformer is a model for handling sequence-to-sequence (Seq2Seq) tasks. It abandons the recurrent structure of RNNs and the convolutional structure of CNNs, instead using multi-head self-attention to capture long-range dependencies. Because Transformer networks can easily capture global dependencies within input sequences, they have achieved excellent results in natural language processing (NLP) tasks. Because image enhancement and text translation can both be abstracted as problems from known sequences to unknown sequences, much research has focused on introducing Transformers into the field of computer vision. While current research demonstrates the impressive performance of Transformers as backbone networks in computer vision, the enormous computational resources required, the large-scale pre-training process, and the large number of parameters required restrict their development towards lightweight models. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to address the deficiencies in the above-mentioned prior art and provide a lightweight Transformer image enhancement method based on curve adjustment. The method has simple steps, reasonable design, and easy implementation. It can be effectively applied to lightweight image enhancement. While having a small number of parameters, it has achieved performance that is superior to current popular methods, can achieve resolution-independent image enhancement, and exhibits good real-time performance when processing ultra-high resolution images. The method has significant effects and is easy to promote.

[0007] To solve the above technical problems, the present invention adopts a technical solution: a lightweight Transformer image enhancement method based on curve adjustment, comprising the following steps:

[0008] Step 1: preprocess the original input image to obtain a one-dimensional image block;

[0009] Step 2: inputting the one-dimensional image block into the Transformer curve adjustment processing module to generate a first-stage adjustment curve and a second-stage adjustment curve;

[0010] Step 3: Perform pixel-by-pixel adjustment on the first-stage adjustment curve and the original input image to obtain a first-stage adjustment residual map;

[0011] Step 4: Perform a residual connection between the first-stage adjusted residual image and the original input image to obtain the first-stage adjusted result image;

[0012] Step 5: Perform pixel-by-pixel adjustment on the second-stage adjustment curve and the first-stage adjustment result image to obtain a second-stage adjustment residual image;

[0013] Step 6: Perform weighted averaging on the first-stage adjusted residual graph and the second-stage adjusted residual graph to obtain an overall adjusted residual graph;

[0014] Step 7: Perform a residual connection between the overall adjustment residual map and the original input image to obtain the final enhanced image.

[0015] In the above-mentioned lightweight Transformer image enhancement method based on curve adjustment, the specific process of preprocessing the original input image in step 1 to obtain a one-dimensional image block includes:

[0016] Step 101: The original input image I∈R H×W×C Compress to obtain an image of fixed resolution size I M ∈R M ×M×C , achieving resolution-independent image enhancement; where C is the number of image channels, (H, W) is the spatial resolution of the input image, and (M, M) is the spatial resolution of the intermediate state image;

[0017] Step 102: The fixed resolution image I M ∈R M×M×C Cropped into image patches of fixed size T∈R ((M / P)×(M / P)×C ) (p×p) , where (p×p) represents the number of input image blocks T, and (M / P)×(M / P) is the resolution of each image block T;

[0018] Step 103: Transform the image block T∈R ((M / P)×(M / P)×C)(p×p) Flatten it into a one-dimensional vector to obtain a one-dimensional image block.

[0019] In the above-mentioned lightweight Transformer image enhancement method based on curve adjustment, the Transformer curve adjustment processing module in step 2 includes a Transformer encoder and a two-stage network adjustment module, and an MLP layer for connecting the Transformer encoder and the two-stage network adjustment module.

[0020] The above-mentioned lightweight Transformer image enhancement method based on curve adjustment, the Transformer encoder includes N multi-head self-attention layers and a feedforward neural network connected by LayerNorm.

[0021] The above-mentioned lightweight Transformer image enhancement method based on curve adjustment, the two-stage network adjustment module includes a first-stage adjustment module and a second-stage adjustment module, the first-stage adjustment module includes a PRelu activation layer and a Linear layer, and the second-stage adjustment module includes a PRelu activation layer and a Linear layer that are circulated twice.

[0022] In the above-mentioned lightweight Transformer image enhancement method based on curve adjustment, the two-stage network adjustment module also includes a curve adjustment function, which includes the position information (x, y) of the pixel value, expressed as:

[0023]

[0024] Among them, Y * represents the adjusted pixel value, I(x,y) is the input pixel value, x and y are the horizontal and vertical coordinates of the pixel respectively, channel represents the image color channel, channel∈{R,G,B}, F is the pixel-by-pixel mapping function, and G(I↓,θ) is the adjustment parameter obtained by the network through downsampling the image.

[0025] The above-mentioned lightweight Transformer image enhancement method based on curve adjustment, in the two-stage network adjustment module, curve nodes are generated through the network and then interpolated into the target range.

[0026] In the above-mentioned lightweight Transformer image enhancement method based on curve adjustment, the two-stage network adjustment module also includes a loss function, which is:

[0027]

[0028] Among them, loss(x,y) represents the deviation between the input x and the true value y, y i is the true value, f(x i) is the predicted value, and n is the function calculation setting value.

[0029] In the above-mentioned lightweight Transformer image enhancement method based on curve adjustment, the second stage in step 5 uses a rectified linear unit on the residual map. The specific process is expressed as follows:

[0030] R one_stage =F(I;E one_stage )

[0031] Y one_stage =I+R one_stage

[0032] R two_stage =F(Y one_stage ;E two_stage )

[0033] relu(x)=max(0,x)

[0034] Y * =I+α×R one_stage +relu((1-α)R two_stage )

[0035] Where I represents the original input image, E one_stage represents the first stage adjustment curve, F represents the pixel-by-pixel mapping function, R one_stage Represents the first stage adjusted residual map, Y one_stage It represents the result of the first stage adjustment, E two_stage represents the second stage adjustment curve, R two_stage Represents the second-stage adjusted residual graph, relu(x) represents the corrected linear function, and α is the weight of the two stages.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] 1. The method of the present invention has simple steps, reasonable design and easy implementation.

[0038] 2. The present invention uses the Transformer encoder for image enhancement, achieving a balance between lightweight, real-time and high quality.

[0039] 3. The position information of the pixel value is added to the curve adjustment function of the two-stage network adjustment module of the present invention. With reference to the pixel values ​​of adjacent channels, the adjustment range of the pixel value in a certain area can be limited to prevent abnormal fluctuations in the adjustment curve.

[0040] 4. The present invention generates curve nodes through the network and then interpolates to the target range, avoiding the generation of full-resolution adjustment parameters and reducing the amount of calculation.

[0041] 5. The present invention adopts a two-stage network adjustment module, which can avoid the difficulty of adjusting the overall image style and enhancing image details due to single-stage adjustment.

[0042] 6. The present invention can be effectively applied to lightweight image enhancement. While having a smaller number of parameters, it achieves performance that is superior to currently popular methods, can realize resolution-independent image enhancement, and exhibits good real-time performance when processing ultra-high resolution images. The effect is significant and easy to promote.

[0043] In summary, the method of the present invention has simple steps, reasonable design, and easy implementation. It can be effectively applied to lightweight image enhancement. While having a small number of parameters, it has achieved performance that is superior to current popular methods. It can realize resolution-independent image enhancement and exhibits good real-time performance when processing ultra-high resolution images. The effect is significant and easy to promote.

[0044] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of the method of the present invention;

[0046] Figure 2 Schematic diagram of the network structure of the present invention. DETAILED DESCRIPTION

[0047] like Figure 1 and Figure 2 As shown, the lightweight Transformer image enhancement method based on curve adjustment of the present invention includes the following steps:

[0048] Step 1: preprocess the original input image to obtain a one-dimensional image block;

[0049] Step 2: inputting the one-dimensional image block into the Transformer curve adjustment processing module to generate a first-stage adjustment curve and a second-stage adjustment curve;

[0050] Step 3: Perform pixel-by-pixel adjustment on the first-stage adjustment curve and the original input image to obtain a first-stage adjustment residual map;

[0051] Step 4: Perform a residual connection between the first-stage adjusted residual image and the original input image to obtain the first-stage adjusted result image;

[0052] Step 5: Perform pixel-by-pixel adjustment on the second-stage adjustment curve and the first-stage adjustment result image to obtain a second-stage adjustment residual image;

[0053] Step 6: Perform weighted averaging on the first-stage adjusted residual graph and the second-stage adjusted residual graph to obtain an overall adjusted residual graph;

[0054] Step 7: Perform a residual connection between the overall adjustment residual map and the original input image to obtain the final enhanced image.

[0055] In this embodiment, the specific process of performing preprocessing on the original input image in step 1 to obtain a one-dimensional image block includes:

[0056] Step 101: The original input image I∈R H×W×C Compress to obtain an image of fixed resolution size I M ∈R M ×M×C , achieving resolution-independent image enhancement; where C is the number of image channels, (H, W) is the spatial resolution of the input image, and (M, M) is the spatial resolution of the intermediate state image;

[0057] Step 102: The fixed resolution image I M ∈R M×M×C Cropped into image patches of fixed size T∈R ((M / P)×(M / P)×C)(p×p) , where (p×p) represents the number of input image blocks T, and (M / P)×(M / P) is the resolution of each image block T;

[0058] Step 103: Transform the image block T∈R ((M / P)×(M / P)×C ) (p×p) Flatten it into a one-dimensional vector to obtain a one-dimensional image block.

[0059] In this embodiment, the Transformer curve adjustment processing module in step 2 includes a Transformer encoder and a two-stage network adjustment module, and an MLP layer for connecting the Transformer encoder and the two-stage network adjustment module.

[0060] In practice, this invention employs a Transformer encoder for image enhancement, achieving a balance between lightweight, real-time, and high-quality performance. Specifically, it employs a curve-based color transformation function method, obtaining adjustment parameters through a network to achieve image-to-parameter mapping. This method exhibits good real-time performance when processing ultra-high-resolution images.

[0061] In this embodiment, the Transformer encoder includes N multi-head self-attention layers and a feedforward neural network connected by LayerNorm.

[0062] In this embodiment, the two-stage network adjustment module includes a first-stage adjustment module and a second-stage adjustment module, the first-stage adjustment module includes a PRelu activation layer and a Linear layer, and the second-stage adjustment module includes a PRelu activation layer and a Linear layer that are cycled twice.

[0063] In specific implementation, the entire network structure model is represented as:

[0064] T0=T0+p

[0065] T'=LN(MSA(T n-1 ))+T n-1 ,n=1...N

[0066] T"=LN(FF(T n ' -1 ))+T n ' -1 ,n=1...N

[0067] y = MLP(T")

[0068] y one =Liner(PRelu(y))

[0069] y two =Liner(PRelu(PRelu(y)))

[0070] Among them, N, T0, p represent the Transformer depth, input token sequence and position encoding respectively, T' and T ” is the intermediate state, MSA, LN, FF, and MLP represent the multi-head self-attention layer, layer normalization, feedforward neural network, and multi-layer perceptron, respectively. Liner and PRelu represent the linear connection layer and PRelu activation layer, respectively. y is the network parameter generated by MLP, and y one and y two They represent the adjustment parameters generated by the two-stage network.

[0071] In this embodiment, the two-stage network adjustment module further includes a curve adjustment function, which includes the position information (x, y) of the pixel value and is expressed as:

[0072]

[0073] Among them, Y * represents the adjusted pixel value, I(x,y) is the input pixel value, x and y are the horizontal and vertical coordinates of the pixel respectively, channel represents the image color channel, channel∈{R,G,B}, F is the pixel-by-pixel mapping function, and G(I↓,θ) is the adjustment parameter obtained by the network through downsampling the image.

[0074] In specific implementation, the position information of the pixel value is added to the curve adjustment function of the two-stage network adjustment module. With reference to the pixel values ​​of adjacent channels, the adjustment range of the pixel value in a certain area can be limited to prevent abnormal fluctuations in the adjustment curve.

[0075] In this embodiment, in the two-stage network adjustment module, curve nodes are generated through the network and then interpolated into the target range.

[0076] In practice, due to the large range of pixel values ​​and coordinate positions in images, generating full-pixel and full-resolution adjustment parameters requires a massive amount of parameters, which would undoubtedly make the network structure extremely large and consume enormous computing resources. Considering the trade-off between efficiency and effectiveness, curve nodes are generated through the network and then interpolated to the target range to reduce the amount of computation.

[0077] In this embodiment, the two-stage network adjustment module further includes a loss function, which is:

[0078]

[0079] Among them, loss(x,y) represents the deviation between the input x and the true value y, y i is the true value, f(x i ) is the predicted value, and n is the function calculation setting value.

[0080] In this embodiment, the second-stage adjustment residual map in step five uses a rectified linear unit, and the specific process is expressed as follows:

[0081] R one_stage =F(I;E one_stage )

[0082] Y one_stage =I+R one_stage

[0083] R two_stage =F(Y one_stage ;E two_stage )

[0084] relu(x)=max(0,x)

[0085] Y * =I+α×R one_stage +relu((1-α)R two_stage )

[0086] Where I represents the original input image, E one_stage represents the first stage adjustment curve, F represents the pixel-by-pixel mapping function, R one_stageRepresents the first stage adjusted residual map, Y one_stage It represents the result of the first stage adjustment, E two_stage represents the second stage adjustment curve, R two_stage Represents the second-stage adjusted residual graph, relu(x) represents the corrected linear function, and α is the weight of the two stages.

[0087] In practice, a single-stage adjustment process struggles to balance adjusting the overall image style with enhancing image details, failing to meet user requirements. Therefore, a two-stage enhancement curve was designed. The second stage fine-tunes the first-stage enhancement results, generating a residual image rather than directly generating the final adjustment result. Furthermore, the enhancement process from low-quality images to high-quality ones requires more pixel value increases, so the second stage limits the direction of image adjustment.

[0088] The present invention relies on the Transformer encoder to efficiently extract long-distance dependencies between image blocks and the global semantic information of the image. At the same time, the two-stage curve adjustment strategy overcomes the limited adjustment capability of the global adjustment function, enabling the method of the present invention to combine global modification and local fine-tuning. While having a small number of parameters, the present invention performs well in image modification and low-light image enhancement tasks, and can better preserve image texture details and color information.

[0089] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A lightweight Transformer image enhancement method based on curve adjustment, characterized in that: The following steps are involved: Step 1: preprocess the original input image to obtain a one-dimensional image block; Step 2: inputting the one-dimensional image block into the Transformer curve adjustment processing module to generate a first-stage adjustment curve and a second-stage adjustment curve; The Transformer curve adjustment processing module in step 2 includes a Transformer encoder and a two-stage network adjustment module, and an MLP layer for connecting the Transformer encoder and the two-stage network adjustment module; The two-stage network adjustment module includes a first-stage adjustment module and a second-stage adjustment module, wherein the first-stage adjustment module includes a PRelu activation layer and a Linear layer, and the second-stage adjustment module includes a PRelu activation layer and a Linear layer that are circulated twice; Step 3: Perform pixel-by-pixel adjustment on the first-stage adjustment curve and the original input image to obtain a first-stage adjustment residual map; Step 4: Perform a residual connection between the first-stage adjusted residual image and the original input image to obtain the first-stage adjusted result image; Step 5: Perform pixel-by-pixel adjustment on the second-stage adjustment curve and the first-stage adjustment result image to obtain a second-stage adjustment residual image; Step 6: Perform weighted averaging on the first-stage adjusted residual graph and the second-stage adjusted residual graph to obtain an overall adjusted residual graph; Step 7: Perform a residual connection between the overall adjustment residual map and the original input image to obtain the final enhanced image.

2. The lightweight Transformer image enhancement method based on curve adjustment according to claim 1, characterized in that: The specific process of performing preprocessing operations on the original input image in step 1 to obtain a one-dimensional image block includes: Step 101: The original input image I∈R H×W×C Compress to obtain an image of fixed resolution size I M ∈R M×M×C , achieving resolution-independent image enhancement; where C is the number of image channels, (H, W) is the spatial resolution of the input image, and (M, M) is the spatial resolution of the intermediate state image; Step 102: The fixed resolution image I M ∈R M×M×C Cropped into image patches of fixed size T∈R ((M / P)×(M / P)×C)(P×P) , where (P×P) represents the number of input image blocks T, and (M / P)×(M / P) is the resolution of each image block T; Step 103: Transform the image block T∈R ((M / P)×(M / P)×C)(P×P) Flatten it into a one-dimensional vector to get a one-dimensional image block.

3. The lightweight Transformer image enhancement method based on curve adjustment according to claim 1, characterized in that: The Transformer encoder consists of N multi-head self-attention layers and a feedforward neural network connected by LayerNorm.

4. The lightweight Transformer image enhancement method based on curve adjustment according to claim 1, characterized in that: The two-stage network adjustment module also includes a curve adjustment function, which includes the position information (x, y) of the pixel value, expressed as: Among them, Y * represents the adjusted pixel value, I(x,y) is the input pixel value, x and y are the horizontal and vertical coordinates of the pixel respectively, channel represents the image color channel, channel∈{R,G,B}, F is the pixel-by-pixel mapping function, and G(I↓,θ) is the adjustment parameter obtained by the network through downsampling the image.

5. The lightweight Transformer image enhancement method based on curve adjustment according to claim 1, characterized in that: In the two-stage network adjustment module, curve nodes are generated through the network and then interpolated into the target range.

6. The lightweight Transformer image enhancement method based on curve adjustment according to claim 1, characterized in that: The two-stage network adjustment module also includes a loss function, which is: Among them, loss(x,y) represents the deviation between the input x and the true value y, y i is the true value, f(x i ) is the predicted value, and n is the function calculation setting value.

7. The lightweight Transformer image enhancement method based on curve adjustment according to claim 1, characterized in that: In step 5, the second stage of adjusting the residual map uses a rectified linear unit. The specific process is expressed as follows: R one_stage =F(I;E one_stage ) Y one_stage =I+R one_stage R two_stage =F(Y one_stage ;E two_stage ) relu(x)=max(0,x) Y * =I+α×R one_stage +relu((1-α)R two_stage ) Where I represents the original input image, E one_stage represents the first stage adjustment curve, F represents the pixel-by-pixel mapping function, R one_stage Represents the first stage adjusted residual map, Y one_stage It represents the result of the first stage adjustment, E two_stage represents the second stage adjustment curve, R two_stage Represents the second-stage adjusted residual graph, relu(x) represents the corrected linear function, and α is the weight of the two stages.

Images