Lightweight image restoration model based on mixed linear Transform and double knowledge distillation
By constructing a lightweight image restoration model using a hybrid linear Transformer and dual knowledge distillation techniques, the problems of high computational cost and dense parameters in resource-constrained scenarios are solved, achieving efficient image restoration results.
Patent Information
- Application Number
- CN202511067760.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-31
- Publication Date
- 2025-11-07
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing image restoration models based on visual Transformers are computationally intensive and parameter-dense, making it difficult to perform efficient image restoration in resource-constrained scenarios.
We employ a hybrid linear Transformer and dual knowledge distillation techniques to construct a lightweight image restoration model through multi-head spatial linear attention and multi-head channel linear attention, and utilize the dual feature knowledge distillation algorithm to transfer feature knowledge from a powerful teacher network.
This reduces the parameter capacity and inference time of the image restoration model, improving the efficiency of image restoration in resource-constrained scenarios.
Smart Images
Figure CN120912458A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a lightweight image restoration model based on mixed linear Transformer and double knowledge distillation. BACKGROUND
[0002] With the rapid development of consumer and industrial cameras as important information carriers, images are widely used in medical imaging, security monitoring, remote sensing, multimedia communication and other fields. However, in the process of image acquisition, transmission and storage, various undesirable image degradation phenomena such as noise, blur and low illumination will inevitably occur, resulting in a decrease in image quality. Image restoration is an effective solution, which aims to restore high-quality images from degraded images. As a basic task in low-level computer vision, image restoration can greatly promote a wide range of real-world applications such as video surveillance, autonomous driving and remote sensing.
[0003] Due to the non-uniqueness of the solution, image restoration is a challenging ill-posed problem. In order to solve this problem, deep learning-based methods have shown great potential due to their powerful ability to learn general priors. However, due to the diversity of degradation factors, the image restoration task is further specific to image denoising, image deblurring, image defogging, image deraining, image desnowing, low-light enhancement, etc. At present, the image restoration research field is committed to developing a general model to unify various image restoration tasks. In order to achieve this goal, the dominant neural architecture in the image restoration research field has shifted from convolutional neural networks to visual Transformers.
[0004] Many advanced image restoration models are built on the basis of visual Transformers, thanks to their powerful dynamic weighting and global dependency capturing capabilities. The core of these capabilities lies in the multi-head self-attention mechanism. On the one hand, early models adopted spatial attention, which can improve the perception of local image structure by scoring the spatial importance of image regions. On the other hand, researchers introduced channel attention into image restoration models, which can obtain differentiated feature representations by dynamically assigning weights to different channels. Recently, many studies have attempted to combine spatial attention and channel attention to obtain high-performance image restoration models. Although the spatial self-attention mechanism and the channel self-attention mechanism respectively employ local windows and depth separable convolutions to improve computational efficiency, the computation of the self-attention mechanism still has quadratic complexity relative to the input size. Therefore, image restoration models based on visual Transformers are usually computationally intensive and parameter-dense.
[0005] In order to improve the efficiency of the image restoration model in the resource limited scene, the application discloses a lightweight image restoration model processing method based on mixed linear Transformer and double feature knowledge distillation. To this end, the application uses multi-head spatial linear attention and multi-head channel linear attention to design a new mixed linear Transformer and establish a lightweight image restoration model. In addition, in order to further improve the performance of the lightweight image restoration model, the application uses the lightweight image restoration model as the student network, and uses the double feature knowledge distillation algorithm to guide the powerful teacher restoration model to learn. SUMMARY
[0006] The application aims to solve the problems in the prior art and proposes a lightweight image restoration model based on mixed linear Transformer and double knowledge distillation, which is used to further improve the processing efficiency of the image restoration model in the resource limited scene.
[0007] In order to achieve the above purpose, the application adopts the following technical scheme:
[0008] A lightweight image restoration model based on mixed linear Transformer and double knowledge distillation includes the following steps:
[0009] Step 1, establish a powerful image restoration teacher network: use multi-head spatial self-attention and multi-head channel self-attention to build a mixed visual Transformer module, and use the mixed visual Transformer module to establish an image restoration teacher network;
[0010] Step 2, train the image restoration teacher network: establish a multiple loss function, and use gradient descent method to train the image restoration teacher network;
[0011] Step 3, establish a lightweight image restoration model: use multi-head spatial linear attention and multi-head channel linear attention to build a mixed linear visual Transformer module, and use the mixed linear visual Transformer module to establish a lightweight image restoration model;
[0012] Step 4, establish a double feature knowledge distillation algorithm to complete the training of the lightweight image restoration model: the double feature knowledge distillation algorithm transfers the spatial attention feature knowledge and the channel attention feature knowledge from the powerful image restoration teacher network to the lightweight image restoration model.
[0013] Preferably, in step 1, the specific method is as follows:
[0014] S11, establish a bottom visual feature extraction module The module is composed of 3x3 convolutional layers, which perform the following mathematical operations:
[0015] (1)
[0016] In formula (1), denotes any one degraded image, denotes the bottom-level visual features, denotes a 3x3 convolutional layer;
[0017] S12: Establish a deep visual feature extraction module , arranged in a U-shaped structure and mainly composed of an encoder and a decoder; the encoder and the decoder are decomposed into four stages, and each stage is stacked by different numbers of mixed visual Transformer blocks; the encoder uses a down-sampling layer to gradually compress the input data to extract high-level features, while the decoder uses an up-sampling layer to gradually decompress the latent representation to reconstruct the original input size; each mixed visual Transformer block is composed of multi-head spatial self-attention and multi-head channel self-attention, assuming that the input features of each mixed visual Transformer block are represented as , then the signal processing process of the mixed visual Transformer block is represented as:
[0018] (2)
[0019] In formula (2), denotes the output features of the mixed visual Transformer block, , , denotes the intermediate features of the mixed visual Transformer block, denotes multi-head spatial self-attention, multi-head channel self-attention, denotes a regularization operation, denotes a multi-layer perceptron layer;
[0020] Assuming that the number of multi-head attention is M, in , is divided into several non-overlapping local small windows, each window has a size of , and each window is represented as The calculation formula of the mth attention head is as follows:
[0021] (3)
[0022] In formula (3), is a spatial query neural network layer, a spatial key-value neural network layer, a spatial value neural network layer, denotes the feature dimension of is the output of the m-th spatial self-attention; the attention operation in equation (3) is repeated M times, and all the outputs are reorganized and merged together; in the channels of are split into M parts, each part is denoted as The calculation formula of the m-th attention head is as follows:
[0023] (4)
[0024] In equation (4), is a channel query neural network layer, is a channel key-value neural network layer, is a channel value neural network layer, denotes the feature dimension of is the output of the m-th channel self-attention, and the attention operation in equation (4) is repeated M times, and all the outputs are reorganized and merged together;
[0025] S13: An image reconstruction module is established, which processes the output features of the module using a 3x3 convolution layer to obtain a residual image , and the reconstructed image is denoted as:
[0026] (5)
[0027] In equation (5), denotes the output restored image of the image restoration teacher network;
[0028] S14: The bottom-level visual feature extraction module , the deep-level visual feature extraction module and the image reconstruction module are stacked together to form the image restoration teacher network, that is:
[0029] (6)
[0030] In equation (6), denotes a stacking operation operator, and t denotes the image restoration teacher network.
[0031] Preferably, in step 2, the specific method is as follows:
[0032] S21: Randomly extract N images in the data set, and the n-th degraded image is denoted as input into the image restoration teacher network to obtain an output restored image , assuming the corresponding ground truth image is represented as , a reconstruction error loss function between and is established:
[0033] (7)
[0034] in formula (7), represents the L1 norm, and a contrast loss function between and is established, that is:
[0035] (8)
[0036] in formula (8), represents the L1 norm, and a robust loss function between and is established, that is:
[0037] (9)
[0038] in formula (9), represents the L2 norm, represents a constant;
[0039] S22: the reconstruction error loss function, the contrast loss function and the robust loss function are weighted and summed to obtain a multiple loss function, and the mathematical expression is:
[0040] (10)
[0041] in formula (10), is the contrast loss function parameter, is the robust loss parameter, and based on formula (10), the image restoration teacher network is trained by using the gradient descent method.
[0042] Preferably, in step 3, the specific method is as follows:
[0043] S31: a bottom-level visual feature extraction module is established , which is composed of a 3*3 convolution layer and performs the following mathematical operation:
[0044] (11)
[0045] in formula (11), represents any one degraded image, represents a bottom-level visual feature, denotes a 3x3 convolutional layer;
[0046] S32: establishing a deep visual feature extraction module is composed of several mixed linear visual Transformer blocks, each of which is composed of multi-head spatial linear self-attention and multi-head channel linear self-attention, assuming that the input feature of each mixed linear visual Transformer block is denoted as , the signal processing process of the mixed linear visual Transformer block is denoted as:
[0047] (12)
[0048] In formula (12), denotes the output feature of the mixed linear visual Transformer block, denotes the intermediate feature of the mixed linear visual Transformer block, denotes the multi-head spatial self-attention, multi-head channel self-attention, denotes a regularization operation, denotes a multi-layer perceptron layer;
[0049] Assuming that the number of multi-head attention is M, in , the input feature is divided into several non-overlapping local small windows, each window has a size of , each window is denoted as , and the calculation formula of the mth attention head is as follows:
[0050] (13)
[0051] In formula (13), is a linear spatial attention query neural network layer, is a linear spatial attention key-value neural network layer, is a linear spatial attention value neural network layer, denotes a matrix reshaping operator, is the output of the mth spatial self-attention; the attention operation in formula (13) is repeated M times, all outputs are reorganized and combined together, in , the channels of are divided into M parts, each part is denoted as , and the calculation formula of the mth attention head is as follows:
[0052] (14)
[0053] In formula (14), is a linear channel attention query neural network layer, is a linear channel attention key-value neural network layer, is a linear channel attention value neural network layer, denotes a matrix reshaping operator, is the output of the m-th channel self-attention, the attention operation in formula (14) is repeated M times, and all outputs are reorganized and combined together;
[0054] S33: An image reconstruction module is established , which processes the output features of the module using a 3x3 convolution layer to obtain a residual image , then the reconstructed image is represented as:
[0055] (15)
[0056] In formula (15), denotes the output restored image of the image restoration teacher network;
[0057] S34: A bottom-level visual feature extraction module , a deep-level visual feature extraction module and an image reconstruction module are stacked together to form an image restoration teacher network, that is:
[0058] (16)
[0059] In formula (16), denotes a stacking operation operator, and s denotes a lightweight image restoration model.
[0060] Preferably, in step 4, the specific method is as follows:
[0061] S41: Randomly extract N images in the data set, and the n-th degraded image is represented , which is input into the lightweight image restoration model network s constructed in step 3 to obtain the output restored image , assuming the corresponding true value image is represented as , the reconstruction error loss function between and is established:
[0062] (17)
[0063] In formula (17), denotes the L1 norm;
[0064] S42: input the degraded image to the image restoration teacher network t constructed in step 1 to obtain an output restored image , and an output distillation loss function between the lightweight image restoration model and the image restoration teacher network is established, that is:
[0065] (18)
[0066] In formula (18), denotes the L1 norm;
[0067] S43: it is known from step S12 that the feature output of the module in the image restoration teacher network t is , it is known from step S32 that the feature output of the module in the lightweight student network s is , and a multi-head spatial attention feature knowledge distillation loss function is established, that is:
[0068] (19)
[0069] In formula (19), denotes the L1 norm;
[0070] S44: it is known from step S12 that the feature output of the module in the image restoration teacher network t is , it is known from step S32 that the feature output of the module in the lightweight student network s is , and a multi-head channel attention feature knowledge distillation loss function is established, that is:
[0071] (20)
[0072] In formula (20), denotes the L1 norm;
[0073] S45: the reconstruction error loss function, the output distillation loss function, the multi-head spatial attention feature knowledge distillation loss function and the multi-head spatial attention feature knowledge distillation loss function are weighted and summed to obtain a double feature knowledge distillation loss function, and the mathematical expression is:
[0074] (21)
[0075] In formula (21), is an output knowledge distillation loss adjustable parameter, is a multi-head spatial attention feature knowledge distillation loss adjustable parameter, The multi-head channel attention feature knowledge distillation loss adjustable parameter is trained by using the gradient descent method based on formula (21), and the trained lightweight image restoration model is deployed online to perform an image restoration test task.
[0076] Compared with the prior art, the present application has the following beneficial effects:
[0077] The present application can simultaneously use multi-head spatial linear attention and multi-head channel linear attention to construct a lightweight image restoration model, thereby reducing the parameter capacity and inference time of the image restoration model and improving the efficiency of the image restoration model in a resource-limited scenario. BRIEF DESCRIPTION OF DRAWINGS
[0078] Figure 1 The flowchart of the present application. DETAILED DESCRIPTION
[0079] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings, so that those skilled in the art can better understand the advantages and features of the present application, and the protection scope of the present application can be more clearly defined. The described embodiments of the present application are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.
[0080] A lightweight image restoration model based on mixed linear Transformer and double knowledge distillation includes the following steps:
[0081] Step 1, establishing a powerful image restoration teacher network: simultaneously using multi-head spatial self-attention and multi-head channel self-attention to construct a mixed visual Transformer module, and using the mixed visual Transformer module to establish an image restoration teacher network;
[0082] Step 2, training the image restoration teacher network: establishing a multi-loss function, and training the image restoration teacher network by using the gradient descent method;
[0083] Step 3, establishing a lightweight image restoration model: simultaneously using multi-head spatial linear attention and multi-head channel linear attention to construct a mixed linear visual Transformer module, and using the mixed linear visual Transformer module to establish a lightweight image restoration model;
[0084] Step 4, training the lightweight image restoration model by the double feature knowledge distillation algorithm: the double feature knowledge distillation algorithm transfers the spatial attention feature knowledge and the channel attention feature knowledge from the powerful image restoration teacher network to the lightweight image restoration model.
[0085] Specifically, in step 1, the specific method is as follows:
[0086] S11: Establishing a bottom-level visual feature extraction module The module is composed of 3x3 convolutional layers, which performs the following mathematical operations:
[0087] (1)
[0088] In formula (1), denotes any degraded image, denotes the bottom-level visual feature, denotes a 3x3 convolutional layer;
[0089] S12: Establishing a deep-level visual feature extraction module , arranged in a U-shaped structure, mainly composed of an encoder and a decoder; the encoder and the decoder are decomposed into four stages, each stage is stacked by different numbers of mixed visual Transformer blocks; the encoder uses down-sampling layers to gradually compress the input data to extract high-level features, while the decoder uses up-sampling layers to gradually decompress the latent representation to reconstruct the original input size; each mixed visual Transformer block is composed of multi-head spatial self-attention and multi-head channel self-attention, assuming that the input feature of each mixed visual Transformer block is represented as , then the signal processing process of the mixed visual Transformer block is represented as:
[0090] (2)
[0091] In formula (2), denotes the output feature of the mixed visual Transformer block, , , denotes the intermediate feature of the mixed visual Transformer block, denotes multi-head spatial self-attention, multi-head channel self-attention, denotes a regularization operation, denotes a multi-layer perceptron layer;
[0092] Assuming that the number of multi-head attention is M, in , is divided into several non-overlapping local small windows, and the size of each window is Each window is represented as The calculation formula of the mth attention head is as follows:
[0093] (3)
[0094] In formula (3), is a spatial query neural network layer, is a spatial key-value neural network layer, is a spatial value neural network layer, represents the feature dimension of is the output of the mth spatial self-attention; the attention operation in formula (3) is repeated M times, and all outputs are reorganized and combined together; in , the channel of is divided into M parts, and each part is represented as The calculation formula of the mth attention head is as follows:
[0095] (4)
[0096] In formula (4), is a channel query neural network layer, is a channel key-value neural network layer, is a channel value neural network layer, represents the feature dimension of is the output of the mth channel self-attention, and the attention operation in formula (4) is repeated M times, and all outputs are reorganized and combined together.
[0097] S13: An image reconstruction module is established which processes the output features of the module using a 3x3 convolution layer to obtain a residual image , and the reconstructed image is represented as:
[0098] (5)
[0099] In formula (5), represents the output restored image of the image restoration teacher network;
[0100] S14: A bottom-level visual feature extraction module , a deep-level visual feature extraction module and an image reconstruction module are stacked together to constitute an image restoration teacher network, that is:
[0101] (6)
[0102] In formula (6), denotes a stacking operation operator, and t denotes the image restoration teacher network.
[0103] Specifically, in step 2, the specific method is as follows:
[0104] S21: Randomly extract N images in the data set, and the nth degraded image is denoted as , which is input into the image restoration teacher network to obtain the output restored image , assuming the corresponding true value image is denoted as , and the reconstruction error loss function between and is established:
[0105] (7)
[0106] In formula (7), denotes the L1 norm, and the contrast loss function between and is established, that is:
[0107] (8)
[0108] In formula (8), denotes the L1 norm, and the robust loss function between and is established, that is:
[0109] (9)
[0110] In formula (9), denotes the L2 norm, denotes a constant;
[0111] S22: Weighted sum the reconstruction error loss function, the contrast loss function and the robust loss function to obtain a multiple loss function, and the mathematical expression is:
[0112] (10)
[0113] In formula (10), is the contrast loss function parameter, is the robust loss parameter, and based on formula (10), the image restoration teacher network is trained by using the gradient descent method.
[0114] Specifically, in step 3, the specific method is as follows:
[0115] S31: Establishing the bottom-level visual feature extraction module The module is composed of 3x3 convolutional layers, which perform the following mathematical operations:
[0116] (11)
[0117] In equation (11), represents any one degraded image, represents the bottom-level visual feature, represents the 3x3 convolutional layer;
[0118] S32: Establishing the deep-level visual feature extraction module , is composed of several mixed linear visual Transformer blocks, each of which is composed of multi-head spatial linear self-attention and multi-head channel linear self-attention. Assuming that the input feature of each mixed linear visual Transformer block is represented as , the signal processing process of the mixed linear visual Transformer block is represented as:
[0119] (12)
[0120] In equation (12), represents the output feature of the mixed linear visual Transformer block, , , represents the intermediate feature of the mixed linear visual Transformer block, represents the multi-head spatial self-attention, multi-head channel self-attention, represents the regularization operation, represents the multi-layer perceptron layer;
[0121] Assuming that the number of multi-head attention is M, in , is divided into several non-overlapping local small windows, each window has a size of , each window is represented as , the calculation formula of the mth attention head is as follows:
[0122] (13)
[0123] In equation (13), is a linear spatial attention query neural network layer, is a linear spatial attention key-value neural network layer, is a linear spatial attention value neural network layer, is a matrix reshaping operator, is the output of the m-th spatial self-attention, the attention operation in equation (13) is repeated M times, and all outputs are reorganized and combined together, the channels of are split into M parts, each part is denoted as The calculation formula of the m-th attention head is as follows:
[0124] (14)
[0125] in equation (14), is a linear channel attention query neural network layer, is a linear channel attention key-value neural network layer, is a linear channel attention value neural network layer, is a matrix reshaping operator, is the output of the m-th channel self-attention, the attention operation in equation (14) is repeated M times, and all outputs are reorganized and combined together,
[0126] S33: An image reconstruction module is established, which processes the output features of the module using a 3x3 convolutional layer to obtain a residual image , then the reconstructed image is denoted as:
[0127] (15)
[0128] in equation (15), denotes the output restored image of the image restoration teacher network;
[0129] S34: A bottom-level visual feature extraction module , a deep-level visual feature extraction module and an image reconstruction module are stacked together to form an image restoration teacher network, that is:
[0130] (16)
[0131] in equation (16), denotes a stacking operation operator, and s denotes a lightweight image restoration model.
[0132] Specifically, in step 4, the specific method is as follows:
[0133] S41: Randomly extract N images in the data set, and the n-th degraded image is denoted as The image is then input into the lightweight image restoration model network s constructed in step 3 to obtain the output restored image. , assuming The corresponding truth image is represented as ,Establish and Reconstruction error loss function between:
[0134] (17)
[0135] In equation (17), Represents the L1 norm;
[0136] S42: Degrade the image The image is input into the image restoration teacher network t constructed in step 1 to obtain the output restored image. Establish the output distillation loss function between the lightweight image restoration model and the image restoration teacher network, namely:
[0137] (18)
[0138] In equation (18), Represents the L1 norm;
[0139] S43: As can be seen from step S12, in the image restoration teacher network t... The module's feature output is As can be seen from step S32, in the lightweight student network s The module's feature output is Establish a multi-head spatial attention feature knowledge distillation loss function, namely:
[0140] (19)
[0141] In equation (19), Represents the L1 norm;
[0142] S44: As can be seen from step S12, in the image restoration teacher network t... The module's feature output is As can be seen from step S32, in the lightweight student network s The module's feature output is Establish a multi-head spatial attention feature knowledge distillation loss function, namely:
[0143] (20)
[0144] In equation (20), Represents the L1 norm;
[0145] S45: The reconstruction error loss function, the output distillation loss function, the multi-head spatial attention feature knowledge distillation loss function, and the multi-head spatial attention feature knowledge distillation loss function are weighted and summed to obtain a double feature knowledge distillation loss function, and the mathematical expression is:
[0146] (21)
[0147] In formula (21), is an output knowledge distillation loss adjustable parameter, is a multi-head spatial attention feature knowledge distillation loss adjustable parameter, is a multi-head channel attention feature knowledge distillation loss adjustable parameter, based on formula (21), the lightweight image restoration model is trained by using the gradient descent method, and the trained lightweight image restoration model is deployed online to perform an image restoration test task.
[0148] Embodiment:
[0149] As shown in Figure 1 , the specific steps are as follows:
[0150] Step 1: using a 3x3 convolutional layer to establish a bottom layer visual feature extraction module , then using multi-head spatial self-attention and multi-head channel self-attention to construct a mixed visual Transformer block, arranging the mixed visual Transformer module into a U-shaped structure to establish a deep layer visual feature extraction module . Next, using a 3x3 convolutional layer and residual learning to establish an image reconstruction module . Stack the bottom layer visual feature extraction module , the deep layer visual feature extraction module and the image reconstruction module together to form an image restoration teacher network t.
[0151] Step 2: randomly extract N images in the data set, the nth degraded image is represented as , input it into the image restoration teacher network to obtain the output restored image . Assuming , the corresponding true value image is represented as , the reconstruction error loss function between and is established , the contrast loss function between , and is established , and the robust loss function between and is established The reconstruction error loss function, the contrast loss function and the robust loss function are weighted and summed to obtain a multiple loss function . Based on the multiple loss function , the image restoration teacher network t is trained by using the gradient descent method.
[0152] Step 3: A bottom visual feature extraction module is established using a 3*3 convolutional layer Then, a hybrid linear visual Transformer block is constructed using multi-head spatial linear self-attention and multi-head channel linear self-attention, and the linear hybrid visual Transformer module is stacked together to establish a deep visual feature extraction module Next, an image reconstruction module is established using a 3*3 convolutional layer and residual learning The bottom visual feature extraction module , the deep visual feature extraction module and the image reconstruction module are stacked together to form the image restoration teacher network s.
[0153] Step 4: N images are randomly extracted from the data set, and the nth degraded image is represented as It is input into the lightweight image restoration model network s constructed in step 3 to obtain the output restored image . Assuming , the corresponding true value image is represented as , the reconstruction error loss function between and is established . The degraded image is input into the image restoration teacher network t constructed in step 1 to obtain the output restored image . The output distillation loss function between the lightweight image restoration model and the image restoration teacher network is established . For the feature output of the module in the image restoration teacher network t, it is , and for the feature output of the module in the lightweight student network s, it is , a multi-head spatial attention feature knowledge distillation loss function is established . For the feature output of the module in the image restoration teacher network t, it is , and for the feature output of the module in the lightweight student network s, it is , a multi-head channel attention feature knowledge distillation loss function is established The reconstruction error loss function, the output distillation loss function, the multi-head spatial attention feature knowledge distillation loss function and the multi-head channel attention feature knowledge distillation loss function are weighted and summed to obtain a double feature knowledge distillation loss function Based on The light-weight image restoration model is trained by using a gradient descent method, and the trained light-weight image restoration model is deployed online to perform an image restoration test task.
[0154] To sum up, the light-weight image restoration model can be constructed by using the multi-head spatial linear attention and the multi-head channel linear attention, so that the parameter capacity and the inference time of the image restoration model are reduced, and the efficiency of the image restoration model in a resource-limited scene is improved.
[0155] The description and practice disclosed in the present application are easy to think and understand for ordinary skilled in the art, and some improvements and refinements can be made without departing from the principles of the present application. Therefore, the modifications or improvements made without departing from the spirit of the present application should be considered as the protection scope of the present application.
Claims
1. A lightweight image restoration model based on mixed linear Transformer and double knowledge distillation, characterized in that, The method comprises the following steps: Step 1, establishing an image restoration teacher network: a mixed visual Transformer module is constructed by simultaneously using multi-head spatial self-attention and multi-head channel self-attention, and the image restoration teacher network is established by using the mixed visual Transformer module; Step 2, training the image restoration teacher network: a multiple loss function is established, and the image restoration teacher network is trained by using the gradient descent method; Step 3, establishing a lightweight image restoration model: a mixed linear visual Transformer module is constructed by simultaneously using multi-head spatial linear attention and multi-head channel linear attention, and the lightweight image restoration model is established by using the mixed linear visual Transformer module; Step 4, training the lightweight image restoration model by using a double-feature knowledge distillation algorithm: the spatial attention feature knowledge and the channel attention feature knowledge are respectively transferred from the image restoration teacher network to the lightweight image restoration model by using the double-feature knowledge distillation algorithm.
2. The lightweight image restoration model based on mixed linear Transformer and double knowledge distillation according to claim 1, characterized in that, In step 1, the specific method is as follows: S11: Establishing the bottom layer visual feature extraction module This module is composed of 3x3 convolutional layers, which perform the following mathematical operations: (1) In formula (1), denotes an arbitrary degraded image, denotes a bottom-level visual feature, denotes a 3x3 convolutional layer; S12: Establishing a deep visual feature extraction module , The U-shaped structure is arranged, and is composed of an encoder and a decoder; the encoder and the decoder are decomposed into four stages, and each stage is stacked by different numbers of mixed visual Transformer blocks; the encoder uses a down-sampling layer to gradually compress the input data to extract high-level features, and the decoder uses an up-sampling layer to gradually decompress the latent representation to reconstruct the original input size; each mixed visual Transformer block is composed of multi-head spatial self-attention and multi-head channel self-attention, assuming that the input features of each mixed visual Transformer block are represented as , the signal processing process of the mixed visual Transformer block is represented as: (2) in formula (2), denote the output features of the mixed vision Transformer block, , , denote the intermediate features of the mixed vision Transformer block, denote the multi-head spatial self-attention, multi-head channel self-attention, denote the regularization operation, denote the multi-layer perceptron layer; Assume the number of multi-head attentions is M, in which, is divided into several non-overlapping local small windows, the size of each window is , and each window is represented as The calculation formula of the mth attention head is as follows: (3) In formula (3), for the spatial query neural network layer, for the spatial key-value neural network layer, for the spatial value neural network layer, represents the feature dimension of is the output of the m-th spatial self-attention; the attention operation in formula (3) is repeated M times, and all outputs are reorganized and combined together; in , the channel of is divided into M parts, each part is represented as the calculation formula of the m-th attention head is as follows: (4) In formula (4), is a channel query neural network layer, is a channel key-value neural network layer, is a channel value neural network layer, represents is a feature dimension of is an output of the m-th channel self-attention, the attention operation in formula (4) is repeated M times, and all outputs are reorganized and combined together; S13: Establishing an image reconstruction module which processes the output features of module using a 3x3 convolutional layer to obtain a residual image The reconstructed image is then represented as: (5) In formula (5), represents the output restored image of the image restoration teacher network. S14: a bottom visual feature extraction module , a deep visual feature extraction module , and an image reconstruction module Stacked together to form an image restoration teacher network, namely: (6) In formula (6), denotes a stacking operation operator, and t denotes the image restoration teacher network.
3. The lightweight image restoration model based on mixed linear Transformer and double knowledge distillation according to claim 2, characterized in that, In step 2, the specific method is as follows: S21: Randomly extract N images in the dataset, the nth degraded image is represented , input into the image restoration teacher network, get the output restored image , assuming The corresponding true value image is represented as , the reconstruction error loss function between and is established: (7) In equation (7), Describe the L1 norm, and at the same time, establish , and The comparison loss function between them is: (8) In formula (8), represents the L1 norm, and a robust loss function between and is established, that is: (9) In formula (9), denotes the L2 norm, denotes a constant; S22: the reconstruction error loss function, the contrast loss function and the robust loss function are weighted and summed to obtain a multiple loss function, and the mathematical expression is: (10) In formula (10), is a contrast loss function parameter, is a robust loss parameter, based on formula (10), the image restoration teacher network is trained by using a gradient descent method.
4. The lightweight image restoration model based on mixed linear Transformer and double knowledge distillation according to claim 3, characterized in that, In step 3, the specific method is as follows: S31: Establishing a bottom layer visual feature extraction module This module is composed of 3x3 convolutional layers, which perform the following mathematical operations: (11) In formula (11), denotes an arbitrary degraded image, denotes a bottom-level visual feature, denotes a 3x3 convolutional layer; S32: Establishing a deep visual feature extraction module , is composed of several mixed linear visual Transformer blocks, each of which is composed of multi-head spatial linear self-attention and multi-head channel linear self-attention, assuming that the input feature of each mixed linear visual Transformer block is represented as The signal processing process of the mixed linear visual Transformer block is represented as: (12) in formula (12), denotes the output feature of the mixed linear vision Transformer block, , , denotes the intermediate feature of the mixed linear vision Transformer block, denotes a multi-headed spatial linear self-attention, a multi-headed channel linear self-attention, denotes a regularization operation, denotes a multi-layer perceptron layer; Assuming the number of multi-headed attention is M, in middle, It is divided into several non-overlapping local windows, each window being the size of... Each window is represented as The formula for calculating the m-th attention head is as follows: (13) In formula (13), is a linear spatial attention query neural network layer, is a linear spatial attention key-value neural network layer, is a linear spatial attention value neural network layer, denotes a matrix reshaping operator, is the output of the m-th spatial self-attention; the attention operation in formula (13) is repeated M times, all outputs are reorganized and combined together, in the channels of are split into M parts, each part is denoted as The calculation formula of the m-th attention head is as follows: (14) In formula (14), is a linear channel attention query neural network layer, is a linear channel attention key-value neural network layer, is a linear channel attention value neural network layer, denotes a matrix reshaping operator, is the output of the m-th channel self-attention, the attention operation in formula (14) is repeated M times, and all outputs are reorganized and combined together; S33: Establishing an image reconstruction module The module processes the output features of the module using a 3x3 convolutional layer to obtain a residual image The reconstructed image is then represented as: (15) In formula (15), denotes the output restored image of the image restoration teacher network; S34: a bottom visual feature extraction module , a deep visual feature extraction module , and an image reconstruction module are stacked together to form an image restoration teacher network, namely: (16) In formula (16), denotes a stacking operation operator, and s denotes a lightened image restoration model.
5. The lightweight image restoration model based on hybrid linear Transformer and double knowledge distillation according to claim 4, characterized in that, In step 4, the specific method is as follows: S41: Randomly extract N images in the dataset, the nth degraded image is represented , input into the lightweight image restoration model network s constructed in step 3, to obtain the output restored image , assuming The corresponding true value image is represented as , the reconstruction error loss function between and is established: (17) In formula (17), denotes the L1 norm; S42: input the degraded image to the image restoration teacher network t constructed in step 1 to obtain an output restored image , and an output distillation loss function between the light image restoration model and the image restoration teacher network is established, that is: (18) In formula (18), denotes the L1 norm; S43: It is known from step S12 that the image restoration teacher network t in The feature output of the module is It is known from step S32 that the lightweight student network s in The feature output of the module is A multi-head spatial attention feature knowledge distillation loss function is established, that is: (19) In formula (19), denotes the L1 norm; S44: It is known from step S12 that the feature output of the image restoration teacher network t is The feature output of the module is It is known from step S32 that the feature output of the lightweight student network s is The feature output of the module is A multi-head channel attention feature knowledge distillation loss function is established, that is: (20) In formula (20), denotes the L1 norm; S45: the reconstruction error loss function, the output distillation loss function, the multi-head spatial attention feature knowledge distillation loss function and the multi-head channel attention feature knowledge distillation loss function are weighted and summed to obtain a double-feature knowledge distillation loss function, and the mathematical expression is: (21) In formula (21), is an output knowledge distillation loss adjustable parameter, is a multi-head spatial attention feature knowledge distillation loss adjustable parameter, is a multi-head channel attention feature knowledge distillation loss adjustable parameter, based on formula (21), the lightweight image restoration model is trained by using the gradient descent method, and the trained lightweight image restoration model is deployed online to perform an image restoration test task.