A cross-dimension-based transformer image super-resolution method

Abstract

The application provides a cross-dimension-based Transformer image super-resolution method, which comprises the following steps: obtaining a low-resolution image to be processed; inputting the low-resolution image to be processed into a trained cross-dimension Transformer super-resolution network to obtain a high-resolution image after super-resolution; wherein the cross-dimension Transformer super-resolution network is connected in series with a spatial self-attention unit and a channel self-attention unit to realize the aggregation of spatial information and channel information. The method connects the spatial self-attention unit and the channel self-attention unit in series, models the correlation between the channel and the space, guides the self-attention calculation of the spatial unit and the channel unit, realizes the aggregation of the spatial information and the channel information, further improves the feature representation capability, and realizes excellent performance of the image in the super-resolution task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a cross-dimension-based Transformer image super-resolution method. BACKGROUND

[0002] In the era of big data, the wide application of high-quality images in various fields is increasingly significant. In the field of social media, high-quality images help improve the visual experience of users; in the medical field, high-quality images can help professionals make more accurate diagnoses and judgments; in the security field, high-quality images help maintain social order and monitoring. However, due to the limitations of device cost, natural environment and other factors, the collected images are usually of low resolution, poor quality and blurred. Therefore, image enhancement technology has important research value and practical application significance in various fields. Among them, super-resolution technology (SR) has important significance in image enhancement.

[0003] SR is a technology for generating high-resolution (HR) images from low-resolution (LR) images. In recent years, convolutional neural networks (CNN) have become a hot research topic due to their powerful ability to fit complex mappings. Super-resolution (SR) methods based on convolutional neural networks (CNN) are popular due to their powerful ability to extract high-frequency details from images, but it is difficult to establish global dependencies using CNN-based methods. As an alternative, Transformer-based methods use a powerful self-attention mechanism (SA) to model the global dependencies of input data and show impressive performance.

[0004] Recently, Transformers have gained considerable popularity in low-level vision tasks, including image super-resolution (SR). These networks utilize self-attention along either the spatial or channel dimensions and have made outstanding progress. However, the high correlation between space and channels limits the performance of single-dimension Transformers in super-resolution, resulting in poor super-resolution results. SUMMARY

[0005] In view of the defects of the above prior art, the present application provides a cross-dimension-based Transformer image super-resolution method to solve the technical problem of poor super-resolution effect of various image super-resolution methods in the prior art.

[0006] To achieve the above object and other related objects, the present application provides a cross-dimension-based Transformer image super-resolution method, which comprises the following steps: obtaining a low-resolution image to be processed; inputting the low-resolution image to be processed into a trained cross-dimension Transformer super-resolution network to obtain a high-resolution image after super-resolution; wherein the cross-dimension Transformer super-resolution network is realized by connecting a spatial self-attention unit and a channel self-attention unit in series to realize the aggregation of spatial information and channel information.

[0007] In an embodiment of the present application, the cross-dimension Transformer super-resolution network is expressed by the formula: X HR = H RC (H SF (X LR )+ H DF (H SF (X LR ))) ; wherein X LR is the low-resolution image, X HR is the high-resolution image, H SF (), H DF () and H RC () are respectively a shallow feature extraction unit, a deep feature extraction unit and a high-resolution reconstruction unit.

[0008] In an embodiment of the present application, the deep feature extraction unit is expressed by the formula: X d = H DF (X s )= M D (M D-1 (…M2(M1(X s )))) ; wherein X s and X d are respectively the input and output of the deep feature extraction unit, M i is the i-th cross-dimension Transformer module, and i={1, 2, …, D}.

[0009] In an embodiment of the present application, the cross-dimension Transformer module is expressed by the formula: X1=CDSSA(LN(X0))+X0; X2=MLP(LN(X1))+X1; X3=CDCSA(LN(X2))+X2; X4=MLP(LN(X3))+X3; wherein X0 and X4 are respectively the input and output of the cross-dimension Transformer module, X1~X3 are intermediate calculation features, LN() is layer normalization, CDSSA() is a spatial self-attention unit, MLP() is a multi-layer perception, and CDCSA() is a channel self-attention unit.

[0010] In one embodiment of the present invention, the spatial self-attention unit processes the first input features according to the following steps: based on the first input features, a matrix of query, keyword, and value is generated by linear projection, and the matrix Q is reconstructed from the matrix Q. S K S and V S According to the matrix Q S K S V S The intermediate feature Y1 is calculated using the following formula:

[0011] Where softmax() is for normalization, For matrix K S transpose, d k For matrix K S Feature dimensions;

[0012] The output feature Y of the spatial self-attention unit is calculated based on the intermediate feature Y1 and the following formula. CDSSA Y CDSSA =Y1·SEU(X in1 )+Y1, where X in1 The first input feature is SEU(), which is a spatial embedding unit.

[0013] In one embodiment of the present invention, the spatial embedding unit is expressed by the formula: SEU(X) in1 )=Sig(Conv(GELU(Conv(X in1 )))), where Conv() is the convolution operation, GELU() is the activation function, and Sig() is the Sigmoid() function.

[0014] In one embodiment of the present invention, the channel self-attention unit processes the second input feature according to the following steps: based on the second input feature, a matrix of query, keyword, and value is generated along the channel dimension through linear projection, and the matrix Q is reconstructed to obtain matrix Q. C K C and V C According to the matrix Q C K C V C And the intermediate feature Y2 is calculated using the following formula:

[0015] Where softmax() is for normalization, For matrix Q C The transpose of α, where α is a learnable parameter;

[0016] According to the intermediate feature Y2 and the following formula, the output feature Y of the channel self-attention unit is calculated CDCSA : Y CDCSA =Y2·CEU(X in2 )+Y2, wherein X in2 is the second input feature, and CEU() is a channel embedding unit.

[0017] In an embodiment of the present application, the channel embedding unit is expressed by the following formula: CEU(X in2 )=Sig(Conv(GELU(Conv(Gap(X in2 ))))) wherein Gap() is a global average pooling operation, Conv() is a convolution operation, GELU() is an activation function, and Sig() is a Sigmoid() function.

[0018] To achieve the above object and other related objects, the present application further provides an electronic device comprising a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to realize the method provided by any one of the above embodiments.

[0019] To achieve the above object and other related objects, the present application further provides a computer readable storage medium having a computer program stored thereon, the computer program being used to make a computer execute the method provided by any one of the above embodiments.

[0020] The present application has the following beneficial effects: the present application provides a cross-dimension-based Transformer image super-resolution method, which constructs a cross-dimension Transformer super-resolution network through a series of spatial self-attention units and channel self-attention units, and trains the super-resolution network, and then can process a low-resolution image to be processed and obtain a super-resolved high-resolution image; the spatial self-attention unit can capture the features of the spatial dimension and input valuable information of the channel dimension, and correspondingly, the channel self-attention unit captures the features of the channel dimension and inputs valuable information of the spatial dimension, through the series connection of the two units, the correlation between the channel and the space is modeled, the self-attention calculation of the spatial unit and the channel unit is guided, to realize the aggregation of the spatial information and the channel information to further improve the feature representation ability, and excellent performance of the image in the super-resolution task is realized. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0022] Figure 1 A flowchart of an image super-resolution method provided by an embodiment of the present application is shown in FIG. 1.

[0023] Figure 2 An architecture diagram of a cross-dimension Transformer module provided by an embodiment of the present application is shown in FIG. 2.

[0024] Figure 3 An architecture diagram of a spatial self-attention unit provided by an embodiment of the present application is shown in FIG. 3.

[0025] Figure 4 An architecture diagram of a spatial embedding unit provided by an embodiment of the present application is shown in FIG. 4.

[0026] Figure 5 An architecture diagram of a channel self-attention unit provided by an embodiment of the present application is shown in FIG. 5.

[0027] Figure 6 An architecture diagram of a channel embedding unit provided by an embodiment of the present application is shown in FIG. 6.

[0028] Figure 7 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 7.

[0029] Legend of reference signs: 101, processor; 102, memory. DETAILED DESCRIPTION

[0030] The advantages and effects of the present application can be easily understood by those skilled in the art from the content disclosed in the specification. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. In addition to the specific methods, devices and materials used in the embodiments, any method, device and material of the prior art similar or equivalent to the methods, devices and materials described in the embodiments of the present application can also be used to implement the present application according to the mastery of the prior art by those skilled in the art and the description of the present application.

[0031] It should be understood that the terms used in the embodiments of the present application are for describing specific specific embodiments, and are not intended to limit the protection scope of the present application. Unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as generally understood by those skilled in the art.

[0032] In the following description, numerous specific details are discussed in order to provide a thorough understanding of embodiments of the present application. However, it will be apparent to one skilled in the art that embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring embodiments of the present application.

[0033] The flow and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of various embodiments presented in this disclosure. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0034] See Figure 1 , Figure 1 A cross-dimension-based Transformer image super-resolution method provided by an embodiment of the present application includes: obtaining a low-resolution image to be processed; inputting the low-resolution image to be processed into a trained cross-dimension Transformer super-resolution network to obtain a super-resolved high-resolution image; and the cross-dimension Transformer super-resolution network is implemented by connecting a spatial self-attention unit and a channel self-attention unit in series to aggregate spatial information and channel information.

[0035] In this embodiment, the training process of the cross-dimension Transformer super-resolution network is the same as the common network model training process, specifically as follows: (1) first, an image pair is obtained as a training sample, the image pair is composed of a low-resolution image I LR and a high-resolution image I HR , wherein the low-resolution image I LR may be obtained by down-sampling an I HR image; (2) a cross-dimension Transformer super-resolution network is built; and (3) the cross-dimension Transformer super-resolution network is trained by using the training sample to obtain a trained cross-dimension Transformer super-resolution network.

[0036] In a specific embodiment of the present application, the cross-dimension Transformer super-resolution network is expressed by a formula as follows: X HR = H RC (H SF (X LR )+ H DF (H SF (X LR ))) ; wherein X LR is a low-resolution image, X HR is a high-resolution image, H SF (), H DF () and H RC () are respectively a shallow feature extraction unit, a deep feature extraction unit and a high-resolution reconstruction unit.

[0037] In this embodiment, the shallow feature extraction unit is used to extract low-level features of the input image, such as edge, texture and other basic information, to provide preliminary feature representation for subsequent deep feature extraction. The shallow feature extraction unit may, for example, be a convolutional layer.

[0038] The deep feature extraction unit is used to capture high-level features of the image, enhance the expression ability of the features by modeling the global dependence relationship of the spatial and channel dimensions, strengthen the acquisition of local information in the interaction of spatial and channel information, and generate feature pairs with short-range and long-range context.

[0039] The high-resolution reconstruction unit is used to up-sample and fuse the two levels of information to generate the final high-resolution image, and complete the super-resolution task from low resolution to high resolution.

[0040] In the above three units, the deep feature extraction unit is connected in series with the spatial self-attention unit and the channel self-attention unit to realize the aggregation of spatial information and channel information.

[0041] In a specific embodiment of the present application, the deep feature extraction unit is expressed by a formula as follows:

[0042] X d = H DF (X s )= M D (M D-1 (…M2(M1(X s )))) ;

[0043] wherein X s and X d are respectively the input and output of the deep feature extraction unit, and M iFor the i-th Cross-Dimension Transformer Block, i = {1, 2, …, D}. In this embodiment, the deep feature extraction unit is composed of D Cross-Dimension Transformer Blocks (CDTBs), and the output of the deep feature extraction unit is obtained after the D CDTBs.

[0044] See Figure 2 In a specific embodiment of the present application, the Cross-Dimension Transformer Block is expressed by the formula:

[0045] X1=CDSSA(LN(X0))+X0;

[0046] X2=MLP(LN(X1))+X1;

[0047] X3=CDCSA(LN(X2))+X2;

[0048] X4=MLP(LN(X3))+X3;

[0049] wherein X0 and X4 are the input and output of the Cross-Dimension Transformer Block respectively, X1-X3 are intermediate calculation features, LN() is layer normalization, CDSSA() is a spatial self-attention unit, MLP() is a multi-layer perceptron, and CDCSA() is a channel self-attention unit.

[0050] In this embodiment, the CDTB is composed of a Cross-Dimension Spatial Self-Attention (CDSSA) and a Cross-Dimension Channel Self-Attention (CDCSA) in series. When the CDSSA and the CDCSA generate outputs, a Spatial Embedding Unit (SEU) and a Channel Embedding Unit (CEU) will respectively input valuable information of the spatial dimension and the channel dimension. The CDSSA and the CDCSA are respectively based on spatial direction cross-dimension self-attention and channel direction cross-dimension self-attention, and by connecting the CDSSA and the CDCSA in series, the CDTB can realize element aggregation between the spatial dimension and the channel dimension.

[0051] In this embodiment, LN() is Layer Normalization, which is a normalization operation used to standardize the output of a certain layer in a neural network, making its mean 0 and variance 1. MLP() is Multi-Layer Perceptron, which is a fully connected neural network used for non-linear transformation and feature extraction of input features, usually composed of multiple fully connected layers (Linear Layer) and activation functions (such as ReLU).

[0052] See Figure 3 In a specific embodiment of the present application, the spatial self-attention unit processes the first input feature as follows.

[0053] (1) According to the first input feature, generate query, key and value matrices through linear projection, and reconstruct them to obtain matrix Q S , K S and V S . Let the first input feature X in1 ∈R C×H×W , the query, key and value matrices generated by linear projection can be represented as Q, K, V, where all matrices are in R C×H×W space, and the process is represented as:

[0054] Q=X in1 W Q , K=X in1 W K , V=X in1 W V ; in the formula, W Q , W K , W V ∈R C×C , is a linear projection with omitted bias. Reconstruct Q, K, V into two-dimensional matrix form (combine height and width dimensions) of R C×HW to obtain matrices Q S , K S and V S .

[0055] (2) Calculate the intermediate feature Y1 according to matrices Q S , K S , V S and the following formula:

[0056] where softmax() is a normalization, is the transpose of matrix K S , and d k is the feature dimension of matrix K S . In the self-attention mechanism, matrices Q S and KS The dot product is used to calculate the similarity between input features (i.e., attention weights), and the transpose operation is used to match the dimensions of the matrix multiplication. k This is used to scale the dot product result to prevent the dot product value from becoming too large, which could lead to unstable gradients.

[0057] (3) Calculate the output feature Y of the spatial self-attention unit based on the intermediate feature Y1 and the following formula. CDSSA Y CDSSA =Y1·SEU(X in1 )+Y1, where X in1 The first input feature is SEU(), which is the spatial embedding unit. The design of the spatial embedding unit utilizes valuable channel information, enabling channel-dimensional guidance in the self-attention computation of the spatial dimension.

[0058] Please see Figure 4 In a specific embodiment of the present invention, the spatial embedding unit is expressed by the formula: SEU(X) in1 )=Sig(Conv(GELU(Conv(X in1 In this context, Conv() is the convolution operation, GELU() is the activation function, and Sig() is the sigmoid function. SEU is used to extract valuable spatial information from the spatial dimension and embed it into a self-attention mechanism to enhance the expressive power of spatial features. Specifically, SEU generates spatial information weights through convolution and activation functions, and these weights are used to guide the computation of spatial self-attention (CDSSA).

[0059] Please see Figure 5 In a specific embodiment of the present invention, the channel self-attention unit processes the second input feature according to the following steps:

[0060] (1) Based on the second input features, generate a matrix of query, key and value along the channel dimension through linear projection, and reconstruct it to obtain matrix Q. C K C and V C The reconstructed matrix has a size of R. HW×C .

[0061] (2) Based on matrix Q C K C V C And the intermediate feature Y2 is calculated using the following formula:

[0062] Where softmax() is for normalization, For matrix Q C, where is the transpose of, and is a learnable parameter. The self-attention mechanism in the channel dimension is different from that in the spatial dimension. The channel dimension is usually much smaller than the spatial dimension. Using the learnable parameter can make the model more flexible to adapt to the feature distribution in the channel dimension, so that the model can dynamically adjust the scaling factor according to the data, thereby better capturing the inter-channel dependency.

[0063] (3) The output feature Y of the channel self-attention unit is calculated according to the intermediate feature Y2 and the following formula: CDCSA : Y CDCSA =Y2·CEU(X in2 )+Y2, where X in2 is the second input feature, and CEU() is the channel embedding unit. The design of the channel embedding unit utilizes valuable spatial information, so that the spatial dimension can guide the self-attention calculation in the channel dimension.

[0064] Please refer to Figure 6 , in an embodiment of the present application, the channel embedding unit is expressed by the formula: CEU(X in2 )=Sig(Conv(GELU(Conv(Gap(X in2 ))))), where Gap() is the global average pooling operation, Conv() is the convolution operation, GELU() is the activation function, and Sig() is the Sigmoid() function. The channel embedding unit is used to extract valuable channel information from the channel dimension and embed it into the self-attention mechanism to enhance the expression ability of the channel feature. Specifically, CEU generates channel information weights through global average pooling (GAP), convolution, and activation function, and these weights are used to guide the calculation of channel self-attention (CDCSA).

[0065] Please refer to Figure 7 , Figure 7 An electronic device provided by an embodiment of the present application includes a processor 101, a memory 102, and a communication bus. The communication bus is used to connect the processor 101 and the memory 102. The processor 101 is used to execute the computer program stored in the memory 102 to implement the above-mentioned cross-dimension-based Transformer image super-resolution method.

[0066] An embodiment of the present application further provides a computer readable storage medium having a computer program stored thereon, and the computer program is used to make a computer execute the above-mentioned cross-dimension-based Transformer image super-resolution method.

[0067] In an embodiment of the present application, the high-resolution reconstruction unit is expressed by the formula:

[0068] X h =H RC (Xs +X d )=Reshape(Conv(Upsampling(X s ,X d )));

[0069] where Upsampling() is an up-sampling operation, which is an operation to enlarge the spatial size (height and width) of the input feature map, usually implemented by interpolation or transposed convolution, in this embodiment, the transposed convolution layer is used to up-sample these feature maps to the desired scale by a scale factor s. Then after Conv() convolution operation and Reshape(), the output size becomes BxsHxsW, that is, a high-resolution image X HR .

[0070] In the image enhancement task, the L1 loss function is used to calculate the error between the super-resolution result I SR and the real data I HR , and the network parameters are optimized using the back propagation method, so as to complete the training of the layer-by-layer context information aggregation network. Specifically, the above task of training the network using the training samples includes the following steps: substituting the image I LR in the LR-HR image pair into the cross-dimension Transformer super-resolution network to obtain the image I SR ; according to the image I SR , the image I HR in the LR-HR image pair and the following loss function calculation formula to calculate the loss loss:

[0071] ;

[0072] The above steps can conveniently help us to realize the training of network parameters, and the trained cross-dimension Transformer super-resolution network can be directly used to perform super-resolution on the input X LR image to obtain its corresponding X HR image.

[0073] The effect of the present application is further illustrated below by comparison with other super-resolution algorithms. In the comparison, the training data set and the verification data set used are the training set and the verification set of the DF2K data set, and five test sets are used to comprehensively evaluate their performance, which are: Set5 test set, Set14 test set, BSD100 test set, Urban100 test set and Manga109 test set. In the evaluation, the Peak Signal to Noise Ratio (PSNR) and the Structural Similarity (SSIM) are used as the evaluation indexes to evaluate the super-resolution results. In order to verify the reliability of the results, the present application compares the performance of the present application and other super-resolution methods in the image enhancement task of ×2, ×3 and ×4 super-resolution multiples.

[0074] Table 1: Comparison of various super-resolution algorithms (the evaluation parameters corresponding to 2-4 after the first Scale are PSNR, and the evaluation parameters corresponding to 2-4 after the first Scale are SSIM)

[0075]

[0076] In the table, the first column Method is the abbreviation of each algorithm, the number of parameters to be learned in the model is indicated by Params, the number of floating point operations in the inference process of the model is indicated by FLOPs, and the super-resolution multiple is indicated by Scale. According to the comparison in Table 1, the present application is superior to other state-of-the-art methods while maintaining a lightweight parameter. The present application aggregates the spatial and channel two dimensions by using CDTB, which is superior to many CNN-based methods. The present application combines spatial and channel features to obtain strong representation ability and obtains excellent results on various test sets, which proves the potential of the present application in the image enhancement task. The image super-resolution method used in the present application uses the joint information of the spatial and channel of the image to construct the network, which can realize the image enhancement task with higher quality.

[0077] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical thought disclosed by the present application should be covered by the claims of the present application.

Claims

1. A cross-dimensional Transformer-based image super-resolution method, characterized in that, include: Acquire the low-resolution image to be processed; The low-resolution image to be processed is input into the trained cross-dimensional Transformer super-resolution network to obtain the super-resolution high-resolution image; The cross-dimensional Transformer super-resolution network uses a series of spatial self-attention units and channel self-attention units to aggregate spatial and channel information. The cross-dimensional Transformer super-resolution network is expressed by the following formula: X HR =H RC (H SF (X LR )+H DF (H SF (X LR ))); Among them, X LR For the low-resolution image, X HR For the high-resolution image, H SF (), H DF (), H RC () represent shallow feature extraction unit, deep feature extraction unit, and high-resolution reconstruction unit, respectively; The deep feature extraction unit is expressed by the following formula: X d =H DF (X s )=M D (M D-1 (…M2(M1(X s )))); Among them, X s and X d M represents the input and output of the deep feature extraction unit, respectively. i Let i be the i-th cross-dimensional Transformer module, i={1,2,…,D}0; The cross-dimensional Transformer module is represented by the following formula: X1 = CDSA(LN(X0)) + X0; X2 = MLP(LN(X1)) + X1; X3 = CDCSA(LN(X2)) + X2; X4 = MLP(LN(X3)) + X3; Wherein, X0 and X4 are the input and output of the cross-dimensional Transformer module, respectively, X1~X3 are intermediate computational features, LN() is layer normalization, CDSA() is spatial self-attention unit, MLP() is multilayer perceptron, and CDCSA() is channel self-attention unit.

2. The image super-resolution method based on cross-dimensional Transformer as described in claim 1, characterized in that, The spatial self-attention unit processes the first input feature according to the following steps: Based on the first input features, a matrix of query, keyword, and value is generated through linear projection, and then reconstructed to obtain matrix Q. S K S and V S ; According to the matrix Q S K S V S The intermediate feature Y1 is calculated using the following formula: Where softmax() is for normalization, For matrix K S transpose, d k For matrix K S Feature dimensions; The output feature Y of the spatial self-attention unit is calculated based on the intermediate feature Y1 and the following formula. CDSSA : , where X in1 The first input feature is SEU(), which is a spatial embedding unit.

3. The image super-resolution method based on cross-dimensional Transformer as described in claim 2, characterized in that, The spatial embedding unit is expressed by the formula: YOUR(X in1 )=Sig(Conv(GELU(Conv(X in1 )))), Here, Conv() is the convolution operation, GELU() is the activation function, and Sig() is the Sigmoid() function.

4. The image super-resolution method based on cross-dimensional Transformer as described in claim 1, characterized in that, The channel self-attention unit processes the second input feature according to the following steps: Based on the second input feature, a matrix of query, keyword, and value is generated along the channel dimension through linear projection, and then reconstructed to obtain matrix Q. C K C and V C ; According to the matrix Q C K C V C And the intermediate feature Y2 is calculated using the following formula: Where softmax() is for normalization, For matrix Q C The transpose of α, where α is a learnable parameter; The output feature Y of the channel self-attention unit is calculated based on the intermediate feature Y2 and the following formula. CDCSA : , where X in2 The second input feature is CEU(), which is a channel embedding unit.

5. The image super-resolution method based on cross-dimensional Transformer as described in claim 4, characterized in that, The channel embedding unit is expressed by the formula: CEU(X in2 )=Sig(Conv(GELU(Conv(Gap(X in2 ))))), Here, Gap() is the global average pooling operation, Conv() is the convolution operation, GELU() is the activation function, and Sig() is the Sigmoid() function.

6. An electronic device, characterized in that, It includes a processor, a memory, and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, It stores a computer program thereon, the computer program being used to cause the computer to perform the method as described in any one of claims 1 to 5.

Images