Lightweight image super-resolution method and device based on efficient frequency domain Transformer

Abstract

The present invention discloses a lightweight image super-resolution method and device based on an efficient frequency domain transformer, which relates to the field of image processing, and includes: constructing and training an image super-resolution model based on an efficient frequency domain transformer to obtain a trained image super-resolution model, wherein the image super-resolution model includes a first convolution layer, a sub-pixel convolution layer, a second convolution layer, and a plurality of efficient frequency domain transformer modules; obtaining a low allocation rate image to be reconstructed and an upsampling factor and inputting them into the trained image super-resolution model, first passing through the first convolution layer to obtain the output features of the first convolution layer, the output features of the first convolution layer sequentially passing through a plurality of efficient frequency domain transformer modules, adding the output features of the last efficient frequency domain transformer module to the output features of the first convolution layer to obtain a second addition result, and the second addition result sequentially passing through the sub-pixel convolution layer and the second convolution layer to obtain a high-resolution reconstructed image. The present invention overcomes the problem of high computational complexity of the existing transformer method.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a lightweight image super-resolution method and device based on an efficient frequency domain Transformer. Background Art

[0002] Single Image Super-Resolution (SISR) is a key task in the field of computer vision. It can restore a high-resolution image with high-fidelity details from a degraded low-resolution image. Currently, image super-resolution is widely used in various practical applications, including intelligent manufacturing, information security, medical diagnosis and other fields.

[0003] In recent years, Transformer has been widely used in the field of image super-resolution as a popular deep learning backbone network. Transformer models global feature interactions by calculating the similarity scores between all tags and shows good potential in image super-resolution. However, existing Transformer-based image super-resolution methods are difficult to use for high-resolution image restoration because its internal self-attention mechanism has a quadratic complexity related to the length of the token sequence. Therefore, further reducing the computational complexity of self-attention is an urgent need for current image super-resolution methods. Summary of the invention

[0004] The purpose of this application is to propose a lightweight image super-resolution method and device based on an efficient frequency domain Transformer to address the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a lightweight image super-resolution method based on an efficient frequency domain Transformer, comprising the following steps:

[0006] An image super-resolution model based on an efficient frequency domain Transformer is constructed and trained to obtain a trained image super-resolution model, wherein the image super-resolution model includes a first convolution layer, a sub-pixel convolution layer, a second convolution layer, and a number of sequentially connected efficient frequency domain Transformer modules, wherein the efficient frequency domain Transformer module includes a first normalization layer, a frequency domain mixed feature interaction module, a second normalization layer, and a multi-scale convolutional interactive feedforward network, wherein the input features of the efficient frequency domain Transformer module sequentially pass through the first normalization layer and the frequency domain mixed feature interaction module, wherein the input features of the frequency domain mixed feature interaction module are added to the input features of the efficient frequency domain Transformer module to obtain a first addition result, wherein the first addition result sequentially passes through a second normalization layer and a multi-scale convolutional interactive feedforward network, wherein the output features of the multi-scale convolutional interactive feedforward network are added to the first addition result again to obtain the output features of the efficient frequency domain Transformer module;

[0007] The low allocation rate image to be reconstructed and the upsampling factor are obtained and input into the trained image super-resolution model. The upsampling factor is the scale factor of the sub-pixel convolutional layer. The low-resolution image to be reconstructed is input into the first convolutional layer to obtain the output features of the first convolutional layer. The output features of the first convolutional layer are sequentially passed through several efficient frequency domain Transformer modules. The output features of the last efficient frequency domain Transformer module are added to the output features of the first convolutional layer to obtain a second addition result. The second addition result is sequentially passed through the sub-pixel convolutional layer and the second convolutional layer to obtain a high-resolution reconstructed image.

[0008] Preferably, the frequency domain mixed feature interaction module includes a first linear layer, a second linear layer, a third linear layer, a first Fourier transform layer, a second Fourier transform layer, a third Fourier transform layer, a global filter, a first inverse Fourier transform layer, a second inverse Fourier transform layer, a third normalization layer and a fourth linear layer. The input feature of the frequency domain mixed feature interaction module is , where C, H and W represent the number of channels, length and width of the input features of the frequency domain mixed feature interaction module, respectively. represents the set of real numbers;

[0009] The input feature X of the frequency domain mixed feature interaction module is divided into N non-overlapping windows of size S×S ,in , S is the length and width of the non-overlapping window, and then the first linear layer is used , the second linear layer and the third linear layer Non-overlapping windows Convert to query matrix , key matrix Sum Matrix , as shown below:

[0010] ;

[0011] ;

[0012] ;

[0013] The query matrix Q and the key matrix K are transformed from the spatial domain to the frequency domain using the first Fourier transform layer and the second Fourier transform layer respectively. The query matrix Q and the key matrix K are multiplied in the frequency domain to obtain the product result. The product result is transformed back to the spatial domain using the first inverse fast Fourier transform layer and the third normalization layer to generate the similarity matrix , as shown below:

[0014] ;

[0015] in, represents the fast Fourier transform, represents the inverse fast Fourier transform, represents the layer normalization operation, T represents the conjugate transpose operation, represents the temperature factor;

[0016] Perform matrix multiplication on the similarity matrix M and the value matrix and pass through the fourth linear layer , and obtain local enhanced features , as shown below:

[0017] ;

[0018] Next, the third Fourier transform layer is used to transform the input feature X of the frequency domain mixed feature interaction module from the spatial domain to the frequency domain, and the global frequency feature of the input feature of the frequency domain mixed feature interaction module in the frequency domain is obtained, and the global filter is used The global frequency features of the input features of the frequency domain hybrid feature interaction module are adaptively scaled in the frequency domain to obtain the scaled result, and then the second inverse fast Fourier transform layer is used to convert the scaled result from the frequency domain back to the spatial domain to obtain the global enhanced feature. , as shown below:

[0019] ;

[0020] Finally, using learnable parameters and Local enhancement features and global enhancement features Weighted to obtain enhanced hybrid enhancement features , as shown below:

[0021] .

[0022] Preferably, the multi-scale convolutional interactive feedforward network includes a third convolutional layer, a first depth-separable convolutional layer, a second depth-separable convolutional layer, a first GELU activation function layer, a second GELU activation function layer and a fourth convolutional layer;

[0023] The input features of the multi-scale convolutional interactive feedforward network are , where C, H, and W represent the number of channels, length, and width of the input features of the multi-scale convolutional interactive feedforward network, respectively;

[0024] The input features of the multi-scale convolutional interactive feedforward network are passed through a third convolutional layer with a convolution kernel size of 1×1 to remove redundant channel features and obtain the first intermediate feature , as shown below:

[0025] ;

[0026] in, Indicates a convolution operation with a convolution kernel size of 1×1;

[0027] The first intermediate feature is input into the first depth-separable convolutional layer and the second depth-separable convolutional layer respectively to extract the first scale feature and second scale features The first scale feature and the second scale feature are respectively passed through the first GELU activation function layer and the second GELU activation function layer, and then cross-multiplied and added to obtain the convolution interaction feature , as shown below:

[0028] ;

[0029] in, Represents the GELU activation function corresponding to the first GELU activation function layer or the second GELU activation function layer;

[0030] Convolutional Interaction Features After the fourth convolution layer with a convolution kernel size of 1×1, the output features of the multi-scale convolutional interactive feedforward network are obtained. , as shown below:

[0031] .

[0032] Preferably, the convolution kernel sizes of the first depthwise separable convolution layer and the second depthwise separable convolution layer are 3×3 and 5×5 respectively.

[0033] As a preference, the expression of the efficient frequency domain Transformer module is as follows:

[0034] ;

[0035] ;

[0036] in, represents the input features of the efficient frequency domain Transformer module, where C, H, and W represent the number of channels, length, and width of the input features of the efficient frequency domain Transformer module, respectively. Representation layer normalization operation, Represents the function corresponding to the frequency domain mixed feature interaction module, Represents the function corresponding to the multi-scale convolutional interactive feedforward network, represents the second intermediate feature, Represents the output features of the efficient frequency-domain Transformer module.

[0037] Preferably, the calculation process of the image super-resolution model is as follows:

[0038] The first convolution layer with a convolution kernel size of 1×1 is used to reconstruct the low-resolution image. Perform feature extraction, where H and W represent the length and width of the low-resolution image to be reconstructed, respectively, and extract shallow features , where C represents the number of channels of shallow features, as shown in the following formula:

[0039] ;

[0040] in, Indicates a convolution operation with a convolution kernel size of 1×1;

[0041] Deep features are extracted using A efficient frequency domain Transformer modules and a residual connection , as shown below:

[0042] ;

[0043] ;

[0044] in, represents the function corresponding to the i-th efficient frequency domain Transformer module, , represents the output features of the i-th efficient frequency domain Transformer module, represents the output features of the i-1th efficient frequency domain Transformer module;

[0045] The depth features are convolutionally transformed using a sub-pixel convolution layer with a scale factor of scale and a second convolution layer with a convolution kernel size of 1×1. Upsampling and reconstruction are performed to generate high-resolution reconstructed images , which is expressed as follows:

[0046] ;

[0047] in, Function representing a sub-pixel convolutional layer.

[0048] In a second aspect, the present invention provides a lightweight image super-resolution device based on an efficient frequency domain Transformer, comprising:

[0049] A model building module is configured to build and train an image super-resolution model based on an efficient frequency domain Transformer to obtain a trained image super-resolution model, wherein the image super-resolution model includes a first convolution layer, a sub-pixel convolution layer, a second convolution layer, and a plurality of sequentially connected efficient frequency domain Transformer modules, wherein the efficient frequency domain Transformer module includes a first normalization layer, a frequency domain mixed feature interaction module, a second normalization layer, and a multi-scale convolution interactive feedforward network, wherein the input features of the efficient frequency domain Transformer module sequentially pass through the first normalization layer and the frequency domain mixed feature interaction module, wherein the input features of the frequency domain mixed feature interaction module are added to the input features of the efficient frequency domain Transformer module to obtain a first addition result, wherein the first addition result sequentially passes through a second normalization layer and a multi-scale convolution interactive feedforward network, wherein the output features of the multi-scale convolution interactive feedforward network are added to the first addition result again to obtain the output features of the efficient frequency domain Transformer module;

[0050] The reconstruction module is configured to obtain a low-resolution image to be reconstructed and an upsampling factor and input them into a trained image super-resolution model, where the upsampling factor is a scale factor of a sub-pixel convolutional layer. The low-resolution image to be reconstructed is input into a first convolutional layer to obtain output features of the first convolutional layer. The output features of the first convolutional layer are sequentially passed through a number of efficient frequency domain Transformer modules, and the output features of the last efficient frequency domain Transformer module are added to the output features of the first convolutional layer to obtain a second addition result. The second addition result is sequentially passed through a sub-pixel convolutional layer and a second convolutional layer to obtain a high-resolution reconstructed image.

[0051] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0052] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0053] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] (1) The lightweight image super-resolution method based on efficient frequency domain Transformer proposed in this invention introduces a frequency domain mixed feature interaction module into the image super-resolution model, and uses Fourier transform to optimize the similarity score measurement process in the window self-attention calculation, thereby realizing efficient local feature modeling. At the same time, the frequency domain mixed feature interaction module uses a learnable global filter to adaptively adjust the global frequency features of the input features in the frequency domain to realize global feature modeling with linear complexity. Finally, learnable parameters are introduced to realize the adaptive mixing of local and global features, thereby improving the accuracy of Transformer reasoning.

[0056] (2) The lightweight image super-resolution method based on the efficient frequency domain Transformer proposed in this invention introduces a multi-scale convolutional interactive feedforward network into the image super-resolution model, eliminates spatial feature redundancy through multi-scale feature interaction, and avoids ignoring spatial modeling.

[0057] (3) The lightweight image super-resolution method based on the efficient frequency domain Transformer proposed in this paper can effectively avoid intensive self-attention calculations, so that the image super-resolution model can achieve better performance with less computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1A flowchart of a lightweight image super-resolution method based on an efficient frequency domain Transformer according to an embodiment of the present application;

[0060] Figure 2 A schematic diagram of a frequency domain mixed feature interaction module of a lightweight image super-resolution method based on an efficient frequency domain Transformer according to an embodiment of the present application;

[0061] Figure 3 A schematic diagram of a multi-scale convolutional interactive feedforward network of a lightweight image super-resolution method based on an efficient frequency domain Transformer in an embodiment of the present application;

[0062] Figure 4 A schematic diagram of an efficient frequency domain Transformer module of a lightweight image super-resolution method based on an efficient frequency domain Transformer according to an embodiment of the present application;

[0063] Figure 5 A schematic diagram of an image super-resolution model of a lightweight image super-resolution method based on an efficient frequency domain Transformer according to an embodiment of the present application;

[0064] Figure 6 A schematic diagram of a lightweight image super-resolution device based on an efficient frequency domain Transformer according to an embodiment of the present application;

[0065] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Figure 1 A lightweight image super-resolution method based on an efficient frequency domain Transformer provided in an embodiment of the present application is shown, comprising the following steps:

[0068] S1, construct an image super-resolution model based on an efficient frequency domain Transformer and train it to obtain a trained image super-resolution model, the image super-resolution model includes a first convolution layer, a sub-pixel convolution layer, a second convolution layer and a number of sequentially connected efficient frequency domain Transformer modules, the efficient frequency domain Transformer module includes a first normalization layer, a frequency domain mixed feature interaction module, a second normalization layer and a multi-scale convolution interactive feedforward network, the input features of the efficient frequency domain Transformer module pass through the first normalization layer and the frequency domain mixed feature interaction module in sequence, the input features of the frequency domain mixed feature interaction module are added to the input features of the efficient frequency domain Transformer module to obtain a first addition result, the first addition result passes through the second normalization layer and the multi-scale convolution interactive feedforward network in sequence, the output features of the multi-scale convolution interactive feedforward network are added to the first addition result again to obtain the output features of the efficient frequency domain Transformer module.

[0069] In a specific embodiment, the frequency domain mixed feature interaction module includes a first linear layer, a second linear layer, a third linear layer, a first Fourier transform layer, a second Fourier transform layer, a third Fourier transform layer, a global filter, a first inverse Fourier transform layer, a second inverse Fourier transform layer, a third normalization layer and a fourth linear layer. The input feature of the frequency domain mixed feature interaction module is , where C, H and W represent the number of channels, length and width of the input features of the frequency domain mixed feature interaction module, respectively. represents the set of real numbers;

[0070] The input feature X of the frequency domain mixed feature interaction module is divided into N non-overlapping windows of size S×S ,in , S is the length and width of the non-overlapping window, and then the first linear layer is used , the second linear layer and the third linear layer Non-overlapping windows Convert to query matrix , key matrix Sum Matrix , as shown below:

[0071] ;

[0072] ;

[0073] ;

[0074] The query matrix Q and the key matrix K are transformed from the spatial domain to the frequency domain using the first Fourier transform layer and the second Fourier transform layer respectively. The query matrix Q and the key matrix K are multiplied in the frequency domain to obtain the product result. The product result is transformed back to the spatial domain using the first inverse fast Fourier transform layer and the third normalization layer to generate the similarity matrix , as shown below:

[0075] ;

[0076] in, represents the fast Fourier transform, represents the inverse fast Fourier transform, represents the layer normalization operation, T represents the conjugate transpose operation, represents the temperature factor;

[0077] Perform matrix multiplication on the similarity matrix M and the value matrix and pass through the fourth linear layer , and obtain local enhanced features , as shown below:

[0078] ;

[0079] Next, the third Fourier transform layer is used to transform the input feature X of the frequency domain mixed feature interaction module from the spatial domain to the frequency domain, and the global frequency feature of the input feature of the frequency domain mixed feature interaction module in the frequency domain is obtained, and the global filter is used The global frequency features of the input features of the frequency domain hybrid feature interaction module are adaptively scaled in the frequency domain to obtain the scaled result, and then the second inverse fast Fourier transform layer is used to convert the scaled result from the frequency domain back to the spatial domain to obtain the global enhanced feature. , as shown below:

[0080] ;

[0081] Finally, using learnable parameters and Local enhancement features and global enhancement features Weighted to obtain enhanced hybrid enhancement features , as shown below:

[0082] .

[0083] In a specific embodiment, the multi-scale convolutional interactive feedforward network includes a third convolutional layer, a first depth-separable convolutional layer, a second depth-separable convolutional layer, a first GELU activation function layer, a second GELU activation function layer, and a fourth convolutional layer;

[0084] The input features of the multi-scale convolutional interactive feedforward network are , where C, H, and W represent the number of channels, length, and width of the input features of the multi-scale convolutional interactive feedforward network, respectively;

[0085] The input features of the multi-scale convolutional interactive feedforward network are passed through a third convolutional layer with a convolution kernel size of 1×1 to remove redundant channel features and obtain the first intermediate feature , as shown below:

[0086] ;

[0087] in, Indicates a convolution operation with a convolution kernel size of 1×1;

[0088] The first intermediate feature is input into the first depth-separable convolutional layer and the second depth-separable convolutional layer respectively to extract the first scale feature and second scale features The first scale feature and the second scale feature are respectively passed through the first GELU activation function layer and the second GELU activation function layer, and then cross-multiplied and added to obtain the convolution interaction feature , as shown below:

[0089] ;

[0090] in, Represents the GELU activation function corresponding to the first GELU activation function layer or the second GELU activation function layer;

[0091] Convolutional Interaction Features After the fourth convolution layer with a convolution kernel size of 1×1, the output features of the multi-scale convolutional interactive feedforward network are obtained. , as shown below:

[0092] .

[0093] In a specific embodiment, the convolution kernel sizes of the first depthwise separable convolution layer and the second depthwise separable convolution layer are 3×3 and 5×5 respectively.

[0094] In a specific embodiment, the expression of the efficient frequency domain Transformer module is as follows:

[0095] ;

[0096] ;

[0097] in, represents the input features of the efficient frequency domain Transformer module, where C, H, and W represent the number of channels, length, and width of the input features of the efficient frequency domain Transformer module, respectively. Representation layer normalization operation, Represents the function corresponding to the frequency domain mixed feature interaction module, Represents the function corresponding to the multi-scale convolutional interactive feedforward network, represents the second intermediate feature, Represents the output features of the efficient frequency-domain Transformer module.

[0098] Specifically, refer to Figure 2 and Figure 3 First, a frequency domain mixed feature interaction module and a multi-scale convolutional interactive feedforward network are constructed respectively. In the frequency domain mixed feature interaction module, the fast Fourier transform is used to optimize the self-attention calculation, and the global feature-local feature mixed interaction with linear complexity is realized, thereby effectively avoiding the dense self-attention calculation. Since the similarity scores of the query matrix and the key matrix in the spatial domain can be equivalently represented by the element-wise product of the two in the frequency domain, the fast Fourier transform is used to transform Q and K from the spatial domain to the frequency domain, and Q and K are multiplied in the frequency domain. Finally, the inverse fast Fourier transform is used to transform the product result back to the spatial domain. In the calculation formula of the similarity matrix, the temperature factor is used to scale the dot product result. In the multi-scale convolutional interactive feedforward network, the spatial feature redundancy is eliminated through multi-scale feature interaction. Reference Figure 4 , integrating the frequency domain mixed feature interaction module and the multi-scale convolutional interactive feedforward network to build an efficient frequency domain Transformer module. On the basis of the efficient frequency domain Transformer module, an image super-resolution model based on the efficient frequency domain Transformer is built, so that the image super-resolution model can achieve better performance with less computational overhead.

[0099] S2, obtain the low allocation rate image to be reconstructed and the upsampling factor and input them into the trained image super-resolution model, the upsampling factor is the scale factor of the sub-pixel convolution layer, the low-resolution image to be reconstructed is input into the first convolution layer, and the output features of the first convolution layer are obtained. The output features of the first convolution layer are sequentially passed through several efficient frequency domain Transformer modules, and the output features of the last efficient frequency domain Transformer module are added to the output features of the first convolution layer to obtain a second addition result, and the second addition result is sequentially passed through the sub-pixel convolution layer and the second convolution layer to obtain a high-resolution reconstructed image.

[0100] In a specific embodiment, the calculation process of the image super-resolution model is as follows:

[0101] The first convolution layer with a convolution kernel size of 1×1 is used to reconstruct the low-resolution image. Perform feature extraction, where H and W represent the length and width of the low-resolution image to be reconstructed, respectively, and extract shallow features , where C represents the number of channels of shallow features, as shown in the following formula:

[0102] ;

[0103] in, Indicates a convolution operation with a convolution kernel size of 1×1;

[0104] Deep features are extracted using A efficient frequency domain Transformer modules and a residual connection , as shown below:

[0105] ;

[0106] ;

[0107] in, represents the function corresponding to the i-th efficient frequency domain Transformer module, , represents the output features of the i-th efficient frequency domain Transformer module, represents the output features of the i-1th efficient frequency domain Transformer module;

[0108] The depth features are convolutionally transformed using a sub-pixel convolution layer with a scale factor of scale and a second convolution layer with a convolution kernel size of 1×1. Upsampling and reconstruction are performed to generate high-resolution reconstructed images , which is expressed as follows:

[0109] ;

[0110] in, Function representing a sub-pixel convolutional layer.

[0111] Specifically, refer to Figure 5 , the low-resolution image to be reconstructed and upsampling factor scale as input, where the value of scale is the required magnification, for example, scale is 2, 3, 4. After the low-resolution image to be reconstructed passes through the first convolution layer, shallow features are obtained. The shallow features are sequentially passed through A efficient frequency domain Transformer modules and then added to the shallow features to obtain deep features; Figure 6In the figure, the first efficient frequency domain Transformer module is represented as efficient frequency domain Transformer module-1, the second efficient frequency domain Transformer module is represented as efficient frequency domain Transformer module-2, and the Ath efficient frequency domain Transformer module is represented as efficient frequency domain Transformer module-A. The deep features are successively reconstructed through the sub-pixel convolution layer and the second convolution layer with a scale factor equal to the upsampling factor scale to obtain a high-resolution reconstructed image.

[0112] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a lightweight image super-resolution device based on an efficient frequency domain Transformer. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0113] The embodiment of the present application provides a lightweight image super-resolution device based on an efficient frequency domain Transformer, including:

[0114] The model construction module 1 is configured to construct and train an image super-resolution model based on an efficient frequency domain Transformer to obtain a trained image super-resolution model, wherein the image super-resolution model includes a first convolution layer, a sub-pixel convolution layer, a second convolution layer, and a plurality of sequentially connected efficient frequency domain Transformer modules, wherein the efficient frequency domain Transformer module includes a first normalization layer, a frequency domain mixed feature interaction module, a second normalization layer, and a multi-scale convolution interactive feedforward network, wherein the input features of the efficient frequency domain Transformer module sequentially pass through the first normalization layer and the frequency domain mixed feature interaction module, wherein the input features of the frequency domain mixed feature interaction module are added to the input features of the efficient frequency domain Transformer module to obtain a first addition result, wherein the first addition result sequentially passes through a second normalization layer and a multi-scale convolution interactive feedforward network, wherein the output features of the multi-scale convolution interactive feedforward network are added to the first addition result again to obtain the output features of the efficient frequency domain Transformer module;

[0115] Reconstruction module 2 is configured to obtain the low-resolution image to be reconstructed and the upsampling factor and input them into the trained image super-resolution model, the upsampling factor is the scale factor of the sub-pixel convolution layer, the low-resolution image to be reconstructed is input into the first convolution layer, and the output features of the first convolution layer are obtained. The output features of the first convolution layer are sequentially passed through several efficient frequency domain Transformer modules, and the output features of the last efficient frequency domain Transformer module are added to the output features of the first convolution layer to obtain a second addition result, and the second addition result is sequentially passed through the sub-pixel convolution layer and the second convolution layer to obtain a high-resolution reconstructed image.

[0116] Figure 7 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 7 As shown, the electronic device of this embodiment includes: a processor 701 and a memory 702; wherein the memory 702 is used to store computer-executable instructions; the processor 701 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0117] Optionally, the memory 702 may be independent or integrated with the processor 701 .

[0118] When the memory 702 is independently provided, the electronic device further includes a bus 703 for connecting the memory 702 and the processor 701 .

[0119] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 701 executes the computer execution instructions, the above method is implemented.

[0120] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 701, the above method is implemented.

[0121] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0122] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0123] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0124] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 701 to perform some steps of the methods of various embodiments of the present application.

[0125] It should be understood that the processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 701 may be any conventional processor 701, etc. The steps of the method disclosed in the invention may be directly embodied in the execution of the hardware processor 701, or may be executed by a combination of hardware and software modules in the processor 701.

[0126] The memory 702 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0127] The bus 703 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 703 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 703 in the drawings of the present application is not limited to only one bus 703 or one type of bus 703.

[0128] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0129] An exemplary storage medium is coupled to the processor 701, so that the processor 701 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 701. The processor 701 and the storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as ASIC). Of course, the processor 701 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0130] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight image super-resolution method based on an efficient frequency domain Transformer, characterized in that: The following steps are involved: An image super-resolution model based on an efficient frequency domain Transformer is constructed and trained to obtain a trained image super-resolution model, wherein the image super-resolution model comprises a first convolutional layer, a sub-pixel convolutional layer, a second convolutional layer, and a plurality of sequentially connected efficient frequency domain Transformer modules, wherein the efficient frequency domain Transformer module comprises a first normalization layer, a frequency domain mixed feature interaction module, a second normalization layer, and a multi-scale convolutional interactive feedforward network, wherein the input features of the efficient frequency domain Transformer module sequentially pass through the first normalization layer and the frequency domain mixed feature interaction module, and the output features of the frequency domain mixed feature interaction module are compared with the output features of the efficient frequency domain Transformer module. The input features of the random sampling module are added to obtain a first addition result, and the first addition result is sequentially passed through the second normalization layer and the multi-scale convolutional interactive feedforward network. The output features of the multi-scale convolutional interactive feedforward network are added to the first addition result again to obtain the output features of the efficient frequency domain Transformer module; the frequency domain mixed feature interaction module includes a first linear layer, a second linear layer, a third linear layer, a first Fourier transform layer, a second Fourier transform layer, a third Fourier transform layer, a global filter, a first inverse Fourier transform layer, a second inverse Fourier transform layer, a third normalization layer and a fourth linear layer. The input features of the frequency domain mixed feature interaction module are Where C, H and W represent the number of channels, length and width of the input features of the frequency domain hybrid feature interaction module, respectively. represents the set of real numbers; The input feature X of the frequency domain hybrid feature interaction module is divided into N non-overlapping windows with a window size of S×S Where N = H × W / S 2 , S is the length and width of the non-overlapping window, and then the first linear layer L Q , the second linear layer L K and the third linear layer L V The non-overlapping window X win Convert to query matrix Key Matrix Sum Matrix As shown below: Q=L Q (X win ); K=L K (X win ); V=L V (X win ); The query matrix Q and the key matrix K are respectively transformed from the spatial domain to the frequency domain using the first Fourier transform layer and the second Fourier transform layer, and the query matrix Q and the key matrix K are multiplied in the frequency domain to obtain a product result, and the product result is transformed back to the spatial domain using the first inverse Fourier transform layer and the third normalization layer to generate a similarity matrix As shown below: Where F(·) represents the fast Fourier transform, F -1 (·) represents the inverse fast Fourier transform, LN(·) represents the layer normalization operation, T represents the conjugate transpose operation, represents the dimension factor; The similarity matrix M and the value matrix perform matrix multiplication and pass through the fourth linear layer L out , and obtain local enhanced features As shown below: F local =L out (MV); Next, the third Fourier transform layer is used to transform the input feature X of the frequency domain mixed feature interaction module from the spatial domain to the frequency domain, and the global frequency feature of the input feature of the frequency domain mixed feature interaction module in the frequency domain is obtained, and the global filter is used The global frequency features of the input features of the frequency domain hybrid feature interaction module are adaptively scaled in the frequency domain to obtain a scaling result, and then the scaling result is converted from the frequency domain back to the spatial domain using the second inverse Fourier transform layer to obtain a global enhanced feature. As shown below: F global =F -1 (G·F(X)); Finally, the learnable parameters α1 and α2 are used to enhance the local feature F local and the global enhancement feature F global Weighted to obtain enhanced hybrid enhancement features As shown below: F mix =α1·F local +α2·F global ; The multi-scale convolutional interactive feedforward network includes a third convolutional layer, a first depth-separable convolutional layer, a second depth-separable convolutional layer, a first GELU activation function layer, a second GELU activation function layer and a fourth convolutional layer; The input features of the multi-scale convolutional interactive feedforward network are Wherein C, H and W represent the number of channels, length and width of the input features of the multi-scale convolutional interactive feedforward network respectively; The input features of the multi-scale convolutional interactive feedforward network are passed through a third convolutional layer with a convolution kernel size of 1×1 to remove redundant channel features, and the first intermediate feature As shown below: And′=Conv 1×1 (AND); Among them, Conv 1×1 (·) indicates a convolution operation with a kernel size of 1×1; The first intermediate feature is input into the first depth-separable convolutional layer and the second depth-separable convolutional layer respectively to extract the first scale feature and second scale features The first scale feature and the second scale feature are respectively passed through the first GELU activation function layer and the second GELU activation function layer, and then cross-multiplied and added to obtain a convolution interaction feature. As shown below: AND CI =Y′ 3×3 ·σ(Y′ 5×5 )+Y′ 5×5 ·σ(Y′ 3×3 ); Wherein, σ(·) represents the GELU activation function corresponding to the first GELU activation function layer or the second GELU activation function layer; The convolution interaction feature Y CI After the fourth convolution layer with a convolution kernel size of 1×1, the output feature Y of the multi-scale convolutional interactive feedforward network is obtained. out , as shown below: AND out =Conv 1×1 (AND CI ); A low allocation rate image to be reconstructed and an upsampling factor are obtained and input into the trained image super-resolution model, wherein the upsampling factor is the scale factor of the sub-pixel convolutional layer, the low-resolution image to be reconstructed is input into the first convolutional layer, and the output features of the first convolutional layer are obtained. The output features of the first convolutional layer are sequentially passed through a plurality of efficient frequency domain Transformer modules, and the output features of the last efficient frequency domain Transformer module are added to the output features of the first convolutional layer to obtain a second addition result. The second addition result is sequentially passed through the sub-pixel convolutional layer and the second convolutional layer to obtain a high-resolution reconstructed image.

2. The lightweight image super-resolution method based on efficient frequency domain Transformer according to claim 1, characterized in that: The convolution kernel sizes of the first depthwise separable convolution layer and the second depthwise separable convolution layer are 3×3 and 5×5 respectively.

3. The lightweight image super-resolution method based on efficient frequency domain Transformer according to claim 1, characterized in that: The expression of the efficient frequency domain Transformer module is as follows: Z′=f MF (LN(Z))+Z; Z out =f CI (LN(Z′))+Z′; in, represents the input features of the efficient frequency domain Transformer module, where C, H and W represent the number of channels, length and width of the input features of the efficient frequency domain Transformer module, respectively, LN(·) represents the layer normalization operation, and f MF (·) represents the function corresponding to the frequency domain mixed feature interaction module, f CI (·) represents the function corresponding to the multi-scale convolutional interactive feedforward network, Z′ represents the second intermediate feature, and Z out Represents the output features of the efficient frequency-domain Transformer module.

4. The lightweight image super-resolution method based on efficient frequency domain Transformer according to claim 1, characterized in that: The calculation process of the image super-resolution model is as follows: The low-resolution image to be reconstructed is reconstructed by using the first convolution layer with a convolution kernel size of 1×1. Perform feature extraction, where H and W represent the length and width of the low-resolution image to be reconstructed, respectively, and extract shallow features Where C represents the number of channels of shallow features, as shown in the following formula: F0=Conv 1×1 (I LR ); Among them, Conv 1×1 (·) indicates a convolution operation with a kernel size of 1×1; Deep features are extracted using A efficient frequency domain Transformer modules and a residual connection As shown below: in, represents the function corresponding to the i-th efficient frequency domain Transformer module, i∈[1,A], F i represents the output feature of the i-th efficient frequency domain Transformer module, F i-1 represents the output features of the i-1th efficient frequency domain Transformer module; The depth feature F is convolutionally transformed using a sub-pixel convolution layer with a scale factor of scale and a second convolution layer with a convolution kernel size of 1×1. D Upsampling and reconstruction are performed to generate high-resolution reconstructed images Its expression is as follows: I SR =Conv 1×1 (f up (F D )); Among them, f up (·) represents the function of the sub-pixel convolution layer.

5. A lightweight image super-resolution device based on an efficient frequency domain Transformer, characterized in that: include: A model building module is configured to build and train an image super-resolution model based on an efficient frequency domain Transformer to obtain a trained image super-resolution model, wherein the image super-resolution model includes a first convolution layer, a sub-pixel convolution layer, a second convolution layer, and a plurality of sequentially connected efficient frequency domain Transformer modules, wherein the efficient frequency domain Transformer module includes a first normalization layer, a frequency domain mixed feature interaction module, a second normalization layer, and a multi-scale convolution interaction feedforward network, wherein the input features of the efficient frequency domain Transformer module sequentially pass through the first normalization layer and the frequency domain mixed feature interaction module, and the output features of the frequency domain mixed feature interaction module are compared with the output features of the frequency domain mixed feature interaction module. The input features of the efficient frequency domain Transformer module are added to obtain a first addition result, the first addition result passes through the second normalization layer and the multi-scale convolutional interactive feedforward network in sequence, and the output features of the multi-scale convolutional interactive feedforward network are added to the first addition result again to obtain the output features of the efficient frequency domain Transformer module; the frequency domain mixed feature interaction module includes a first linear layer, a second linear layer, a third linear layer, a first Fourier transform layer, a second Fourier transform layer, a third Fourier transform layer, a global filter, a first Fourier inverse transform layer, a second Fourier inverse transform layer, a third normalization layer and a fourth linear layer, and the input features of the frequency domain mixed feature interaction module are Where C, H and W represent the number of channels, length and width of the input features of the frequency domain hybrid feature interaction module, respectively. represents the set of real numbers; The input feature X of the frequency domain hybrid feature interaction module is divided into N non-overlapping windows with a window size of S×S Where N = H × W / S 2 , S is the length and width of the non-overlapping window, and then the first linear layer L Q , the second linear layer L K and the third linear layer L V The non-overlapping window X win Convert to query matrix Key Matrix Sum Matrix As shown below: Q=L Q (X win ); K=L K (X win ); V=L V (X win ); The query matrix Q and the key matrix K are respectively transformed from the spatial domain to the frequency domain using the first Fourier transform layer and the second Fourier transform layer, and the query matrix Q and the key matrix K are multiplied in the frequency domain to obtain a product result, and the product result is transformed back to the spatial domain using the first inverse Fourier transform layer and the third normalization layer to generate a similarity matrix As shown below: Where F(·) represents the fast Fourier transform, F -1 (·) represents the inverse fast Fourier transform, LN(·) represents the layer normalization operation, T represents the conjugate transpose operation, represents the dimension factor; The similarity matrix M and the value matrix perform matrix multiplication and pass through the fourth linear layer L out , and obtain local enhanced features As shown below: F local =L out (MV); Next, the third Fourier transform layer is used to transform the input feature X of the frequency domain mixed feature interaction module from the spatial domain to the frequency domain, and the global frequency feature of the input feature of the frequency domain mixed feature interaction module in the frequency domain is obtained, and the global filter is used The global frequency features of the input features of the frequency domain hybrid feature interaction module are adaptively scaled in the frequency domain to obtain a scaling result, and then the scaling result is converted from the frequency domain back to the spatial domain using the second inverse Fourier transform layer to obtain a global enhanced feature. As shown below: F global =F -1 (G·F(X)); Finally, the learnable parameters α1 and α2 are used to enhance the local feature F local and the global enhancement feature F global Weighted to obtain enhanced hybrid enhancement features As shown below: F mix =α1·F local +α2·F global ; The multi-scale convolutional interactive feedforward network includes a third convolutional layer, a first depth-separable convolutional layer, a second depth-separable convolutional layer, a first GELU activation function layer, a second GELU activation function layer and a fourth convolutional layer; The input features of the multi-scale convolutional interactive feedforward network are Wherein C, H and W represent the number of channels, length and width of the input features of the multi-scale convolutional interactive feedforward network respectively; The input features of the multi-scale convolutional interactive feedforward network are passed through a third convolutional layer with a convolution kernel size of 1×1 to remove redundant channel features, and the first intermediate feature As shown below: And′=Conv 1×1 (AND); Among them, Conv 1×1 (·) indicates a convolution operation with a kernel size of 1×1; The first intermediate feature is input into the first depth-separable convolutional layer and the second depth-separable convolutional layer respectively to extract the first scale feature and second scale features The first scale feature and the second scale feature are respectively passed through the first GELU activation function layer and the second GELU activation function layer, and then cross-multiplied and added to obtain a convolution interaction feature. As shown below: AND CI =Y′ 3×3 ·σ(Y′ 5×5 )+Y′ 5×5 ·σ(Y′ 3×3 ); Wherein, σ(·) represents the GELU activation function corresponding to the first GELU activation function layer or the second GELU activation function layer; The convolution interaction feature Y CI After the fourth convolution layer with a convolution kernel size of 1×1, the output feature Y of the multi-scale convolutional interactive feedforward network is obtained. out , as shown below: AND out =Conv 1×1 (AND CI ); The reconstruction module is configured to obtain a low-resolution image to be reconstructed and an upsampling factor and input them into the trained image super-resolution model, wherein the upsampling factor is a scale factor of the sub-pixel convolutional layer, the low-resolution image to be reconstructed is input into the first convolutional layer, and the output features of the first convolutional layer are obtained. The output features of the first convolutional layer are sequentially passed through a plurality of efficient frequency domain Transformer modules, and the output features of the last efficient frequency domain Transformer module are added to the output features of the first convolutional layer to obtain a second addition result. The second addition result is sequentially passed through the sub-pixel convolutional layer and the second convolutional layer to obtain a high-resolution reconstructed image.

6. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images