Single-frame image super-resolution method and apparatus based on a mixed feature interaction Transformer

Abstract

The present invention discloses a single-frame image super-resolution method and apparatus based on a mixed feature interaction Transformer, which relates to the field of image processing. It includes the steps of obtaining a low-resolution image to be reconstructed; constructing and training a single-frame image super-resolution model based on a mixed feature interaction Transformer to obtain a trained single-frame image super-resolution model, where the single-frame image super-resolution model includes a shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit connected in sequence, and the deep feature extraction unit includes P mixed feature interaction Transformer modules connected in sequence; and inputting the low-resolution image into the trained single-frame image super-resolution model, extracting shallow features using the shallow feature extraction unit, inputting the shallow features into the deep feature extraction unit to extract deep features, inputting the deep features into the upsampling reconstruction unit, and reconstructing to obtain a high-resolution reconstructed image, which solves the problem that the Transformer's SR method ignores the potential correlation between features of different dimensions, which affects the reconstruction performance.

Description

[Technical Field]

[0001] The present invention relates to the field of image processing, and more specifically to a single-frame image super-resolution method and apparatus based on a mixed feature interaction Transformer. [Background technology]

[0002] Image super-resolution (SR) is a crucial task in computer vision and image processing, aiming to reconstruct high-quality high-resolution (HR) images from existing low-resolution (LR) images. In recent years, SR methods based on convolutional neural networks (CNNs) have dominated the field of image SR due to their powerful feature representation, end-to-end learnable paradigm, and superior performance. However, because convolution operations extract local features within a narrow neighborhood using a fixed sliding window, CNN-based SR methods have limited information pixels. Currently, Transformers offer a novel alternative to CNNs, capable of achieving superior performance in a variety of low-level vision tasks.

[0003] Regarding image SR, Liang et al. proposed SwinIR, an SR model based on the Swin Transformer. SwinIR employs a hierarchical design, restricting similarity calculations to local windows and enhancing information interaction between windows using a shift window mechanism. However, SwinIR uses window-based self-attention and abandons global information inference, thus limiting the performance of the Transformer. To activate more information pixels that contribute to image SR, Chen et al. proposed HAT, which better aggregates information between windows by introducing channel attention. Wang et al. proposed Omni-SR, which can simultaneously model pixel-level information interaction between spatial and window dimensions. However, conventional Transformer-based SR methods generally capture relationships between space and channels through serial or parallel operations, but ignore potential relationships between features of different dimensions, thus limiting the performance of Transformer-based SR methods. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] To address the technical challenges described above, the objective of the embodiments of this application is to propose a single-frame image super-resolution method and apparatus based on a mixed-feature interaction Transformer, thereby overcoming the problem that conventional Transformer methods ignore potential relationships between features of different dimensions and promoting interdimensional feature interactions, thereby significantly improving the global feature representation capability and detailed reconstruction capability of the image super-resolution method. [Means for solving the problem]

[0005] In a first aspect, the present invention relates to a single-frame image super-resolution method based on a mixed feature interaction Transformer, The steps include: acquiring a low-resolution image to be reconstructed, A step of constructing and training a single-frame image super-resolution model based on a mixed feature interaction Transformer to obtain a trained single-frame image super-resolution model, wherein the single-frame image super-resolution model includes sequentially connected shallow feature extraction units, deep feature extraction units, and upsampling reconstruction units, and the deep feature extraction unit includes sequentially connected P mixed feature interaction Transformer modules. The present invention provides a single-frame image super-resolution method based on a mixed feature interaction Transformer, which includes the steps of: inputting a low-resolution image to be reconstructed into a pre-trained single-frame image super-resolution model; extracting shallow features using a shallow feature extraction unit; inputting the shallow features into a deep feature extraction unit to extract deep features; inputting the deep features into an upsampling reconstruction unit for reconstruction to obtain a high-resolution reconstructed image.

[0006] Preferably, the mixed feature interaction Transformer module includes an efficient local feature extraction unit, a first normalization layer, a mixed feature interaction self-attention unit, a second normalization layer, and a mixed scale feedforward neural network. The input to the mixed feature interaction Transformer module passes sequentially through the efficient local feature extraction unit and the first normalization layer. The output of the first normalization layer is added to the input to the mixed feature interaction Transformer module to obtain a first mixed feature. The first mixed feature then passes sequentially through the mixed feature interaction self-attention unit and the second normalization layer. The output of the second normalization layer is added to the first mixed feature to obtain a second mixed feature. The second mixed feature is input to the mixed scale feedforward neural network to obtain the output of the mixed feature interaction Transformer module.

[0007] Preferably, the efficient local feature extraction unit includes a sequentially connected first shift convolutional layer, a first GeLU activation function layer, a second shift convolutional layer, an SE module, and a third shift convolutional layer, and the computation process for the efficient local feature extraction unit is as follows: JPEG0007863804000001.jpg13170 Here, H ELF (·) represents the function of an efficient local feature extraction unit, F shift-conv (·) represents the shift convolution operation of the first shift convolution layer, the second shift convolution layer, or the third shift convolution layer, F SE (·) represents the function of the SE module, and GeLU(·) represents the GeLU activation function.

[0008] Preferably, the hybrid feature interaction self-attention unit includes a local window self-attention branch, a deep convolution branch, and a bidirectional feature interaction unit. The bidirectional feature interaction unit includes a spatial interaction unit and a channel interaction unit. The channel interaction unit includes a global average pooling layer, a first convolution layer, a first batch normalization layer, a second GeLU activation function layer, a second convolution layer, and a first Sigmoid activation function layer connected in sequence. The spatial interaction unit includes a third convolution layer, a second batch normalization layer, a third GeLU activation function layer, a fourth convolution layer, and a second Sigmoid activation function connected in sequence. The local window self-attention branch includes a query linear layer, a key linear layer, a value linear layer, and a local window self-attention module. The deep convolution branch includes a first deep convolution layer with a convolution kernel size of 3×3. The local features output by the first deep convolution layer are input into the channel interaction unit to obtain dynamic weights at the channel level. The dynamic weights at the channel level are input into the local window self-attention branch to adaptively correct the value feature map output by the value linear layer. The global features output by the local window self-attention module are input into the spatial interaction unit to obtain dynamic weights at the spatial level. The dynamic weights at the spatial level are input into the deep convolution branch to adaptively correct the local features.

[0009] Preferably, the calculation process of the hybrid feature interaction self-attention unit is as follows: JPEG0007863804000002.jpg 249170 Here, SI(·) represents the function of the spatial interaction unit, the dynamic weight W at the spatial level sa is multiplied by the global feature F local to adaptively correct the global feature F local and record the correction result as F′ local JPEG0007863804000003.jpg 23170

[0010] Preferably, the hybrid scale feed-forward neural network includes a first branch, a second branch, and a fifth convolutional layer. The first branch includes a second deep convolutional layer, a first ReLU activation function layer, a third deep convolutional layer, and a second ReLU activation function layer connected in sequence. The second branch includes a fourth deep convolutional layer, a third ReLU activation function layer, a fifth deep convolutional layer, and a fourth ReLU activation function layer connected in sequence. The convolutional kernel sizes of the second deep convolutional layer and the fifth deep convolutional layer are 7×7, and the convolutional kernel sizes of the third deep convolutional layer and the fourth deep convolutional layer are 5×5. The specific calculation process is as follows. JPEG0007863804000004.jpg 57170 Output the first cross feature and the second cross feature respectively. After stitching the first cross feature and the second cross feature, input them into the fifth convolutional layer, and add the output of the fifth convolutional layer to the second feature map. JPEG0007863804000005.jpg 70170 Here, ReLU(·) represents the ReLU activation function, DwConv 5×5 (·) and DwConv 7×7 (·) respectively represent the functions of deep convolutional layers with convolutional kernels of 5×5 and 7×7. Conv 1×1 (·) represents the function of the fifth convolutional layer with a convolutional kernel size of 1×1. [·] represents the stitching operation. JPEG0007863804000006.jpg 15170

[0011] ​Preferably, the specific structure and computation process of the single-frame image super-resolution model are as follows: The shallow feature extraction unit uses the sixth convolutional layer, and the calculation process of the shallow feature extraction unit is as follows: JPEG0007863804000007.jpg23170Conv 3×3 (·) represents the function of the 6th convolutional layer, which is a 3x3 convolutional kernel, I LR This represents a low-resolution image. Features are extracted using P mixed-feature interaction Transformer modules, F0 is transmitted to the network terminals using long-skip connections, and residual learning is performed by adding it to the output of the Pth mixed-feature interaction Transformer module. The formula is as follows: JPEG0007863804000008.jpg20170 Here, F i-1 This represents the output of the i-1th mixed feature interaction Transformer module, MF P This represents the function of the P-th mixed feature interaction Transformer module, MF 1 This represents the function of the first mixed feature interaction Transformer module, MF 2 This represents the function of the second mixed feature interaction Transformer module, MF i represents the function of the i-th mixed feature interaction Transformer module, and F i This represents the output of the i-th mixed feature interaction Transformer module, JPEG0007863804000009.jpg32170 Here, f up (·) represents the function of the subpixel convolutional layer, Conv 3×3 (·) represents the function of the 7th convolutional layer, I SR This represents a high-resolution reconstructed image. JPEG0007863804000010.jpg represents three-dimensional data with dimensions of 111703 × (H × scale) × (W × scale).

[0012] In a second aspect, the present invention relates to a single-frame image super-resolution device based on a mixed feature interaction Transformer, An image acquisition module configured to acquire low-resolution images to be reconstructed, A model building module configured to construct and train a single-frame image super-resolution model based on a mixed feature interaction Transformer, and to obtain a trained single-frame image super-resolution model, wherein the single-frame image super-resolution model includes a sequentially connected shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit, and the deep feature extraction unit includes a model building module that includes P sequentially connected mixed feature interaction Transformer modules, The present invention provides a single-frame image super-resolution device based on a mixed-feature interaction Transformer, which includes a reconstruction module configured to input a low-resolution image to be reconstructed into a trained single-frame image super-resolution model, extract shallow features using a shallow feature extraction unit, input the shallow features into a deep feature extraction unit to extract deep features, input the deep features into an upsampling reconstruction unit for reconstruction, and obtain a high-resolution reconstructed image.

[0013] In a third aspect, the present invention provides an electronic device comprising one or more processors and a storage device for storing one or more programs, wherein when one or more programs are executed by one or more processors, one or more processors implement the method according to any of the implementations of the first aspect. [Effects of the Invention]

[0014] Compared to the prior art, the present invention has the following beneficial effects.

[0015] (1) In the single-frame image super-resolution method based on the mixed-feature interaction Transformer according to the present invention, the mixed-feature interaction self-attention unit uses a dual-branch structure combined with a bidirectional feature interaction unit. The dual-branch structure is based on a standard local-window self-attention unit, and by adding a deep convolutional branch in parallel with the local-window self-attention unit, the feature interaction capability between windows of the Transformer can be enhanced. The bidirectional feature interaction unit can provide complementary cues to the dual-branch structure, and by fully considering the complementarity between different types of features, information utilization and image super-resolution performance can be greatly improved.

[0016] (2) The single-frame image super-resolution method based on the mixed feature interaction Transformer according to the present invention can overcome the problem that conventional Transformer methods ignore the potential relationships between features of different dimensions, and by promoting interdimensional feature interaction, it significantly improves the global feature representation capability and detailed reconstruction capability of the image super-resolution method.

[0017] (3) The single-frame image super-resolution method based on the mixed feature interaction Transformer according to the present invention has a lower number of parameters and Flops value compared to conventional single-frame image super-resolution methods, has the best integrated performance, and can achieve high-performance image super-resolution reconstruction with fewer computational resources. [Brief explanation of the drawing]

[0018] To more clearly explain the technical solutions in the embodiments of the present invention, the drawings necessary for describing the embodiments will be briefly described below. However, the drawings in the following description represent only a few embodiments of the present invention, and it will be obvious to those skilled in the art that other drawings can be obtained based on these drawings without any creative effort. [Figure 1] This is a diagram illustrating an exemplary apparatus architecture to which one embodiment of the present invention can be applied. [Figure 2]This is a schematic flowchart of a single-frame image super-resolution method based on a mixed feature interaction Transformer in an embodiment of the present invention. [Figure 3] This is a schematic diagram of the structure of an efficient local feature extraction unit for a single-frame image super-resolution method based on a mixed feature interaction Transformer in an embodiment of the present invention. [Figure 4] This is a schematic diagram of the structure of the mixed feature interaction self-attention unit of the single-frame image super-resolution method based on a mixed feature interaction Transformer in an embodiment of the present invention. [Figure 5] This is a schematic diagram of the structure of the mixed-scale feedforward neural network for a single-frame image super-resolution method based on a mixed-feature interaction Transformer in an embodiment of the present invention. [Figure 6] This is a schematic diagram of the structure of the mixed feature interaction Transformer module of the single-frame image super-resolution method based on the mixed feature interaction Transformer in an embodiment of the present invention. [Figure 7] This is a schematic diagram of the structure of the single-frame image super-resolution model based on a mixed-feature interaction Transformer in an embodiment of the present invention. [Figure 8] This is a schematic diagram of a single-frame image super-resolution device based on a mixed feature interaction Transformer in an embodiment of the present invention. [Figure 9] This is a schematic diagram of the structure of a computer device suitable for realizing the electronic device in the embodiment of the present invention. [Modes for carrying out the invention]

[0019] To further clarify the object, technical solutions, and advantages of the present invention, the present invention will be described in more detail below with reference to the drawings, although it is clear that the described embodiments represent only some, and not all, embodiments of the present invention. All other embodiments that a person skilled in the art could obtain without creative effort based on the embodiments of the present invention are within the scope of the protection of the present invention.

[0020] Figure 1 shows an exemplary apparatus architecture 100 to which the single-frame image super-resolution method or single-frame image super-resolution apparatus based on a mixed-feature interaction Transformer, as described in the embodiment of the present invention, can be applied.

[0021] As shown in Figure 1, the device architecture 100 may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 provides a medium for communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types such as wired, wireless, or fiber optic cables.

[0022] The single-frame image super-resolution method based on a mixed-feature interaction Transformer provided in the embodiment of this application can be executed by the server 105, or by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Accordingly, the single-frame image super-resolution device based on a mixed-feature interaction Transformer may be provided in the server 105, or in the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0023] The number of terminal devices, networks, and servers in Figure 1 is illustrative only. Any number of terminal devices, networks, and servers may be used as needed. If it is not necessary to retrieve processing data remotely, the above device architecture may include only servers or terminal devices, without including a network.

[0024] Figure 2 shows a single-frame image super-resolution method based on a mixed feature interaction Transformer provided by an embodiment of the present invention, which comprises steps S1 to S3.

[0025] Step S1: Obtain the low-resolution image to be reconstructed.

[0026] Specifically, low-resolution images to be reconstructed are collected, and these low-resolution images are single-frame images.

[0027] Step S2: Construct and train a single-frame image super-resolution model based on mixed feature interaction Transformers to obtain a trained single-frame image super-resolution model which includes, in order, a shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit, and the deep feature extraction unit includes, in order, P mixed feature interaction Transformer modules.

[0028] In a specific embodiment, the mixed feature interaction Transformer module includes an efficient local feature extraction unit, a first-layer normalization layer, a mixed feature interaction self-attention unit, a second-layer normalization layer, and a mixed-scale feedforward neural network. The input to the mixed feature interaction Transformer module passes sequentially through the efficient local feature extraction unit and the first-layer normalization layer. The output of the first-layer normalization layer is added to the input to the mixed feature interaction Transformer module to obtain a first mixed feature. The first mixed feature then passes sequentially through the mixed feature interaction self-attention unit and the second-layer normalization layer. The output of the second-layer normalization layer is added to the first mixed feature to obtain a second mixed feature. The second mixed feature is input to the mixed-scale feedforward neural network, and the output of the mixed feature interaction Transformer module is obtained.

[0029] In a specific example, an efficient local feature extraction unit includes a sequentially connected first shift convolutional layer, a first GeLU activation function layer, a second shift convolutional layer, an SE module, and a third shift convolutional layer, and the computation process for the efficient local feature extraction unit is as follows: JPEG0007863804000011.jpg12170 Here, H ELF(·) represents a function of an efficient local feature extraction unit, F shift-conv (·) represents a shift convolution operation in the first shift convolution layer, the second shift convolution layer, or the third shift convolution layer, F SE (·) represents a function of the SE module, and GeLU(·) represents the GeLU activation function.

[0030] In a specific example, the mixed feature interaction self-attention unit includes a local window self-attention branch, a deep convolution branch, and a bidirectional feature interaction unit, the bidirectional feature interaction unit includes a spatial interaction unit and a channel interaction unit, the channel interaction unit includes a sequentially connected global mean pooling layer, a first convolutional layer, a first batch normalization layer, a second GeLU activation function layer, a second convolutional layer, and a first sigmoid activation function layer, the spatial interaction unit includes a sequentially connected third convolutional layer, a second batch normalization layer, a third GeLU activation function layer, a fourth convolutional layer, and a second sigmoid activation function, the local window self-attention branch includes a query linear layer, The deep convolution branch includes a key linear layer, a value linear layer, and a local window self-attention module. The deep convolution branch includes a first deep convolution layer with a convolution kernel size of 3x3. Local features output by the first deep convolution layer are input to a channel interaction unit to obtain channel-level dynamic weights. These channel-level dynamic weights are input to the local window self-attention branch to adaptively correct the value feature map output by the value linear layer. Global features output by the local window self-attention module are input to a spatial interaction unit to obtain spatial-level dynamic weights. These spatial-level dynamic weights are input to the deep convolution branch to adaptively correct the local features.

[0031] In a specific example, the computation process for the mixed-feature interaction self-attention unit is as follows: JPEG0007863804000012.jpg52170 Here, DwConv 3×3(·) represents the function of the first deep convolutional layer, JPEG0007863804000013.jpg92170 Here, L Q , L K , L V These represent the functions of the query-linear layer, key-linear layer, and value-linear layer, respectively. The data format of the Value Feature Map V is NS 2 Convert from ×C to C×H×W, and add channel-level dynamic weights W. ca Multiply by this to adaptively correct the value feature map V, and then convert the data format to NS. 2 Restore to ×C and record the correction result as V′. JPEG0007863804000014.jpg60170 Here, SI(·) represents a function of the spatial interaction unit, Spatial-level dynamic weights W sa Global Features F local By multiplying by the global feature F, local Adaptively correct this, and the corrected result is F′ local Record it as, Ultimately, global feature F local and F' local Adding that, JPEG0007863804000015.jpg12170

[0032] In a specific example, the mixed-scale feedforward neural network includes a first branch, a second branch, and a fifth convolutional layer. The first branch includes a sequentially connected second deep convolutional layer, a first ReLU activation function layer, a third deep convolutional layer, and a second ReLU activation function layer. The second branch includes a sequentially connected fourth deep convolutional layer, a third ReLU activation function layer, a fifth deep convolutional layer, and a fourth ReLU activation function layer. The convolutional kernel size of the second and fifth deep convolutional layers is 7x7, and the convolutional kernel size of the third and fourth deep convolutional layers is 5x5. The specific computation process is as follows: JPEG0007863804000016.jpg56170 The first and second cross-sectional features are output separately, the first and second cross-sectional features are stitched together and then input into the fifth convolutional layer, and the output of the fifth convolutional layer is added to the second feature map. JPEG0007863804000017.jpg69170 Here, ReLU(·) represents the ReLU activation function, and DwConv 5×5 (·) and DwConv 7×7 (·) represents the functions of deep convolutional layers with convolution kernels of 5×5 and 7×7, respectively. 1×1 (·) represents the function of the 5th convolutional layer with a convolution kernel size of 1×1, and [·] represents the stitching operation. JPEG0007863804000018.jpg20170

[0033] Specifically, referring to Figure 3, first, an efficient local feature extraction unit can be constructed. This efficient local feature extraction unit is composed of a first shift convolutional layer, a first GeLU activation function layer, a second shift convolutional layer, an SE module, and a third shift convolutional layer connected in cascade order, with the SE module being a Squeeze-Excitation Module. Referring to Figure 4, next, a mixed feature interaction self-attention unit is constructed. The mixed feature interaction self-attention unit is constructed by adding two main design elements to a standard local window self-attention unit: (1) a dual-branch structure including a local window self-attention branch and a deep convolutional branch, and (2) a bidirectional feature interaction unit. Specifically, by designing a concise dual-branch structure, a deep convolutional layer parallel to the standard local window self-attention unit is introduced to enhance feature interactions between windows. The bidirectional feature interaction unit includes a spatial interaction unit and a channel interaction unit. Information from the deep convolution branch first flows into the local window self-attention branch via the spatial interaction unit, and then information from the local window self-attention branch flows into the deep convolution branch via the spatial interaction unit. Thus, the bidirectional feature interaction unit according to the embodiment of the present invention can provide complementary cues to the dual branch structure and increase the information utilization rate. Specifically, the channel interaction unit is configured by cascading a global average pooling layer, a first convolutional layer with a convolution kernel size of 3x3, a first batch normalization layer, a second GeLU activation function layer, a second convolutional layer with a convolution kernel size of 3x3, and a first sigmoid activation function layer. This spatial interaction unit is constructed by cascading a third convolutional layer with a convolutional kernel size of 3x3, a second batch normalization layer, a third GeLU activation function layer, a fourth convolutional layer with a convolutional kernel size of 3x3, and a second sigmoid activation function layer.Next, referring to Figure 5, a mixed-scale feedforward neural network is constructed that includes two multiscale deep convolutional branches. These two multiscale deep convolutional branches achieve mixed feature extraction by alternately using deep convolutional layers with a convolutional kernel size of 5x5 and deep convolutional layers with a convolutional kernel size of 7x7, with a ReLU activation function layer connected after each deep convolutional layer, and finally, the outputs of the two branches are fused using a fifth convolutional layer with a convolutional kernel size of 1x1 to extract mixed-scale features. JPEG0007863804000019.jpg9170

[0034] Furthermore, referring to Figure 6, the mixed-feature interaction Transformer module can be constructed by integrating an efficient local feature extraction unit, a mixed-feature interaction self-attention unit, and a mixed-scale feedforward neural network. This mixed-feature interaction Transformer module is configured by sequentially cascading the efficient local feature extraction unit, the first normalization layer, the mixed-feature interaction self-attention unit, the second normalization layer, and the mixed-scale feedforward neural network.

[0035] Finally, referring to Figure 7, a single-frame image super-resolution model based on a mixed-feature interaction Transformer is constructed and trained, resulting in a trained single-frame image super-resolution model.

[0036] Step S3: The low-resolution image to be reconstructed is input into a trained single-frame image super-resolution model. A shallow feature extraction unit extracts shallow features, which are then input into a deep feature extraction unit to extract deep features. These deep features are then input into an upsampling reconstruction unit for reconstruction, resulting in a high-resolution reconstructed image.

[0037] In specific embodiments, the specific structure and computation process of the single-frame image super-resolution model are described below. The shallow feature extraction unit uses the sixth convolutional layer, and the calculation process of the shallow feature extraction unit is as follows: JPEG0007863804000020.jpg23170Conv 3×3 (·) represents the function of the 6th convolutional layer, which is a 3x3 convolutional kernel, I LR This represents a low-resolution image. Features are extracted using P mixed-feature interaction Transformer modules, F0 is transmitted to the network terminals using long-skip connections, and residual learning is performed by adding it to the output of the Pth mixed-feature interaction Transformer module. The formula is as follows: JPEG0007863804000021.jpg19170 Here, F i-1 This represents the output of the i-1th mixed feature interaction Transformer module, MF P This represents the function of the P-th mixed feature interaction Transformer module, MF 1 This represents the function of the first mixed feature interaction Transformer module, MF 2 This represents the function of the second mixed feature interaction Transformer module, MF i represents the function of the i-th mixed feature interaction Transformer module, and F i This represents the output of the i-th mixed feature interaction Transformer module, and F P0 This represents deep features, JPEG0007863804000022.jpg represents three-dimensional data with dimensions of 581703 × (H × scale) × (W × scale).

[0038] Specifically, a pre-trained single-frame image super-resolution module is used to reconstruct a low-resolution image to be reconstructed, and the reconstruction result is obtained. The pre-trained single-frame image super-resolution module consists of three parts: a shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit. For JPEG0007863804000023.jpg9170, The scaling factor `scale` is used as input. The value of `scale` is the desired scaling factor; for example, `scale` can be 2, 3, 4, or 8.

[0039] This study compares a single-frame image super-resolution method based on a mixed-feature interaction Transformer according to an embodiment of this invention with a state-of-the-art single-frame image super-resolution method. In this comparison experiment, DIV2K was used as the training set, and Set5, Se14, BSD100, and Urban100 were used as test sets, with a target scaling factor of 2. The quality of the reconstructed image was evaluated using the quantitative metrics PSNR and SSIM; higher PSNR and SSIM values ​​indicate higher SR performance. The model size and execution speed were measured using the quantitative metrics parameter count (Params) and Flops; lower parameter counts indicate smaller model size, and lower Flops values ​​indicate faster model execution speed. Designing an image super-resolution method that can generate reconstructed images with low parameter counts and Flops values ​​but high PSNR and SSIM to meet the needs of actual application scenarios is a key goal in the field of image super-resolution. As shown in Table 1, compared to other methods, the method according to the embodiment of the present application obtained the best PSNR and SSIM in four test sets with the lowest parameter values ​​and the lowest Flops values. Therefore, Table 1 clearly demonstrates that the single-frame image super-resolution method based on a mixed feature interaction Transformer according to the embodiment of the present application exhibits the best overall performance compared to other methods.

[0040] JPEG0007863804000024.jpg173170

[0041] Steps S1 to S3 above do not simply represent the order of steps, but are symbolic representations of the steps.

[0042] Furthermore, referring to Figure 8, as an example of realizing each of the illustrated methods described above, the present application provides an embodiment of a single-frame image super-resolution device based on a mixed feature interaction Transformer, the embodiment of which corresponds to the embodiment of the method shown in Figure 2, and this device can be specifically applied to various electronic devices.

[0043] An embodiment of the present application is a single-frame image super-resolution device based on a mixed feature interaction Transformer, Image acquisition module 1 configured to acquire low-resolution images to be reconstructed, A model building module 2 is configured to construct and train a single-frame image super-resolution model based on a mixed feature interaction Transformer, and to obtain a trained single-frame image super-resolution model, wherein the single-frame image super-resolution model includes a sequentially connected shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit, and the deep feature extraction unit includes a model building module 2 that includes P sequentially connected mixed feature interaction Transformer modules, The present invention provides a single-frame image super-resolution device based on a mixed feature interaction Transformer, which includes a reconstruction module 3 configured to input a low-resolution image to be reconstructed into a trained single-frame image super-resolution model, extract shallow features using a shallow feature extraction unit, input the shallow features into a deep feature extraction unit to extract deep features, input the deep features into an upsampling reconstruction unit for reconstruction, and obtain a high-resolution reconstructed image.

[0044] Referring below to Figure 9, it shows a schematic diagram of a computer device 900 suitable for realizing the electronic device according to the embodiment of the present application (for example, the server and terminal device shown in Figure 1). The electronic device shown in Figure 9 is merely an example and should not impose any limitations on the functions or scope of use of the embodiment of the present application.

[0045] As shown in Figure 9, the computer device 900 includes a central processing unit (CPU) 901 and a graphics processing unit (GPU) 902, and can perform various appropriate operations and processes according to programs stored in read-only memory (ROM) 903 or programs loaded from storage unit 909 into random access memory (RAM) 904. RAM 904 also stores various programs and data necessary for the operation of the computer device 900. The CPU 901, GPU 902, ROM 903, and RAM 904 are interconnected via a bus 905. An input / output (I / O) interface 906 is also connected to the bus 905.

[0046] The I / O interface 906 is connected to an input unit 907 including a keyboard, mouse, etc., an output unit 908 including, for example, a liquid crystal display (LCD) and speakers, a storage unit 909 including a hard disk, etc., and a communication unit 910 including a network interface card such as a LAN card or modem. The communication unit 910 performs communication processing via a network such as the Internet. A driver 911 may be connected to the I / O interface 906 as needed. Removable media 912 such as magnetic disks, optical disks, magneto-optical disks, and semiconductor memory are installed on the drive 911 as needed, and computer programs read from there are installed on the storage unit 909 as needed.

[0047] In particular, according to embodiments of the present disclosure, the process described with reference to the flowchart above can also be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product which includes a computer program mounted on a computer-readable medium which includes program code for performing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via a communication unit 910 and / or installed from removable media 912. When the computer program is executed by a central processing unit (CPU) 901 and a graphics processing unit (GPU) 902, it performs the functions limited by the present invention as described above.

[0048] The computer-readable medium described in this application may be a computer-readable signal medium, a computer-readable medium, or any combination thereof. The computer-readable medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or device, or any combination thereof. More specific examples of computer-readable media include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical memory devices, magnetic memory devices, or any suitable combination thereof. In this application, the computer-readable medium may be any tangible medium containing or storing a program used by or in combination with an instruction execution device, apparatus, or device. On the other hand, in this application, the computer-readable signal medium may include data signals propagated in the baseband or as part of a carrier carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium on which programs can be transmitted, propagated, or transmitted for use by or in combination with instruction execution units, apparatus, or devices. Program code contained in a computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0049] The computer program code for performing the operations of this invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the C language or similar programming languages. The program code may be fully executed on the user's computer, partially executed on the user's computer, executed as a standalone software package, partially executed on the user's computer, partially executed on a remote computer, or fully executed on a remote computer or server. If a remote computer is involved, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or it may be connected to an external computer (for example, connected via the Internet using an Internet service provider).

[0050] The flowcharts and block diagrams in the attached drawings illustrate the feasible architectures, functions, and operations of apparatus, methods, and computer program products relating to various embodiments of the present application. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or part of code containing one or more executable instructions for realizing a given logical function. It should also be noted that in some alternative implementations, the functions described in a block may occur in a different order than those shown in the drawings. For example, two blocks shown consecutively may actually be executed substantially in parallel, or in reverse order depending on the functions involved. Furthermore, each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, may be realized by a dedicated hardware-based device that performs a given function or operation, or by a combination of dedicated hardware and computer instructions.

[0051] The modules described in the embodiments of this application may be implemented by software or by hardware. The described modules may be provided within a processor.

[0052] The above description is merely a description of preferred embodiments and applicable technical principles of the present application. Those skilled in the art should understand that the scope of the present invention is not limited to technical solutions formed by specific combinations of the above technical features, but also includes other technical solutions formed by any combination of the above technical features, or equivalent features, without departing from the conceptual framework of the invention. For example, technical solutions formed by substituting the above features with similar functional technical features disclosed herein (but not limited to these). [Industrial applicability]

[0053] The present invention provides a single-frame image super-resolution method and apparatus based on a mixed-feature interaction Transformer, comprising P mixed-feature interaction Transformer modules, each having an extraction unit connected in sequence. This method solves the problem that the Transformer's SR method ignores potential relationships between features of different dimensions, affecting reconstruction performance.

Claims

1. A single-frame image super-resolution method based on a mixed feature interaction transformer, The steps include: acquiring a low-resolution image to be reconstructed, A step of constructing and training a single-frame image super-resolution model based on a mixed feature interaction transformer to obtain a trained single-frame image super-resolution model, wherein the single-frame image super-resolution model includes sequentially connected shallow feature extraction units, deep feature extraction units, and upsampling reconstruction units, and the deep feature extraction units include sequentially connected P mixed feature interaction transformer modules. The process includes the steps of inputting the low-resolution image to be reconstructed into the trained single-frame image super-resolution model, extracting shallow features using the shallow feature extraction unit, inputting the shallow features into the deep feature extraction unit to extract deep features, inputting the deep features into the upsampling reconstruction unit for reconstruction, and obtaining a high-resolution reconstructed image. A single-frame image super-resolution method based on a mixed feature interaction transformer, characterized in that the mixed feature interaction transformer module includes an efficient local feature extraction unit, a first layer normalization layer, a mixed feature interaction self-attention unit, a second layer normalization layer, and a mixed scale feedforward neural network, the input to the mixed feature interaction transformer module passes sequentially through the efficient local feature extraction unit and the first layer normalization layer, the output of the first layer normalization layer is added to the input to the mixed feature interaction transformer module to obtain a first mixed feature, the first mixed feature passes sequentially through the mixed feature interaction self-attention unit and the second layer normalization layer, the output of the second layer normalization layer is added to the first mixed feature to obtain a second mixed feature, the second mixed feature is input to the mixed scale feedforward neural network to obtain the output of the mixed feature interaction transformer module.

2. The efficient local feature extraction unit comprises a first shift convolutional layer, a first GeLU activation function layer, a second shift convolutional layer, an SE module, and a third shift convolutional layer, connected in sequence, and the computation process of the efficient local feature extraction unit is as follows: Here, H ＥＬＦ (•) represents a function of an efficient local feature extraction unit, F ｓｈｉｆｔ－ｃｏｎｖ (•) represents a shift convolution operation in the first shift convolution layer, the second shift convolution layer, or the third shift convolution layer, F ＳＥ The single-frame image super-resolution method based on a mixed feature interaction transformer according to claim 1, characterized in that (•) represents a function of the SE module and GeLU(•) represents a GeLU activation function.

3. The mixed feature interaction self-attention unit includes a local window self-attention branch, a deep convolution branch, and a bidirectional feature interaction unit, the bidirectional feature interaction unit includes a spatial interaction unit and a channel interaction unit, the channel interaction unit includes, in order, a global mean pooling layer, a first convolution layer, a first batch normalization layer, a second GeLU activation function layer, a second convolution layer, and a first Sigmoid activation function layer, the spatial interaction unit includes, in order, a third convolution layer, a second batch normalization layer, a third GeLU activation function layer, a fourth convolution layer, and a second Sigmoid activation function, and the local window self-attention branch includes a query linear layer, a key linear layer, a value linear layer, and a local window self-attention module. The single-frame image super-resolution method based on a mixed-feature interaction transformer according to claim 1, characterized in that the deep convolution branch includes a first deep convolution layer with a convolution kernel size of 3 × 3, local features output by the first deep convolution layer are input to the channel interaction unit to obtain channel-level dynamic weights, the channel-level dynamic weights are input to the local window self-attention branch to adaptively correct the value feature map output by the value linear layer, global features output by the local window self-attention module are input to the spatial interaction unit to obtain spatial-level dynamic weights, the spatial-level dynamic weights are input to the deep convolution branch to adaptively correct the local features.

4. The computation process for the aforementioned mixed-feature interaction self-attention unit is as follows: Here, L Ｑ , L Ｋ , L Ｖ These represent the functions of the query-linear layer, key-linear layer, and value-linear layer, respectively. Convert the data format of the value feature map V from NS ２ × C to C × H × W, and multiply by the dynamic weight W ｃａ at the channel level to adaptively correct the value feature map V, and then restore the data format to NS ２ × C, record the correction result as V', Perform calculations for the local window self-attention module to determine global features. Here, SI(•) represents a function of the spatial interaction unit, The spatial-level dynamic weight W ｓａ Global Features F ｌｏｃａｌ By multiplying by the global feature F, ｌｏｃａｌ Adaptively correct this, and the corrected result is F′ ｌｏｃａｌ Record it as, Ultimately, the global feature F ｌｏｃａｌ and F' ｌｏｃａｌ Adding them together, mixed features A single-frame image super-resolution method based on a mixed feature interaction transformer according to claim 3, characterized by obtaining the following.

5. The mixed-scale feedforward neural network includes a first branch, a second branch, and a fifth convolutional layer. The first branch includes, in sequence, a second deep convolutional layer, a first ReLU activation function layer, a third deep convolutional layer, and a second ReLU activation function layer. The second branch includes, in sequence, a fourth deep convolutional layer, a third ReLU activation function layer, a fifth deep convolutional layer, and a fourth ReLU activation function layer. The convolutional kernel size of the second and fifth deep convolutional layers is 7 × 7, and the convolutional kernel size of the third and fourth deep convolutional layers is 5 × 5. The specific computation process is as follows: The first cross-sectional feature and the second cross-sectional feature are output, the first and second cross-sectional features are stitched together and input to the fifth convolutional layer, and the output of the fifth convolutional layer is added to the second feature map. Here, ReLU(•) represents the ReLU activation function, and DwConv ５×５ (・) and DwConv ７×７ (•) represents the functions of deep convolutional layers with convolution kernels of 5x5 and 7x7, respectively. １×１ (•) represents the function of the fifth convolutional layer with a convolution kernel size of 1x1, and [•] represents the stitching operation. These represent the first cross feature and the second cross feature, respectively. A single-frame image super-resolution method based on a mixed feature interaction transformer as described in claim 1.

6. A single-frame image super-resolution method based on a mixed feature interaction transformer, The steps include: acquiring a low-resolution image to be reconstructed, A step of constructing and training a single-frame image super-resolution model based on a mixed feature interaction transformer to obtain a trained single-frame image super-resolution model, wherein the single-frame image super-resolution model includes sequentially connected shallow feature extraction units, deep feature extraction units, and upsampling reconstruction units, and the deep feature extraction units include sequentially connected P mixed feature interaction transformer modules. The process includes the steps of inputting the low-resolution image to be reconstructed into the trained single-frame image super-resolution model, extracting shallow features using the shallow feature extraction unit, inputting the shallow features into the deep feature extraction unit to extract deep features, inputting the deep features into the upsampling reconstruction unit for reconstruction, and obtaining a high-resolution reconstructed image. The specific structure and computational process of the single-frame image super-resolution model described above are as follows: The shallow feature extraction unit uses a sixth convolutional layer, and the calculation process of the shallow feature extraction unit is as follows: Here, F ｉ－１ This represents the output of the i-1th mixed feature interaction Transformer module, MF Ｐ This represents the function of the P-th mixed feature interaction Transformer module, and MF １ This represents the function of the first mixed feature interaction Transformer module, MF ２ This represents the function of the second mixed feature interaction Transformer module, MF ｉ represents the function of the i-th mixed feature interaction Transformer module, F ｉ This represents the output of the i-th mixed feature interaction Transformer module, A single-frame image super-resolution method based on a mixed feature interaction transformer, characterized by representing three-dimensional data with a shape of 3 × (H × scale) × (W × scale).

7. A single-frame image super-resolution device based on a mixed feature interaction transformer that applies the method described in any one of claims 1 to 6, An image acquisition module configured to acquire low-resolution images to be reconstructed, A model building module configured to construct and train a single-frame image super-resolution model based on a mixed feature interaction Transformer, and to obtain a trained single-frame image super-resolution model, wherein the single-frame image super-resolution model includes a sequentially connected shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit, and the deep feature extraction unit includes a model building module that includes P sequentially connected mixed feature interaction Transformer modules, A single-frame image super-resolution device based on a mixed-feature interaction transformer, characterized by including a reconstruction module configured to input the low-resolution image to be reconstructed into the trained single-frame image super-resolution model, extract shallow features using the shallow feature extraction unit, input the shallow features into the deep feature extraction unit to extract deep features, input the deep features into the upsampling reconstruction unit for reconstruction, and obtain a high-resolution reconstructed image.

8. A single-frame image super-resolution apparatus based on a mixed-feature interaction transformer, to which a single-frame image super-resolution method based on a mixed-feature interaction transformer is applied, The single-frame image super-resolution method based on the aforementioned mixed feature interaction transformer is, The steps include: acquiring a low-resolution image to be reconstructed, A step of constructing and training a single-frame image super-resolution model based on a mixed feature interaction transformer to obtain a trained single-frame image super-resolution model, wherein the single-frame image super-resolution model includes sequentially connected shallow feature extraction units, deep feature extraction units, and upsampling reconstruction units, and the deep feature extraction units include sequentially connected P mixed feature interaction transformer modules. The process includes the steps of inputting the low-resolution image to be reconstructed into the trained single-frame image super-resolution model, extracting shallow features using the shallow feature extraction unit, inputting the shallow features into the deep feature extraction unit to extract deep features, inputting the deep features into the upsampling reconstruction unit for reconstruction, and obtaining a high-resolution reconstructed image. The single-frame image super-resolution device based on the aforementioned mixed feature interaction transformer is, An image acquisition module configured to acquire low-resolution images to be reconstructed, A model building module configured to construct and train a single-frame image super-resolution model based on a mixed feature interaction Transformer, and to obtain a trained single-frame image super-resolution model, wherein the single-frame image super-resolution model includes a sequentially connected shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit, and the deep feature extraction unit includes a model building module that includes P sequentially connected mixed feature interaction Transformer modules, The reconstruction module includes a configuration that inputs the low-resolution image to be reconstructed into the trained single-frame image super-resolution model, extracts shallow features using the shallow feature extraction unit, inputs the shallow features into the deep feature extraction unit to extract deep features, inputs the deep features into the upsampling reconstruction unit for reconstruction, and obtains a high-resolution reconstructed image. The mixed-feature interaction transformer module is characterized by comprising an efficient local feature extraction unit, a first normalization layer, a mixed-feature interaction self-attention unit, a second normalization layer, and a mixed-scale feedforward neural network, thus providing a single-frame image super-resolution device based on a mixed-feature interaction transformer.

9. It is an electronic device, One or more processors, Includes a memory device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement an electronic device according to any one of claims 1 to 6.

Images