Single-frame image super-resolution method and device based on mixed feature interaction transformer

Abstract

The present invention discloses a single-frame image super-resolution method and apparatus based on a mixed feature interaction Transformer, which relates to the field of image processing. It includes the steps of obtaining a low-resolution image to be reconstructed; constructing and training a single-frame image super-resolution model based on a mixed feature interaction Transformer to obtain a trained single-frame image super-resolution model, where the single-frame image super-resolution model includes a shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit connected in sequence, and the deep feature extraction unit includes P mixed feature interaction Transformer modules connected in sequence; and inputting the low-resolution image into the trained single-frame image super-resolution model, extracting shallow features using the shallow feature extraction unit, inputting the shallow features into the deep feature extraction unit to extract deep features, inputting the deep features into the upsampling reconstruction unit, and reconstructing to obtain a high-resolution reconstructed image, which solves the problem that the Transformer's SR method ignores the potential correlation between features of different dimensions, which affects the reconstruction performance.

Description

[Technical Field]

[0001] The present invention relates to the field of image processing, and in particular to a single-frame image super-resolution method and apparatus based on mixed feature interaction Transformer. [Background technology]

[0002] Image super-resolution (SR) is an important task in computer vision and image processing. It aims to reconstruct high-resolution (HR) images with high quality from existing low-resolution (LR) images. In recent years, SR methods based on convolutional neural networks (CNNs) have dominated the field of image SR due to their powerful feature representation, end-to-end trainable paradigm, and excellent performance. However, because convolutional operations extract local features within a narrow neighborhood using a fixed sliding window, CNN-based SR methods have limited information pixels. Currently, the Transformer, as a novel alternative to CNNs, can achieve excellent performance in various low-level vision tasks.

[0003] For image SR, Liang et al. proposed an SR model based on the Swin Transformer, i.e., SwinIR. SwinIR employs a hierarchical design, restricting similarity calculation to a local window and utilizing a shifting window mechanism to enhance information interactions between windows. However, SwinIR uses window-based self-attention and abandons global information inference, limiting the performance of the Transformer. To activate more information pixels contributing to image SR, Chen et al. proposed HAT, which better aggregates information between windows by introducing channel attention. Wang et al. proposed Omni-SR, which can simultaneously model pixel-level information interactions between spatial and window dimensions. However, conventional Transformer-based SR methods generally capture spatial and inter-channel relationships through serial or parallel operations, but ignore potential associations between features across different dimensions, limiting the performance of Transformer-based SR methods. Summary of the Invention [Problem to be solved by the invention]

[0004] To address the above technical challenges, the embodiments of this application aim to propose a single-frame image super-resolution method and apparatus based on a mixed feature interaction Transformer, which solves the problem that the traditional Transformer method ignores the potential correlation between features of different dimensions and promotes feature interaction between dimensions, thereby significantly improving the global feature representation ability and detailed reconstruction ability of the image super-resolution method. [Means for solving the problem]

[0005] In a first aspect, the present invention provides a single-frame image super-resolution method based on a mixed feature interaction Transformer, comprising: acquiring a low-resolution image to be reconstructed; Constructing and training a single-frame image super-resolution model based on a mixed feature interaction Transformer to obtain a trained single-frame image super-resolution model, wherein the single-frame image super-resolution model includes a shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit connected in sequence, and the deep feature extraction unit includes P mixed feature interaction Transformer modules connected in sequence; The present invention provides a single-frame image super-resolution method based on a mixed feature interaction Transformer, which includes the steps of inputting a low-resolution image to be reconstructed into a trained single-frame image super-resolution model, extracting shallow features using a shallow feature extraction unit, inputting the shallow features into a deep feature extraction unit to extract deep features, inputting the deep features into an upsampling reconstruction unit, and reconstructing the image to obtain a high-resolution reconstructed image.

[0006] Preferably, the mixed feature interaction Transformer module includes an efficient local feature extraction unit, a first-layer normalization layer, a mixed feature interaction self-attention unit, a second-layer normalization layer, and a mixed-scale feedforward neural network, wherein an input of the mixed feature interaction Transformer module passes through the efficient local feature extraction unit and the first-layer normalization layer in order, an output of the first-layer normalization layer is added to an input of the mixed feature interaction Transformer module to obtain a first mixed feature, the first mixed feature passes through the mixed feature interaction self-attention unit and the second-layer normalization layer in order, an output of the second-layer normalization layer is added to the first mixed feature to obtain a second mixed feature, and the second mixed feature is input to the mixed-scale feedforward neural network to obtain an output of the mixed feature interaction Transformer module.

[0007] Preferably, the efficient local feature extraction unit includes a first shifted convolution layer, a first GeLU activation function layer, a second shifted convolution layer, an SE module, and a third shifted convolution layer connected in sequence, and the calculation process of the efficient local feature extraction unit is as follows: JPEG2026508397000002.jpg13170Here, H ELF (·) represents the function of the efficient local feature extraction unit, F shift-conv (·) represents the shift convolution operation of the first shift convolution layer, the second shift convolution layer, or the third shift convolution layer, and F SE (·) represents the function of the SE module, and GeLU(·) represents the GeLU activation function.

[0008] Preferably, the mixed feature interaction self-attention unit includes a local window self-attention branch, a deep convolution branch, and a bidirectional feature interaction unit, and the bidirectional feature interaction unit includes a spatial interaction unit and a channel interaction unit, and the channel interaction unit includes a global average pooling layer, a first convolution layer, a first batch normalization layer, a second GeLU activation function layer, a second convolution layer, and a first Sigmoid activation function layer, which are connected in sequence; the spatial interaction unit includes a third convolution layer, a second batch normalization layer, a third GeLU activation function layer, a fourth convolution layer, and a second Sigmoid activation function layer, which are connected in sequence; and the local window self-attention branch includes a query linear layer, a key The deep convolutional branch includes a first deep convolutional layer with a convolution kernel size of 3x3, the local features output by the first deep convolutional layer are input to a channel interaction unit to obtain channel-level dynamic weights, the channel-level dynamic weights are input to the local window self-attention branch to adaptively correct the value feature map output by the value linear layer, the global features output by the local window self-attention module are input to a spatial interaction unit to obtain spatial-level dynamic weights, and the spatial-level dynamic weights are input to the deep convolutional branch to adaptively correct the local features.

[0009] Preferably, the calculation process of the mixed feature interaction self-attention unit is as follows: JPEG2026508397000003.jpg249170where SI(·) represents the spatial interaction unit function; Dynamic weights W at the spatial level sa The global feature F local By multiplying with local is adaptively corrected, and the correction result is F′ local Record it as JPEG2026508397000004.jpg23170

[0010] Preferably, the mixed-scale feedforward neural network includes a first branch, a second branch, and a fifth convolutional layer, wherein the first branch includes a second deep convolutional layer, a first ReLU activation function layer, a third deep convolutional layer, and a second ReLU activation function layer connected in sequence; the second branch includes a fourth deep convolutional layer, a third ReLU activation function layer, a fifth deep convolutional layer, and a fourth ReLU activation function layer connected in sequence; the convolutional kernel sizes of the second deep convolutional layer and the fifth deep convolutional layer are 7×7, and the convolutional kernel sizes of the third deep convolutional layer and the fourth deep convolutional layer are 5×5; and the specific calculation process is as follows: JPEG2026508397000005.jpg57170The first cross feature and the second cross feature are output respectively, and the first cross feature and the second cross feature are stitched and then input to the fifth convolutional layer. The output of the fifth convolutional layer is added to the second feature map, JPEG2026508397000006.jpg70170Here, ReLU(·) represents the ReLU activation function, and DwConv 5×5 (·) and DwConv 7×7 (·) represent the functions of the deep convolutional layers with convolution kernels of 5×5 and 7×7, respectively, and Conv 1×1 (·) represents the function of the fifth convolutional layer with a convolution kernel size of 1×1, and [·] represents the stitching operation. JPEG2026508397000007.jpg15170

[0011] Preferably, the specific structure and calculation process of the single-frame image super-resolution model can be The shallow feature extraction unit uses the sixth convolution layer, and the calculation process of the shallow feature extraction unit is as follows: JPEG2026508397000008.jpg23170Conv 3×3 (·) represents the function of the sixth convolution layer with a convolution kernel of 3 × 3, and I LR represents the low-resolution image, P mixed feature interaction Transformer modules are used to extract features, and F0 is transmitted to the network end using long skip connections, and added to the output of the Pth mixed feature interaction Transformer module for residual learning, which is expressed as: JPEG2026508397000009.jpg20170where, F i-1 represents the output of the (i-1)th mixed feature interaction Transformer module, and MF P represents the function of the P-th mixed feature interaction transformer module, and MF 1 represents the function of the first mixed feature interaction Transformer module, and MF 2 represents the function of the second mixed feature interaction transformer module, and MF i represents the function of the i-th mixed feature interaction Transformer module, and F i represents the output of the i-th mixed feature interaction Transformer module, JPEG2026508397000010.jpg32170where, f up (·) represents the function of the subpixel convolution layer, and Conv 3×3 (·) represents the function of the seventh convolutional layer, and I SR represents the high-resolution reconstructed image, JPEG2026508397000011.jpg111703 represents three-dimensional data with a shape of (H×scale)×(W×scale).

[0012] In a second aspect, the present invention provides a single-frame image super-resolution device based on a mixed feature interaction Transformer, comprising: an image acquisition module configured to acquire a low-resolution image to be reconstructed; a model construction module configured to construct and train a single-frame image super-resolution model based on a mixed feature interaction Transformer to obtain a trained single-frame image super-resolution model, wherein the single-frame image super-resolution model includes a shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit connected in sequence, and the deep feature extraction unit includes P mixed feature interaction Transformer modules connected in sequence; The present invention provides a single-frame image super-resolution device based on a mixed feature interaction Transformer, which includes a reconstruction module configured to input a low-resolution image to be reconstructed into a trained single-frame image super-resolution model, extract shallow features using a shallow feature extraction unit, input the shallow features into a deep feature extraction unit to extract deep features, input the deep features into an upsampling reconstruction unit, and perform reconstruction to obtain a high-resolution reconstructed image.

[0013] In a third aspect, the present invention provides an electronic device comprising one or more processors and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a method according to any of the implementations of the first aspect. [Effects of the Invention]

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] (1) The mixed feature interaction self-attention unit in the single-frame image super-resolution method based on the mixed feature interaction Transformer of the present invention uses a dual-branch structure combined with a bidirectional feature interaction unit. The dual-branch structure is based on a standard local window self-attention unit, and adds a deep convolutional branch parallel to the local window self-attention unit, thereby enhancing the Transformer's inter-window feature interaction ability. The bidirectional feature interaction unit can provide complementary clues to the dual-branch structure, fully considering the complementarity between different types of features, and significantly improving the information utilization rate and image super-resolution performance.

[0016] (2) The single-frame image super-resolution method based on the mixed feature interaction Transformer of the present invention can solve the problem that the traditional Transformer method ignores the potential correlation between features of different dimensions, and by promoting feature interactions between dimensions, it significantly improves the global feature representation ability and detailed reconstruction ability of the image super-resolution method.

[0017] (3) Compared with conventional single-frame image super-resolution methods, the single-frame image super-resolution method based on the mixed feature interaction Transformer of the present invention has a lower parameter amount and Flops value, and has the best integration performance, thereby achieving high-performance image super-resolution reconstruction with fewer computing resources. [Brief explanation of the drawings]

[0018] In order to more clearly describe the technical solutions in the embodiments of the present invention, the following will briefly describe the drawings necessary for describing the embodiments. However, the drawings in the following description are only some embodiments of the present invention, and it is obvious to those skilled in the art that other drawings can be obtained based on these drawings without any creative efforts. [Figure 1] FIG. 1 is a diagram of an exemplary device architecture to which an embodiment of the present application can be applied. [Figure 2]1 is a schematic flowchart of a single-frame image super-resolution method based on a mixed feature interaction Transformer in an embodiment of the present application; [Figure 3] 1 is a structural schematic diagram of an efficient local feature extraction unit of a single-frame image super-resolution method based on a mixed feature interaction Transformer in an embodiment of the present application. [Figure 4] 1 is a structural schematic diagram of a mixed feature interaction self-attention unit of a single-frame image super-resolution method based on a mixed feature interaction Transformer in an embodiment of the present application. [Figure 5] 1 is a schematic diagram of the structure of a mixed-scale feedforward neural network of a single-frame image super-resolution method based on a mixed feature interaction Transformer in an embodiment of the present application. [Figure 6] 1 is a structural schematic diagram of the mixed feature interaction Transformer module of the single-frame image super-resolution method based on the mixed feature interaction Transformer in an embodiment of the present application; [Figure 7] 1 is a schematic structural diagram of a single-frame image super-resolution model based on a mixed feature interaction Transformer in a single-frame image super-resolution method based on a mixed feature interaction Transformer in an embodiment of the present application; [Figure 8] 1 is a schematic diagram of a single-frame image super-resolution device based on a mixed feature interaction Transformer in an embodiment of the present application. [Figure 9] 1 is a structural schematic diagram of a computer device suitable for implementing the electronic device in the embodiments of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0019] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be described in more detail below with reference to the drawings. However, it is clear that the described embodiments are only some of the embodiments of the present invention, and do not represent all of the embodiments. All other embodiments that can be obtained by those skilled in the art based on the embodiments of the present invention without any creative efforts fall within the scope of protection of the present invention.

[0020] FIG. 1 illustrates an exemplary device architecture 100 to which the mixed feature interaction Transformer-based single-frame image super-resolution method or mixed feature interaction Transformer-based single-frame image super-resolution device according to the embodiments of the present application can be applied.

[0021] 1, device architecture 100 may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. Network 104 provides a medium for communication links between first terminal device 101, second terminal device 102, third terminal device 103, and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0022] It should be noted that the single-frame image super-resolution method based on the mixed feature interaction Transformer provided in the embodiments of the present application can be executed by the server 105, and can also be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Accordingly, the single-frame image super-resolution device based on the mixed feature interaction Transformer may be provided in the server 105, or in the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0023] The number of terminal devices, networks, and servers in Figure 1 is for illustrative purposes only. Any number of terminal devices, networks, and servers may be included as needed. If there is no need to obtain processing data remotely, the above device architecture may include only servers or terminal devices without including a network.

[0024] FIG. 2 shows a single-frame image super-resolution method based on a mixed feature interaction Transformer provided in an embodiment of the present application, which includes steps S1 to S3.

[0025] Step S1: A low-resolution image to be reconstructed is obtained.

[0026] Specifically, a low-resolution image of the object to be reconstructed is acquired, and the low-resolution image is a single-frame image.

[0027] Step S2: Construct and train a single-frame image super-resolution model based on a mixed feature interaction Transformer to obtain a trained single-frame image super-resolution model, where the single-frame image super-resolution model includes a shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit connected in sequence, and the deep feature extraction unit includes P mixed feature interaction Transformer modules connected in sequence.

[0028] In a specific embodiment, the mixed feature interaction Transformer module includes an efficient local feature extraction unit, a first-layer normalization layer, a mixed feature interaction self-attention unit, a second-layer normalization layer, and a mixed-scale feedforward neural network, where the input of the mixed feature interaction Transformer module passes through the efficient local feature extraction unit and the first-layer normalization layer in order, the output of the first-layer normalization layer is added to the input of the mixed feature interaction Transformer module to obtain a first mixed feature, the first mixed feature passes through the mixed feature interaction self-attention unit and the second-layer normalization layer in order, the output of the second-layer normalization layer is added to the first mixed feature to obtain a second mixed feature, and the second mixed feature is input to the mixed-scale feedforward neural network to obtain the output of the mixed feature interaction Transformer module.

[0029] In a specific embodiment, the efficient local feature extraction unit includes a first shifted convolution layer, a first GeLU activation function layer, a second shifted convolution layer, an SE module, and a third shifted convolution layer connected in sequence, and the calculation process of the efficient local feature extraction unit is as follows: JPEG2026508397000012.jpg12170 where H ELF(·) represents the function of the efficient local feature extraction unit, F shift-conv (·) represents the shift convolution operation of the first shift convolution layer, the second shift convolution layer, or the third shift convolution layer, and F SE (·) represents the function of the SE module, and GeLU(·) represents the GeLU activation function.

[0030] In a specific embodiment, the mixed feature interaction self-attention unit includes a local window self-attention branch, a deep convolution branch, and a bidirectional feature interaction unit, the bidirectional feature interaction unit includes a spatial interaction unit and a channel interaction unit, the channel interaction unit includes a global average pooling layer, a first convolution layer, a first batch normalization layer, a second GeLU activation function layer, a second convolution layer, and a first Sigmoid activation function layer connected in sequence, the spatial interaction unit includes a third convolution layer, a second batch normalization layer, a third GeLU activation function layer, a fourth convolution layer, and a second Sigmoid activation function layer connected in sequence, the local window self-attention branch includes a query linear layer, The deep convolutional branch includes a key linear layer, a value linear layer, and a local window self-attention module. The deep convolutional branch includes a first deep convolutional layer with a convolution kernel size of 3x3. The local features output by the first deep convolutional layer are input to a channel interaction unit to obtain channel-level dynamic weights. The channel-level dynamic weights are input to the local window self-attention branch to adaptively correct the value feature map output by the value linear layer. The global features output by the local window self-attention module are input to a spatial interaction unit to obtain spatial-level dynamic weights. The spatial-level dynamic weights are input to the deep convolutional branch to adaptively correct the local features.

[0031] In a specific embodiment, the calculation process of the mixed feature interaction self-attention unit is as follows: JPEG2026508397000013.jpg52170 where DwConv 3×3(·) represents the function of the first deep convolutional layer, JPEG2026508397000014.jpg92170where, L Q , L K , L V represent the functions of the query linear layer, key linear layer, and value linear layer, respectively. The data format of the value feature map V is NS 2 ×C to C×H×W, and channel-level dynamic weights W ca Multiply by V to adaptively correct the value feature map V, and then change the data format to NS 2 × C, and record the correction result as V'. JPEG2026508397000015.jpg60170where SI(·) represents the spatial interaction unit function; Dynamic weights W at the spatial level sa The global feature F local By multiplying with local is adaptively corrected, and the correction result is F′ local Record it as Finally, the global features F local and F' local Adding JPEG2026508397000016.jpg12170

[0032] In a specific embodiment, the mixed-scale feedforward neural network includes a first branch, a second branch, and a fifth convolutional layer, wherein the first branch includes a second deep convolutional layer, a first ReLU activation function layer, a third deep convolutional layer, and a second ReLU activation function layer, which are connected in sequence; the second branch includes a fourth deep convolutional layer, a third ReLU activation function layer, a fifth deep convolutional layer, and a fourth ReLU activation function layer, which are connected in sequence; the convolutional kernel sizes of the second deep convolutional layer and the fifth deep convolutional layer are 7×7, and the convolutional kernel sizes of the third deep convolutional layer and the fourth deep convolutional layer are 5×5; and the specific calculation process is as follows: JPEG2026508397000017.jpg56170The first cross feature and the second cross feature are output respectively, and the first cross feature and the second cross feature are stitched and then input to the fifth convolutional layer. The output of the fifth convolutional layer is added to the second feature map, JPEG2026508397000018.jpg69170where ReLU(·) represents the ReLU activation function, and DwConv 5×5 (·) and DwConv 7×7 (·) represent the functions of the deep convolutional layers with convolution kernels of 5×5 and 7×7, respectively, and Conv 1×1 (·) represents the function of the fifth convolutional layer with a convolution kernel size of 1×1, and [·] represents the stitching operation. JPEG2026508397000019.jpg20170

[0033] Specifically, referring to Figure 3, an efficient local feature extraction unit is first constructed. This efficient local feature extraction unit is composed of a cascade of a first shifted convolutional layer, a first GeLU activation function layer, a second shifted convolutional layer, an SE module, and a third shifted convolutional layer. The SE module is a squeeze-excitation module. Referring to Figure 4, a mixed feature interaction self-attention unit is then constructed. The mixed feature interaction self-attention unit is constructed based on a standard local window self-attention unit by adding two key design features: (1) a dual-branch structure including a local window self-attention branch and a deep convolutional branch, and (2) a bidirectional feature interaction unit. Specifically, by designing a simple dual-branch structure, a parallel deep convolutional layer is introduced to the standard local window self-attention unit to enhance inter-window feature interaction. The bidirectional feature interaction unit includes a spatial interaction unit and a channel interaction unit, where information from the deep convolutional branch first flows into the local window self-attention branch through the spatial interaction unit, and then information from the local window self-attention branch flows into the deep convolutional branch through the spatial interaction unit. Therefore, the bidirectional feature interaction unit according to the embodiment of the present application can provide complementary clues to the dual-branch structure and improve information utilization. Specifically, the channel interaction unit is cascaded with a global average pooling layer, a first convolutional layer with a convolution kernel size of 3x3, a first batch normalization layer, a second GeLU activation function layer, a second convolutional layer with a convolution kernel size of 3x3, and a first sigmoid activation function layer. This spatial interaction unit consists of a cascade of a third convolutional layer with a convolution kernel size of 3x3, a second batch normalization layer, a third GeLU activation function layer, a fourth convolutional layer with a convolution kernel size of 3x3, and a second sigmoid activation function layer.Next, referring to Figure 5, a mixed-scale feedforward neural network is constructed, which includes two multi-scale deep convolutional branches. The two multi-scale deep convolutional branches alternately use a deep convolutional layer with a convolutional kernel size of 5x5 and a deep convolutional layer with a convolutional kernel size of 7x7 to realize mixed feature extraction. Each deep convolutional layer is followed by a ReLU activation function layer. Finally, a fifth convolutional layer with a convolutional kernel size of 1x1 is used to fuse the outputs of the two branches to generate mixed-scale features. JPEG2026508397000020.jpg9170

[0034] Furthermore, referring to Fig. 6, a mixed feature interaction Transformer module can be constructed by integrating an efficient local feature extraction unit, a mixed feature interaction self-attention unit, and a mixed-scale feedforward neural network. This mixed feature interaction Transformer module is composed of an efficient local feature extraction unit, a first normalization layer, a mixed feature interaction self-attention unit, a second normalization layer, and a mixed-scale feedforward neural network, which are cascaded in order.

[0035] Finally, referring to FIG. 7, a single-frame image super-resolution model based on a mixed feature interaction Transformer is constructed and trained, and a trained single-frame image super-resolution model is obtained.

[0036] Step S3: The low-resolution image to be reconstructed is input into the trained single-frame image super-resolution model, shallow features are extracted by the shallow feature extraction unit, the shallow features are input into the deep feature extraction unit to extract deep features, and the deep features are input into the upsampling reconstruction unit for reconstruction to obtain a high-resolution reconstructed image.

[0037] In a specific embodiment, the specific structure and calculation process of the single-frame image super-resolution model are as follows: The shallow feature extraction unit uses the sixth convolution layer, and the calculation process of the shallow feature extraction unit is as follows: JPEG2026508397000021.jpg23170Conv 3×3 (·) represents the function of the sixth convolution layer with a convolution kernel of 3 × 3, and I LR represents the low-resolution image, P mixed feature interaction Transformer modules are used to extract features, and F0 is transmitted to the network end using long skip connections, and added to the output of the Pth mixed feature interaction Transformer module for residual learning, which is expressed as: JPEG2026508397000022.jpg19170where, F i-1 represents the output of the (i-1)th mixed feature interaction Transformer module, and MF P represents the function of the P-th mixed feature interaction transformer module, and MF 1 represents the function of the first mixed feature interaction Transformer module, and MF 2 represents the function of the second mixed feature interaction transformer module, and MF i represents the function of the i-th mixed feature interaction Transformer module, and F i represents the output of the i-th mixed feature interaction Transformer module, and F P0 represents the deep features, JPEG2026508397000023.jpg581703×(H×scale)×(W×scale) represents three-dimensional data of the shape.

[0038] Specifically, a trained single-frame image super-resolution module is used to reconstruct a low-resolution image of a reconstruction target to obtain a reconstruction result. The trained single-frame image super-resolution module is composed of three parts: a shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit. For JPEG2026508397000024.jpg9170, The scaling factor scale is used as input. The value of scale is the required magnification, for example scale can be 2, 3, 4, or 8.

[0039] The present application compares a single-frame image super-resolution method based on a mixed feature interaction Transformer with current state-of-the-art single-frame image super-resolution methods. In this comparative experiment, DIV2K was used as the training set, and Set5, Se14, BSD100, and Urban100 were used as the test set. The target scaling factor was set to 2. The quantitative metrics PSNR and SSIM were used to evaluate the quality of the reconstructed image; higher PSNR and SSIM values ​​indicate better SR performance. The quantitative metrics parameter size (Params) and Flops were used to measure the model size and execution speed; lower parameter size indicates a smaller model size, and lower Flops indicates a faster model execution speed. To meet the needs of practical application scenarios, designing an image super-resolution method that can generate reconstructed images with high PSNR and SSIM despite low parameter size and Flops values ​​is an important goal in the field of image super-resolution. As shown in Table 1, compared with other methods, the method according to the present embodiment achieved the highest PSNR and SSIM in the four test sets with the lowest parameter amount and the lowest Flops value. Therefore, Table 1 fully demonstrates that the single-frame image super-resolution method based on the mixed feature interaction Transformer according to the present embodiment exhibits the best overall performance compared with other methods.

[0040] JPEG2026508397000025.jpg173170

[0041] The above steps S1 to S3 do not simply represent the order of the steps, but are symbolic representations of the steps.

[0042] Further, referring to Figure 8, as an example of realizing the methods illustrated above, the present application provides an embodiment of a single-frame image super-resolution device based on a mixed feature interaction Transformer, which embodiment of the device corresponds to the embodiment of the method illustrated in Figure 2, and which can be specifically applied to various electronic devices.

[0043] An embodiment of the present application is a single-frame image super-resolution device based on a mixed feature interaction Transformer, comprising: an image acquisition module 1 configured to acquire a low-resolution image to be reconstructed; a model construction module 2 configured to construct and train a single-frame image super-resolution model based on a mixed feature interaction Transformer to obtain a trained single-frame image super-resolution model, wherein the single-frame image super-resolution model includes a shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit connected in sequence, and the deep feature extraction unit includes P mixed feature interaction Transformer modules connected in sequence; The present invention provides a single-frame image super-resolution device based on a mixed feature interaction Transformer, which includes a reconstruction module 3 configured to input a low-resolution image to be reconstructed into a trained single-frame image super-resolution model, extract shallow features using a shallow feature extraction unit, input the shallow features into a deep feature extraction unit to extract deep features, input the deep features into an upsampling reconstruction unit, and perform reconstruction to obtain a high-resolution reconstructed image.

[0044] 9, which shows a schematic structural diagram of a computer device 900 suitable for realizing an electronic device according to an embodiment of the present application (e.g., the server or terminal device shown in FIG. 1). The electronic device shown in FIG. 9 is merely an example and should not limit the functions or scope of use of the embodiment of the present application.

[0045] 9, a computer device 900 includes a central processing unit (CPU) 901 and a graphics processing unit (GPU) 902, and can perform various appropriate operations and processes according to a program stored in a read-only memory (ROM) 903 or a program loaded from a storage unit 909 into a random access memory (RAM) 904. The RAM 904 also stores various programs and data necessary for the operation of the computer device 900. The CPU 901, GPU 902, ROM 903, and RAM 904 are interconnected via a bus 905. An input / output (I / O) interface 906 is also connected to the bus 905.

[0046] Connected to the I / O interface 906 are an input unit 907 including a keyboard, a mouse, etc., an output unit 908 including, for example, a liquid crystal display (LCD) or the like and a speaker, a storage unit 909 including a hard disk or the like, and a communication unit 910 including a network interface card such as a LAN card or a modem. The communication unit 910 performs communication processing via a network such as the Internet. A driver 911 may be connected to the I / O interface 906 as needed. Removable media 912 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is installed in the drive 911 as needed, and computer programs read from the media are installed in the storage unit 909 as needed.

[0047] In particular, according to embodiments of the present disclosure, the processes described with reference to the flowcharts above may also be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product including a computer program embodied on a computer-readable medium containing program code for performing the methods illustrated in the flowcharts. In such embodiments, the computer program may be downloaded and installed from a network via a communication unit 910 and / or may be installed from removable media 912. When executed by a central processing unit (CPU) 901 and a graphics processing unit (GPU) 902, the computer program performs the functions defined in the methods of the present invention described above.

[0048] It should be noted that the computer-readable medium referred to herein may be a computer-readable signal medium, a computer-readable medium, or any combination thereof. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor apparatus, device, or any combination thereof. More specific examples of the computer-readable medium include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution apparatus, apparatus, or device. In this application, a computer-readable signal medium may include a propagated data signal, in baseband or as part of a carrier, carrying computer-readable program code. Such propagated data signals may take various forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can transmit, propagate, or transmit a program for use by or in connection with an instruction execution apparatus, apparatus, or device. The program code contained in the computer-readable medium may be transmitted over any suitable medium, including, but not limited to, wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0049] Computer program code for carrying out operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a separate software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. When remote computers are involved, the remote computers may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., connected via the Internet using an Internet Service Provider).

[0050] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams may represent a module, program segment, or portion of code, including one or more executable instructions for implementing a given logical function. It should also be noted that in some alternative implementations, the functions described in the blocks may occur in a different order than that described in the drawings. For example, two blocks shown in succession may, in fact, be executed substantially in parallel or may be executed in the reverse order, depending on the functionality involved. Furthermore, each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based apparatus that performs a given function or operation, or by a combination of dedicated hardware and computer instructions.

[0051] The modules described in the embodiments of the present application may be implemented by software or hardware. The described modules may be provided in a processor.

[0052] The above description merely describes preferred embodiments and applied technical principles of the present application. Those skilled in the art should understand that the scope of the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but also includes other technical solutions formed by any combination of the above technical features or features equivalent thereto, as long as it does not deviate from the above inventive concept. For example, a technical solution formed by mutually replacing the above features with technical features having similar functions (but not limited to) disclosed in the present application. [Industrial Applicability]

[0053] The single-frame image super-resolution method and apparatus based on the mixed feature interaction Transformer of the present invention includes P mixed feature interaction Transformer modules with extraction units connected in sequence. A low-resolution image is input to a trained single-frame image super-resolution model, shallow features are extracted by a shallow feature extraction unit, the shallow features are input to a deep feature extraction unit to extract deep features, and the deep features are input to an upsampling reconstruction unit for reconstruction to obtain a high-resolution reconstructed image. This solves the problem that the Transformer SR method ignores the potential correlation between features of different dimensions, which affects reconstruction performance.

Claims

1. A single-frame image super-resolution method based on mixed feature interaction transformer, comprising: acquiring a low-resolution image to be reconstructed; Constructing and training a single-frame image super-resolution model based on a mixed feature interaction Transformer to obtain a trained single-frame image super-resolution model, wherein the single-frame image super-resolution model includes a shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit connected in series, and the deep feature extraction unit includes P mixed feature interaction Transformer modules connected in series; a step of inputting the low-resolution image to be reconstructed into the trained single-frame image super-resolution model, extracting shallow features using the shallow feature extraction unit, inputting the shallow features into the deep feature extraction unit to extract deep features, inputting the deep features into the upsampling reconstruction unit, and reconstructing to obtain a high-resolution reconstructed image.

2. 2. The method for single-frame image super-resolution based on the mixed feature interaction Transformer of claim 1, wherein the mixed feature interaction Transformer module includes an efficient local feature extraction unit, a first-layer normalization layer, a mixed feature interaction self-attention unit, a second-layer normalization layer, and a mixed-scale feedforward neural network, wherein an input of the mixed feature interaction Transformer module passes through the efficient local feature extraction unit and the first-layer normalization layer in order, an output of the first-layer normalization layer is added to an input of the mixed feature interaction Transformer module to obtain a first mixed feature, the first mixed feature passes through the mixed feature interaction self-attention unit and the second-layer normalization layer in order, an output of the second-layer normalization layer is added to the first mixed feature to obtain a second mixed feature, and the second mixed feature is input to the mixed-scale feedforward neural network to obtain an output of the mixed feature interaction Transformer module.

3. The efficient local feature extraction unit includes a first shifted convolution layer, a first GeLU activation function layer, a second shifted convolution layer, an SE module, and a third shifted convolution layer connected in sequence, and the calculation process of the efficient local feature extraction unit is as follows: Here, H ＥＬＦ (·) represents the function of the efficient local feature extraction unit, F ｓｈｉｆｔ－ｃｏｎｖ (·) represents the shift convolution operation of the first shift convolution layer, the second shift convolution layer, or the third shift convolution layer, and F ＳＥ The method for single-frame image super-resolution based on mixed feature interaction transformer according to claim 2, characterized in that (·) represents the function of SE module, and GeLU(·) represents the GeLU activation function.

4. The mixed feature interaction self-attention unit includes a local window self-attention branch, a deep convolution branch, and a bidirectional feature interaction unit, the bidirectional feature interaction unit includes a spatial interaction unit and a channel interaction unit, the channel interaction unit includes a global average pooling layer, a first convolution layer, a first batch normalization layer, a second GeLU activation function layer, a second convolution layer, and a first sigmoid activation function layer, which are connected in sequence, the spatial interaction unit includes a third convolution layer, a second batch normalization layer, a third GeLU activation function layer, a fourth convolution layer, and a second sigmoid activation function layer, which are connected in sequence, and the local window self-attention branch includes a query linear layer, a key linear layer, a value linear layer, and a local window self-attention module.

3. The method for single-frame image super-resolution based on a mixed feature interaction transformer according to claim 2, wherein the deep convolutional branch includes a first deep convolutional layer with a convolution kernel size of 3x3; the local features output by the first deep convolutional layer are input to the channel interaction unit to obtain channel-level dynamic weights, which are input to the local window self-attention branch to adaptively correct the value feature map output by the value linear layer; the global features output by the local window self-attention module are input to the spatial interaction unit to obtain spatial-level dynamic weights, which are input to the deep convolutional branch to adaptively correct the local features.

5. The calculation process of the mixed feature interaction self-attention unit is as follows: Here, L Ｑ , L Ｋ , L Ｖ represent the functions of the query linear layer, key linear layer, and value linear layer, respectively. The data format of the value feature map V is NS ２ ×C to C×H×W, and the dynamic weight W ｃａ , and adaptively correct the value feature map V, and then change the data format to NS ２ × C, and record the correction result as V′. Perform the calculation of the local window self-attention module to estimate the global features. where SI(·) represents the spatial interaction unit function, The dynamic weights W of the spatial levels ｓａ The global feature F ｌｏｃａｌ By multiplying it by ｌｏｃａｌ is adaptively corrected, and the correction result is F' ｌｏｃａｌ Record it as Finally, the global features F ｌｏｃａｌ and F' ｌｏｃａｌ Adding these, the mixed feature The method for single-frame image super-resolution based on mixed feature interaction transformer according to claim 4, characterized in that:

6. The mixed-scale feedforward neural network includes a first branch, a second branch, and a fifth convolutional layer, wherein the first branch includes a second deep convolutional layer, a first ReLU activation function layer, a third deep convolutional layer, and a second ReLU activation function layer, which are connected in sequence; the second branch includes a fourth deep convolutional layer, a third ReLU activation function layer, a fifth deep convolutional layer, and a fourth ReLU activation function layer, which are connected in sequence; the convolutional kernel sizes of the second deep convolutional layer and the fifth deep convolutional layer are 7×7, and the convolutional kernel sizes of the third deep convolutional layer and the fourth deep convolutional layer are 5×5; and the specific calculation process is as follows: outputting a first cross feature and a second cross feature, stitching the first cross feature and the second cross feature, and inputting the stitched first cross feature and the second cross feature into the fifth convolutional layer; and adding the output of the fifth convolutional layer to the second feature map; where ReLU(·) represents the ReLU activation function, and DwConv ５×５ (・) and DwConv ７×７ (·) represent the functions of deep convolutional layers with convolution kernels of 5×5 and 7×7, respectively, and Conv １×１ (·) represents the function of the fifth convolutional layer with a convolution kernel size of 1×1, and [·] represents the stitching operation. represent the first cross feature and the second cross feature, respectively. The method for single-frame image super-resolution based on mixed feature interaction transformer according to claim 2 .

7. The specific structure and calculation process of the single-frame image super-resolution model are as follows: The shallow feature extraction unit uses a sixth convolution layer, and the calculation process of the shallow feature extraction unit is as follows: Here, F ｉ－１ represents the output of the i-1th mixed feature interaction Transformer module, and MF Ｐ represents the function of the P-th mixed feature interaction Transformer module, and MF １ represents the function of the first mixed feature interaction Transformer module, and MF ２ represents the function of the second mixed feature interaction Transformer module, and MF ｉ represents the function of the i-th mixed feature interaction Transformer module, and F ｉ represents the output of the i-th mixed feature interaction Transformer module, The method for single-frame image super-resolution based on the mixed feature interaction transformer of claim 1, characterized in that it represents three-dimensional data of a shape of 3×(H×scale)×(W×scale).

8. A single-frame image super-resolution device based on a mixed feature interaction transformer, which applies the method according to any one of claims 1 to 7, an image acquisition module configured to acquire a low-resolution image to be reconstructed; a model construction module configured to construct and train a single-frame image super-resolution model based on a mixed feature interaction Transformer to obtain a trained single-frame image super-resolution model, wherein the single-frame image super-resolution model includes a shallow feature extraction unit, a deep feature extraction unit, and an upsampling reconstruction unit connected in series, and the deep feature extraction unit includes P mixed feature interaction Transformer modules connected in series; a reconstruction module configured to input the low-resolution image to be reconstructed into the trained single-frame image super-resolution model, extract shallow features using the shallow feature extraction unit, input the shallow features into the deep feature extraction unit to extract deep features, input the deep features into the upsampling reconstruction unit, and perform reconstruction to obtain a high-resolution reconstructed image.

9. The single-frame image super-resolution device based on the mixed feature interaction Transformer of claim 8, wherein the mixed feature interaction Transformer module includes an efficient local feature extraction unit, a first normalization layer, a mixed feature interaction self-attention unit, a second normalization layer, and a mixed-scale feedforward neural network.

10. An electronic device, one or more processors; a storage device for storing one or more programs; An electronic device, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

Images