An image super-resolution method based on an image pre-training strategy

Abstract

The application discloses an image super-resolution method based on an image pre-training strategy. The method comprises the following steps: collecting a target image; inputting the target image into a trained image super-resolution model to output a super-resolution image, wherein the image super-resolution model comprises a shallow feature extraction module, a deep feature extraction module and a high-resolution reconstruction module in sequence; the shallow feature extraction module extracts shallow features for the input target image; the deep feature extraction module extracts multi-level intermediate features by adopting a channel attention mechanism and a self-attention mechanism; and the high-resolution reconstruction module takes the output features from the deep feature extraction module and the output features from the shallow feature extraction module transmitted through a residual connection as inputs to obtain a high-resolution image relative to the input target image. The application improves the performance of image super-resolution in terms of numerical indicators and visual effects, and has strong versatility.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and more particularly, to an image super-resolution method based on an image pre-training strategy. BACKGROUND

[0002] Image super-resolution refers to recovering a high-resolution image from a low-resolution image or image sequence, and has important application value in the fields of image processing, computer photography, high-definition video, security monitoring, etc. Image super-resolution can be implemented using a convolutional neural network (CNN) or a Transformer model. Taking the Transformer as an example, it is a network structure based on a self-attention mechanism. Due to its strong representation ability, it has shown great potential in the field of computer vision. In addition to successful applications in many high-level tasks such as image classification, detection, segmentation, etc., many methods based on the Transformer structure have recently emerged in the field of low-level image processing.

[0003] Although the Transformer structure seems to have stronger representation ability than the convolutional neural network, the existing image super-resolution method based on the Transformer structure (i.e. SwinIR) still has defects. This is because the existing method can only utilize information within a limited input range, and is even less effective than some traditional CNN-based methods such as RCAN. On the other hand, since the existing method uses a window-based self-attention mechanism, the intermediate features obtained often have obvious blocking artifacts, which is not conducive to the network learning better restoration results. In addition, the Transformer-based structure usually requires large-scale data for training, and the existing commonly used data sets are not sufficient to develop the potential of this structure.

[0004] In summary, the current image super-resolution method based on the Transformer structure has a very limited range of utilization of input information, and the intermediate features of the network can be observed to have obvious blocking artifacts, which means that the performance of this method in image super-resolution is still limited, and there is a lot of room for improvement. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art, and to provide an image super-resolution method based on an image pre-training strategy, which comprises the following steps:

[0006] Collecting a target image;

[0007] inputting a target image into a trained image super-resolution model, outputting a super-resolution image, the image super-resolution model comprising a shallow feature extraction module, a deep feature extraction module and a high-resolution reconstruction module in sequence, wherein the shallow feature extraction module extracts shallow features for the input target image, the deep feature extraction module extracts multi-level intermediate features by adopting a channel attention mechanism and a self-attention mechanism, and the high-resolution reconstruction module takes as input output features from the deep feature extraction module and output features from the shallow feature extraction module transmitted via a residual connection, and obtains a high-resolution image relative to the input target image.

[0008] Compared with the prior art, the present application has the advantages that the range of input information used by the network can be greatly increased for the existing network structure, and the blocking effect phenomenon existing in the intermediate features of the existing method is greatly improved. In addition, the present application further develops the potential of the network by using a large-scale data pre-training strategy for the same task, so that the image super-resolution function can be better realized, and the performance significantly exceeds the current most advanced method.

[0009] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0011] Figure 1 is a flowchart of an image super-resolution method based on an image pre-training strategy according to an embodiment of the present application;

[0012] Figure 2 is a process schematic diagram of an image super-resolution method based on an image pre-training strategy according to an embodiment of the present application;

[0013] Figure 3 is a schematic diagram of an overlapping cross-attention module according to an embodiment of the present application;

[0014] Figure 4 is an image effect comparison chart according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. Note that the relative arrangement, numerical expressions, and numerical values of the components and steps set forth in these embodiments are not limiting to the scope of the present application, unless otherwise specifically stated.

[0016] The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the application or its application and uses.

[0017] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art can not be discussed in detail herein. However, where appropriate, such techniques, methods, and apparatus should be considered as being part of the specification.

[0018] In all of the examples shown and discussed herein, any specific values should be interpreted as merely exemplary, and not as a limitation. Thus, other examples of exemplary embodiments can have different values.

[0019] It should be noted that like reference numerals and letters refer to like items throughout the attached drawings, and thus once an item is defined in one drawing, it is not necessary that it be further discussed in subsequent drawings.

[0020] For clarity, the present application is described below based on the Transformer network structure. See Figure 1 As shown, the image super-resolution method provided based on the image pre-training strategy includes the following steps.

[0021] Step S110, a large-scale training dataset is constructed using a publicly available dataset, wherein each sample reflects the corresponding relationship between a low-resolution image and a high-resolution image.

[0022] First, a large-scale dataset is collected to construct low-resolution and high-resolution image data pairs. For example, in order to save the cost of collecting sample data, a publicly available large-scale dataset or other means can be used to download an existing dataset, and a low-resolution image and a high-resolution image data pair are obtained through a downsampling method (such as bicubic downsampling) to construct a large-scale training dataset.

[0023] Step S120, an image super-resolution model is constructed.

[0024] In one embodiment, the image super-resolution model is constructed based on the Transformer, which takes a low-resolution image as input and a high-resolution image (i.e., a super-resolution image) as output. The architecture of the model is one of the core improvements of the present application, which will be described in detail below.

[0025] Step S130, the image super-resolution model is pre-trained using the large-scale training dataset.

[0026] The pre-training process takes the minimization of a set loss function as the optimization goal, for example, an LI loss function can be used, and the optimization is performed by gradient descent until convergence. The optimized parameters of the model, such as weights and biases, can be obtained through the pre-training process. Pre-training refers to training part or all of the network using additional data before training the target network.

[0027] In step S140, a small sample training data set is constructed based on the target data source.

[0028] In this step, a certain number of images are collected based on the target data source (i.e., for a specific use case), and low-resolution and high-resolution data pairs are obtained through downsampling (or direct collection). In this way, a small sample data set suitable for the target scene can be obtained, and the data size is much smaller than the training data set constructed in step S110.

[0029] In step S150, the pre-trained image super-resolution model is fine-tuned using the small sample data set.

[0030] In order to enhance the adaptability or robustness of the image super-resolution model to the target scene, the pre-trained image super-resolution model is fine-tuned using the small sample data set. A small learning rate (e.g., 1 / 20 of the initial learning rate) can be used for a small number of iterations to obtain the final model.

[0031] In step S160, the low-resolution image to be reconstructed is input into the trained image super-resolution model to obtain a high-resolution result.

[0032] After the above pre-training and fine-tuning process, the trained model can be used to obtain the super-resolution image reconstruction result corresponding to the input image. For the designed image super-resolution model with stronger representation ability, pre-training can stimulate the greater representation potential of the model, and then fine-tuning with a small sample data set can further enhance the adaptability to a specific scene. In the existing image super-resolution field or other underlying image processing fields, there is no such pre-training strategy, because the existing models have limited representation ability, and even if the pre-training strategy is used, the representation accuracy of the model cannot be improved.

[0033] The structure of the image super-resolution model and the image reconstruction process will be described below.

[0034] As Figure 2As shown, the main structure of the image super-resolution model is to input the low-resolution picture (image) to be reconstructed into the network, sequentially pass through the shallow feature extraction module, the deep feature extraction module, and the high-resolution reconstruction module, and finally output the high-resolution picture. For the shallow feature extraction module, for example, a 3x3 convolution layer can be directly used to implement it. The obtained shallow features are input to the deep feature extraction module for complex mapping, and at the same time, a residual connection is passed to the deep feature extraction module, and then the shallow features and the deep features are added and reconstructed by the high-resolution reconstruction module. In an embodiment, the high-resolution reconstruction module can be composed of two to three convolution layers plus a pixel shuffle layer (Pixel Shuffle), and finally the high-resolution picture result is obtained.

[0035] The core design of the image super-resolution model is mainly reflected in the deep feature extraction module. For example, the deep feature extraction module includes N hybrid attention module groups (HAG). Each hybrid attention module group includes M hybrid attention sub-modules (HAB), an overlapping cross-scale attention module (OCAB), and a residual connection from the input to the output. The number N of hybrid attention module groups can be set to one or more, and it has been verified that the efficiency is optimal when N is 6, and the performance is saturated when N is 12. The number M of hybrid attention sub-modules can be set to one or more, for example, M is 6.

[0036] For each hybrid attention sub-module HAB, the feature first passes through a layer normalization operation, and then passes through a channel attention module (CAB) and a self-attention (WSA) module, respectively. The two attention modules respectively perform information reorganization and mapping on the input features from the spatial dimension and the channel dimension, and then add the obtained results together to contain a residual connection to input to the next part (i.e. the next hybrid attention sub-module or the overlapping cross-scale attention module), including a layer normalization layer and a multi-layer perceptron layer.

[0037] For the overlapping cross-scale attention module OCAB, similar to the hybrid attention sub-module HAB, the feature first passes through a layer normalization layer and then passes through an overlapping cross-scale attention module, and a residual connection is input to the next part.

[0038] The overlapping cross-scale attention module is as follows Figure 3As shown, common self-attention mechanisms use standard window partitioning methods during window computation, that is, partitioning X in a non-overlapping manner. Q ,X K ,X V This yields the query(Q), key(K), and value(V) for calculating the input features X. Figure 3 In the proposed implementation, the overlapping cross-attention module uses a standard window partitioning method to divide X. Q To obtain Q, X is then partitioned using an overlapping method. K and X V This yields K and V. K and V obtained in this way have a larger range of information than Q, thus enabling better integration of information across windows.

[0039] It should be noted that the self-attention mechanism is used to calculate the correlation between different parts (or different regions) of the input image. In existing self-attention calculations, when calculating self-attention for a certain image patch, the corresponding Q, K, and V matrices are calculated based on the internal features of that image patch. Therefore, when using a window-based approach to calculate self-attention, there is no window overlap. However, this invention proposes for the first time that adjacent sliding windows overlap during the calculation of matrices K and V, thereby integrating information from different windows and more accurately calculating the correlation between image patches. Furthermore, existing technologies typically set relatively small window sizes, such as 8 or less. This invention, with its overlapping window design, sets the window size to a larger size, such as 16. Verification has shown that this setting significantly improves the model's representational capability.

[0040] It should be understood that those skilled in the art can make appropriate changes or modifications to the above embodiments without departing from the spirit and scope of the invention. For example, by changing the training dataset and simply adjusting the network input and output sizes, the invention can be used for other image restoration or enhancement tasks, such as image deblurring, denoising, rain removal, defogging, snow removal, and reflection removal. Furthermore, the invention can also be used as a basic structure in multi-frame tasks such as stereo super-resolution (Stereo SR) and video super-resolution.

[0041] To further verify the effectiveness of this invention, simulation verification was conducted. Firstly, in the verification, a large window size was used to calculate self-attention. Existing techniques often use a window size of 8, or at most 12, to calculate self-attention mechanisms. This invention, by using a larger window size of 16, significantly improves the representational ability of the network model and achieves higher performance with a limited increase in computation. See Table 1 below.

[0042] Table 1: Impact of different window sizes on network performance

[0043]

[0044] As can be seen from Table 1, the method using a larger window size 16 has higher PSNR and SSIM than the prior art using 8 as the window size on 5 evaluation data sets (namely Set5, Set14, BSD100, Urban100 and manga109), and the improvement is significant.

[0045] In view of the problem that the prior art cannot dynamically calculate the inter-channel weight according to the global information of the features, the application integrates the channel attention mechanism into the Transformer module, proposes a hybrid attention module, and introducing the channel attention mechanism can dynamically utilize the global information of the intermediate features in the network to dynamically calculate the weight between different channels, thereby significantly improving the ability of the network to utilize the global information, as shown in Table 2 below.

[0046] The prior art can only focus on the information in the current window when calculating the self-attention, which is the root cause of the block effect. The application specifically proposes an overlapping cross-attention module, which can better integrate the information across windows and has obvious gain in improving the network performance, as shown in Table 2 below.

[0047] Table 2: Influence of OCAB and HAB on performance

[0048] Baseline w / OCAB w / CAB Ours PSNR 27.81 27.91 27.91 27.97 SSIM 0.8336 0.8352 0.8355 0.8366

[0049] As can be seen from Table 2, OCAB and HAB (i.e. the module of w / CAB) have obvious performance improvement (up to 0.1 dB), and the simultaneous addition of the two modules has more significant performance improvement.

[0050] In view of the problem that the prior art uses multi-task or multi-degradation level for pre-training, resulting in limited improvement of network performance, the application proposes a large-scale data pre-training strategy based on the same task, which greatly improves the performance of the network, as shown in Table 3, wherein the EDT strategy is the prior art, and the Our strategy refers to the embodiment of the application.

[0051] Table 3: Influence of different pre-training strategies on performance

[0052]

[0053] As can be seen from Table 3, the pre-training strategy of the application significantly improves the representation ability of the model whether for the pre-training model result or for the fine-tuning model result after pre-training, and achieves very significant performance gain.

[0054] In the experiment, it is observed that the present application has a significant performance gain in numerical indicators compared to other existing methods, and this gap is more obvious on standard experimental data sets, especially Urban100 and Manga109. Figure 4 As shown in the figure, wherein, Figure 4 (a) is a low-resolution image, Figure 4 (b) is a super-resolution result, Figure 4 (c) is a high-resolution original image, it can be seen that the super-resolution image obtained by using the present application can restore more clear texture details.

[0055] In summary, through a large number of observations of the result images of the existing methods and through theoretical and experimental analysis, it is found that the existing methods have very limited ability to utilize input information, and the ability to calculate self-attention across windows is insufficient, and the existing Transformer structure has a very large demand for data volume. The present application provides targeted technical solutions, including: using a large window self-attention mechanism to improve the range of calculating the attention matrix and expand the utilization range of the input information; integrating a channel attention module into the network architecture to realize the utilization ability of the network for global information; proposing overlapping cross-attention to further enhance the ability of the network to utilize a larger range of information and better utilize cross-window information to eliminate the influence of blocking effect as much as possible; in addition, a method of using the same task large-scale data pre-training is proposed to develop the potential of the Transformer structure as much as possible and obtain more excellent performance. It has been verified that the present application far exceeds the current most advanced image super-resolution method in both numerical indicators and visual effect comparison. And it has strong versatility, not only can be used for image super-resolution, but also the designed modules OCAB and HAB can be easily adapted to the model structure of other image restoration and enhancement tasks, and the pre-training strategy can also be used in other tasks. In addition, by simply modifying the input and output size of the model of the present application and the training data, it can be quickly used in other bottom-layer image processing tasks. In the application layer, the present application can be used for camera imaging processing and image post-processing, and has room for development in the fields of mobile phones, cameras, security monitoring, etc.

[0056] The present application can be a system, a method and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions loaded thereon for causing a processor to implement various aspects of the present application.

[0057] A computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0058] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0059] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0060] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0061] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or nonvolatile memory, or a suitable combination of the different types of computer readable storage media. The computer readable program instructions can also be downloaded to a computer, other programmable data processing apparatus, or other device from a computer readable storage medium or to an external computer or external storage device via a data signal that can be transmitted for example via a wired medium or a wireless medium such as the Internet or Wireless Application Protocol (WAP) signaling.

[0062] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0063] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0064] Embodiments of the application have been described above, and the description is intended to be illustrative of the embodiments of the application and not exhaustive. Numerous modifications and adaptations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The scope of the application is defined by the appended claims.

Claims

1. An image super-resolution method based on an image pre-training strategy, comprising the following steps: collecting a target image; inputting the target image into a trained image super-resolution model to output a super-resolution image, the image super-resolution model comprising, in sequence, a shallow feature extraction module, a deep feature extraction module and a high-resolution reconstruction module with reference to the input, wherein the shallow feature extraction module extracts shallow features for the input target image, the deep feature extraction module extracts multi-level intermediate features using a channel attention mechanism and a self-attention mechanism, and the high-resolution reconstruction module takes as input the output features from the deep feature extraction module and the output features from the shallow feature extraction module transmitted via a residual connection to obtain a high-resolution image relative to the input target image, wherein the deep feature extraction module comprises overlapping cross-attention modules that use a standard window division method to obtain queries and a window division method with overlap to obtain keys and values when calculating attention.

2. The method of claim 1, wherein, The deep feature extraction module comprises a plurality of mixed attention module groups, each mixed attention module group comprising a plurality of mixed attention sub-modules and an overlapping cross-attention module connected in sequence, the output of the overlapping cross-attention module being fused with the input of the first mixed attention sub-module via a residual connection, and the overlapping cross-attention module obtaining the correlation between different image blocks based on overlapping windows during self-attention calculation.

3. The method of claim 2, wherein, Each mixed attention sub-module comprises a first layer normalization module, a channel attention module, a self-attention module, a second layer normalization module and a multi-layer perceptron, wherein the channel attention module and the self-attention module are used to reorganize and map the output of the first layer normalization module from the spatial dimension and the channel dimension, the obtained results are added together to obtain a first result, and the first result is fused with the input of the first layer normalization module transmitted via a residual connection, the first result is further sequentially fused with the input of the second layer normalization module and the multi-layer perceptron to obtain a second result, and the second result is fused with the first result transmitted via a residual connection.

4. The method of claim 2, wherein, The overlapping cross-attention module comprises a third layer normalization module, an overlapping cross-attention sub-module, a fourth layer normalization module and a multi-layer perceptron, wherein the output of the overlapping cross-attention sub-module has a residual connection with the input of the third layer normalization module, and the input of the fourth layer normalization module has a residual connection with the output of the multi-layer perceptron.

5. The method of claim 1, wherein, The image super-resolution model is trained according to the following steps: constructing a first training dataset using a publicly disclosed dataset, each sample of the first training dataset reflecting a corresponding relationship between a low-resolution image and a high-resolution image; pre-training the image super-resolution model based on the first training dataset until a set loss function criterion is met; fine-tuning the pre-trained image super-resolution model using a second training dataset, wherein the second training dataset is collected for a target data source, and the number of samples in the second training dataset is less than that of the first training dataset.

6. The method of claim 1, wherein, For the overlapping cross-attention module, the window size is set to 16.

7. The method of claim 1, wherein, The image super-resolution model is constructed based on a transformer, the shallow feature extraction module includes one convolution layer, the high-resolution reconstruction module includes multiple convolution layers and one pixel shuffle layer, and the number of intermediate features extracted by the deep feature extraction module is set to six levels.

8. A computer readable storage medium having stored thereon a computer program, wherein, The computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

9. A computer device comprising a memory and a processor, having stored on the memory a computer program capable of running on the processor, characterized in that, The processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 7.

Images