An image super-resolution reconstruction method

Abstract

This invention discloses an image super-resolution reconstruction method, comprising: acquiring a training dataset and performing augmentation processing on the training dataset; constructing an image super-resolution reconstruction network model; training the constructed image super-resolution reconstruction network model using the augmented training dataset; and inputting the image to be reconstructed into the trained image super-resolution reconstruction network model to obtain the reconstructed super-resolution image. This invention can effectively improve the edge sharpness of the acquired image without changing the imaging system hardware.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to an image super-resolution reconstruction method. Background Technology

[0002] Image super-resolution reconstruction is a hot research area in computer vision, with wide applications in video surveillance, medical imaging, video perception, and many other fields. The concept involves reconstructing a low-resolution image into a high-resolution image using specific algorithms. In many other computer vision tasks, such as image segmentation and object detection, image super-resolution reconstruction can further enhance recognition capabilities and accuracy. With the development of deep learning technology in recent years, convolutional neural networks have shown significant advantages in extracting image features, leading to substantial progress in image super-resolution reconstruction. Many efficient algorithms have been proposed to solve the reconstruction problem of images at different scales. However, existing network models still suffer from problems such as large model parameter count, severe artifacts in the reconstructed image, and low local detail resolution. Therefore, research on image super-resolution reconstruction technology remains a challenging task. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention provides an image super-resolution reconstruction method capable of performing image super-resolution reconstruction.

[0004] The technical problem to be solved by the present invention is achieved through the following technical solution:

[0005] Firstly, an image super-resolution reconstruction method is provided, including:

[0006] Obtain the training dataset and perform augmentation processing on the training dataset;

[0007] Construct an image super-resolution reconstruction network model;

[0008] The constructed image super-resolution reconstruction network model was trained using the augmented training dataset;

[0009] The image to be reconstructed is input into the trained image super-resolution reconstruction network model f to obtain the reconstructed super-resolution image;

[0010] The image super-resolution reconstruction network model includes a shallow feature extraction module, a non-local contrast enhancement residual group module, an upsampling module, and a reconstruction module.

[0011] The shallow feature extraction module is used to extract shallow features from the augmented training dataset to obtain a shallow image feature map F0.

[0012] The nonlocal contrast enhancement residual group module is used to obtain the image depth feature output F based on the shallow image feature map F0. DF ;

[0013] The upsampling module is used to output F for image depth features. DF Upsampling is performed to obtain the upsampled feature map, denoted as F. ↑ ;

[0014] The reconstruction module is used to reconstruct the upsampled feature map F. ↑ Reconstruction is performed to obtain a reconstructed high-resolution image, denoted as x. SR .

[0015] In conjunction with the first aspect, the image super-resolution reconstruction network model further includes an adaptive target generation module, which generates adaptive targets to further train the trained image super-resolution reconstruction network model f to obtain the final image super-resolution reconstruction network model F.

[0016] The image to be reconstructed is input into the final image super-resolution reconstruction network model F to obtain the final high-resolution reconstructed image, denoted as F(x). LR ).

[0017] In conjunction with the first aspect, the nonlocal contrast-enhanced residual group module further includes two nonlocal contrast attention modules and a second-order attention shared source residual group module.

[0018] One of the nonlocal contrastive attention modules is used to obtain a nonlocal attention feature map based on the shallow image feature map F0. As shown in equations (1), (2), and (3):

[0019]

[0020]

[0021]

[0022] Where q is the magnification factor, Q, K, and V represent three different mappings on the shallow image feature map F0, and θ, δ, and g are the corresponding feature transformation functions. i and K j Let i be the pixel feature at position i after Q-mapping and j be the pixel feature at position j after K-mapping, respectively; φ is the unbiased approximation function; and D is the normalization term in the softmax function.

[0023] The second-order attention-sharing source residual group module is used to determine the nonlocal attention feature map. Extracting depth features F from the image G As shown in equation (4)

[0024]

[0025] Among them, the second-order attention-shared source residual group module is composed of several second-order attention-shared source residual modules connected in series, W SSC F represents the weights of the convolutional layer. g This is the output of the g-th second-order attention-shared source residual module;

[0026]

[0027] Among them, H g F represents the function F of the g-th second-order attention-shared source residual module. g-1 This is the input to the g-th second-order attention-shared source residual module;

[0028] Another non-local contrast attention module is used to adjust the depth features F of the image. G Obtain deeper image depth features F DF .

[0029] In conjunction with the first aspect, further, the L1 loss function L(Θ) during the training process of the image super-resolution reconstruction network model is:

[0030]

[0031] in, This represents the i-th pixel in the image predicted by the super-resolution reconstruction network model. Let represent the i-th pixel of the original high-resolution image, and N represent the total number of pixels in the image.

[0032] In conjunction with the first aspect, further, the process of generating an adaptive target and training the trained image super-resolution reconstruction network model f to obtain the final image super-resolution reconstruction network model F includes:

[0033] Generate an adaptive target image using equation (11)

[0034]

[0035] Where ATG represents the function corresponding to the ATG module, f(x) LR ), x SR These represent the model's current prediction output and the corresponding original high-resolution image, respectively, through adaptive target image. Further training of model f yields the final image super-resolution reconstruction network model F.

[0036]

[0037] Where l is the L2 loss function, and i represents the order of pixels in the image.

[0038] Secondly, an image super-resolution reconstruction apparatus is provided, including a processor and a storage medium;

[0039] The storage medium is used to store instructions;

[0040] The processor is configured to operate according to the instructions to perform the steps of the method described in any of the first aspects.

[0041] Thirdly, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the first aspects.

[0042] The main beneficial effects of this invention are as follows:

[0043] First, compared to traditional nonlocal attention mechanism network models and traditional nonlocal modules, the nonlocal contrastive attention mechanism introduced in this invention, through Gaussian random feature approximation, can aggregate important high-frequency information of an image with linear complexity input. This achieves superior performance with less computational overhead.

[0044] Secondly, the problem of severe artifacts in reconstructed images was effectively solved by using the Adaptive Target Training (ATG) strategy and RGB permute data augmentation. Attached Figure Description

[0045] Figure 1 This is a diagram of the image super-resolution reconstruction network model architecture in this invention;

[0046] Figure 2 This is a schematic diagram of the structure of the second-order attention-shared source residual module (LSRAG) in this invention;

[0047] Figure 3 This is a schematic diagram of the second-order channel attention module (SOCA) structure in this invention;

[0048] Figure 4 This is a schematic diagram of the Nonlocal Contrast Attention Module (ENLCA) structure in this invention;

[0049] Figure 5 This is a schematic diagram of the adaptive target generation module structure in this invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] To better understand this invention, the relevant technologies in the technical solution of this invention are described below.

[0052] Example 1

[0053] like Figure 1-5 As shown, this invention discloses an image super-resolution reconstruction method, comprising the following steps:

[0054] Step 1: Data Processing

[0055] To prepare the dataset for training the model, we will denote the original high-resolution image as x. HR The essence of super-resolution reconstruction is to add pixels and texture to a low-resolution image. Therefore, we first reconstruct the original high-resolution image x... HR The corresponding low-resolution image is obtained by performing downsampling with a factor of s, denoted as .

[0056] Specifically, to simulate the actual image degradation process, we take a standard downsampling factor s = {2, 3, 4, 8} for the original high-resolution image x. HR Perform bicubic downsampling at the corresponding scale to obtain a low-resolution image scaled accordingly. These images, forming a low-resolution image set, serve as the training dataset.

[0057] To achieve better training results, we need to process the images. Augmentation processing is performed, primarily using a combination of rotation, flipping, and RGB permute data augmentation methods to augment the image pairs in the dataset. The aim is to augment the low-resolution image x... LR Input into the constructed image super-resolution reconstruction network model to explore x LR Image and corresponding x HR Mapping between images. Specifically, for low-resolution images. The training set is randomly rotated by 90°, 180°, 270°, and horizontally flipped to enhance the training images. The RGB channels of the training images are randomly arranged using the RGB permute image augmentation method to augment and enhance the image channel characteristics. The augmented training samples are then... LRThe input is fed into the image super-resolution reconstruction network model for training.

[0058] Step 2: Model Building and Training

[0059] To reconstruct super-resolution images, we need to build an image super-resolution reconstruction network model.

[0060] Once the training set is ready, the next step is to train the constructed image super-resolution reconstruction network model. This image super-resolution reconstruction network model mainly includes: a shallow feature extraction module, a non-local contrast enhancement residual group module, an upsampling module, a reconstruction module, and an adaptive target generation module.

[0061] The shallow feature extraction module is used for shallow feature extraction and includes a shallow feature extraction layer that extracts the augmented training data image x. LR The input is fed into the shallow feature extraction module to obtain shallow image features, denoted as F0. Specifically, these features are extracted from the input training sample x through a convolutional layer. LR The shallow features are extracted as shown in the following formula:

[0062] F0 = H SL (x LR (1)

[0063] Among them, H SL This represents the convolution operation of the shallow feature extraction layer.

[0064] Next, the shallow image features F0 are input into the Non-Local Contrast Enhancement Residual Group (ENLRG) module to obtain the deep feature output of the image, denoted as F. DF The nonlocal contrast enhancement residual group module includes two nonlocal contrast attention modules (ENLCA) and one second-order attention shared source residual group module (SSRG). The second-order attention shared source residual group module (SSRG) is composed of several second-order attention shared source residual modules (LSRAG).

[0065] Specifically, such as Figure 1 As shown, the shallow image features F0 are input into the first non-local contrastive attention module (ENLCA) to obtain the non-local attention feature map. This nonlocal contrastive attention module multiplies the input information by an amplification factor q to increase the weight of relevant information, thereby giving the relevant information higher aggregation weights while maintaining the sparsity of the nonlocal module. It maps shallow image features F0 using Q, K, and V methods.

[0066]

[0067]

[0068]

[0069] θ, δ, and g are the corresponding characteristic transformation functions, and Q is the value of Q. i and K j Let i be the pixel feature at position i after Q-mapping and j be the pixel feature at position j after K-mapping, respectively, and let φ be the unbiased approximation function.

[0070] Furthermore, by setting r = 128 Gaussian random samples and concatenating these samples into a Gaussian random matrix F, the unbiased approximation of the Gaussian random feature map is made to be φ(Q). i ) T φ(K j ),in D is the normalization term in the softmax function.

[0071] D = di ag [φ(Q) T (φ(K) 1 N )](5)

[0072] Where diag represents calculating the diagonal matrix, and N represents the input size.

[0073] Next, the non-local attention feature map is input into a second-order attention shared source residual group (SSRG) module for feature extraction. The SSRG module consists of g = 20 (the number can vary) second-order attention shared source residual groups (LSRAGs) cascaded together. Each LSRAG module contains 10 simplified residual blocks. A second-order channel attention module (SOCA) is embedded at the end of each LSRAG module to obtain the correlation between network channels. (The specific structure of the LSRAG module is shown below.) Figure 3 As shown), different LSRAG modules are connected using a shared source residual skip (SSC) connection method (a residual network connection method). This ensures that the network bypasses low-frequency information of the image to the greatest extent, thus focusing on the training of high-frequency features. The output of the g-th LSRAG module can be expressed as:

[0074]

[0075] Among them, W SSC H represents the weights of the convolutional layer, initially set to 0, and then updated through iterations of the network layers. g F represents the function F of the g-th second-order attention-shared source residual module. g-1 The input to the g-th second-order attention-shared source residual module (LSRAG) (the input to the first layer is...) Non-local attention feature map After passing through several second-order attention-shared source residual modules (LSRAG), we can obtain the depth features F of the image. G .

[0076]

[0077] Next, we input the deep features obtained from the second-order attention shared source residual group module (SSRG) into the last non-local contrastive attention module (ENLCA) to obtain deeper deep features F. DF (The non-local contrast attention operation here is the same as before.)

[0078] Then we will F DF The input is fed into the upsampling module to obtain the result with x. HR Images of the same size are denoted as F. ↑ ;

[0079] Specifically, the depth feature map is upsampled to a suitable size by a factor of s through a subpixel convolutional layer, as shown in Equation (8).

[0080] F ↑ =H ↑ (F DF (8)

[0081] Among them, H ↑ and F ↑ These represent the upsampling function and the image features after upsampling, respectively.

[0082] Next, we will upsample the features F ↑ The input is fed into the reconstruction module to obtain the predicted output of the image super-resolution reconstruction network model, denoted as x. SR .

[0083] Specifically, the magnified image is mapped to a reconstructed image with three color channels through a convolutional layer, as shown in Equation (9).

[0084] f(x LR )=x SR =H R (F ↑ (9)

[0085] Among them, H R This represents the function that reconstructs the previous one.

[0086] Next, the image super-resolution reconstruction network model is further optimized using the loss function to obtain the pre-trained model f. Specifically, the image super-resolution reconstruction network model is optimized based on the L1 loss. During the iteration process, the loss function is optimized using the ADAM algorithm, and the L1 loss function is shown in equation (10).

[0087]

[0088] Where L(Θ) represents the L1 loss function, This represents the i-th pixel in the image predicted by the super-resolution reconstruction network model. Let represent the i-th pixel of the original high-resolution image, and N represent the total number of pixels in the image.

[0089] At this point, we can input the low-resolution image to be reconstructed into model f to obtain the reconstructed super-resolution image x. SR .

[0090] Image x obtained from model f SR The model has met the requirements, but it is not good enough. Therefore, we need to continue training the model f. We improve the model training by adding an Adaptive Target Generation (ATG) module after the reconstruction module.

[0091] Specifically, ATG is activated when 90% of the entire model's training batch is complete. During model training, we use ATG to evaluate the x-values ​​output by the network in the current iteration. SR Perform affine transformation to generate an adaptive target image.

[0092]

[0093] Where ATG represents the function corresponding to the ATG module, f(x) LR ), x SR These represent the model's current prediction output and the corresponding original high-resolution image, respectively, through adaptive target image. Further training of model f yields the final image super-resolution reconstruction network model F.

[0094]

[0095] Where l is the L2 loss function, and i represents the pixel index in the image. The new target of the ATG network is adaptively generated in each iteration based on the current output, so we do not need to prepare additional training data or spend a long time.

[0096] Step 3: Image Reconstruction

[0097] The image to be reconstructed x LR By inputting the trained final model F, the final reconstructed super-resolution image F(x) can be obtained. LR ).

[0098] Example 2

[0099] The present invention also provides an image super-resolution reconstruction apparatus, including a processor and a storage medium;

[0100] The storage medium is used to store instructions;

[0101] The processor is configured to operate according to the instructions to execute the steps of the image super-resolution reconstruction method.

[0102] Example 3

[0103] A storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the steps of an image super-resolution reconstruction method.

[0104] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0106] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

Claims

1. An image super-resolution reconstruction method, characterized in that, include: Obtain the training dataset and perform augmentation processing on the training dataset; Construct an image super-resolution reconstruction network model; The constructed image super-resolution reconstruction network model was trained using the augmented training dataset; The image to be reconstructed is input into the trained image super-resolution reconstruction network model. Obtain the reconstructed super-resolution image; The image super-resolution reconstruction network model includes a shallow feature extraction module, a non-local contrast enhancement residual group module, an upsampling module, and a reconstruction module. The shallow feature extraction module is used to perform shallow feature extraction on the augmented training dataset to obtain shallow image feature maps. ; The nonlocal contrast enhancement residual group module is used to determine the feature map of the shallow image. Obtain image depth feature output ; The upsampling module is used to output image depth features. Upsampling is performed to obtain an upsampled feature map, denoted as... ; The reconstruction module is used to reconstruct the upsampled feature map. Reconstruction is performed to obtain a reconstructed high-resolution image, denoted as... ; The nonlocal contrast-enhanced residual group module includes two nonlocal contrast attention modules and one second-order attention shared source residual group module. One of the non-local contrast attention modules is used to adjust the feature maps of the shallow image. Obtain nonlocal attention feature map As shown in equations (1), (2), and (3): ，（1） （2） （3） in, As the amplification factor, , and These represent the feature maps of shallow images. Three different mappings on, , and These are the corresponding feature transformation functions. and The feature maps are processed respectively Mapped location and the process Mapped location pixel features, It is an unbiased approximation function. This is the normalization term in the softmax function; The second-order attention-sharing source residual group module is used to determine the nonlocal attention feature map. Extracting depth features from an image As shown in equation (4) （4） The second-order attention-shared source residual group module is composed of several second-order attention-shared source residual modules connected in series. Indicates the weights of the convolutional layer. For the first The output of a second-order attention-shared source residual module; （5） in, Indicates the first Functions of a second-order attention-shared source residual module, For the first Input to a second-order attention-shared source residual module; Another non-local contrast attention module is used to consider the depth features of the image. Obtain deeper image depth features .

2. The image super-resolution reconstruction method according to claim 1, characterized in that, The image super-resolution reconstruction network model also includes an adaptive target generation module, used to generate adaptive target pairs for the trained image super-resolution reconstruction network model. Continue training to obtain the final image super-resolution reconstruction network model. ; The image to be reconstructed is input into the final image super-resolution reconstruction network model. The final high-resolution reconstructed image is obtained, denoted as... .

3. The image super-resolution reconstruction method according to claim 1, characterized in that, The training process of the image super-resolution reconstruction network model loss function for: （6） in, The image predicted by the super-resolution reconstruction network model is represented by the first... 1 pixel, The first part represents the original high-resolution image. 1 pixel, This represents the total number of pixels in the image.

4. The image super-resolution reconstruction method according to claim 2, characterized in that, The adaptive target generation method is applied to the trained image super-resolution reconstruction network model. Continue training to obtain the final image super-resolution reconstruction network model. include: Generate an adaptive target image using equation (7) ; （7） in, This indicates the function corresponding to the ATG module. , These represent the model's current prediction output and the corresponding original high-resolution image, respectively, through adaptive target image. Further training of the model The final image super-resolution reconstruction network model is obtained. ; （8） in, for loss function Indicates the order of pixels in an image.

5. An image super-resolution reconstruction device, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 4.

6. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images