A text-guided natural image super-resolution reconstruction method
Through text-guided natural image super-resolution reconstruction method, combined with text description and deep learning technology, low-frequency and high-frequency networks are used to restore image details, solving the loss of details and texture in large-scale super-resolution reconstruction in the prior art. Real problems achieve real and high-resolution image generation.
Patent Information
- Application Number
- CN202011263437.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-12
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-11-12
AI Technical Summary
Existing deep learning-based super-resolution reconstruction methods are difficult to recover real and realistic details in large-scale super-resolution reconstruction tasks, and low-resolution images may lose too much detail, resulting in the loss of important semantic information and cannot be accurate. Infer texture information and generate blurry or unreal image content.
A text-guided natural image super-resolution reconstruction method was designed. By combining text description, low-frequency networks and high-frequency networks gradually recovering image details. The low-frequency network uses text perception module to combine image features with text features to generate low-frequency images. Information, high-frequency network repair texture details, introduce semantic-based discriminator constraint network optimization, and training loss functions include semantic discriminator loss, graphic matching loss, and text-aware reconstruction loss.
Generate real, high-definition and semantic accurate high-resolution images, effectively use text information to restore image object features, generate diverse and beautiful and natural images, solve the problem of insufficient low-resolution images, and restore realistic high-frequency textures and details. .
Smart Images

Figure CN114494007B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of digital image intelligent processing, and specifically relates to an image restoration method, and more specifically, to a text-guided natural image super-resolution reconstruction method. Background Art
[0002] In recent years, with the application of deep learning technology in the field of computer vision, image super-resolution reconstruction has achieved good results. The restored high-resolution image is similar to the real high-resolution image both visually and in terms of objective numerical indicators. Therefore, deep learning technology has become a research hotspot in image super-resolution reconstruction.
[0003] Existing deep learning-based super-resolution reconstruction methods are only applied to a single image mode, estimating the missing pixel information based on the input low-resolution image. Lacking an understanding of high-level semantics, these methods have difficulty reconstructing realistic details. In the case of large super-resolution reconstruction multiples, the low-resolution image may lose too much detail, resulting in the loss of important semantic information and difficulty in recovering sufficient high-frequency information. When the input image is too small to contain enough information, traditional super-resolution reconstruction methods (EDSR[3], SPSR[4]) are unable to infer accurate texture information and produce blurry or unrealistic image content.
[0004] Although some super-resolution reconstruction methods (SFTGAN[5]) have incorporated semantic information, these methods only utilize semantic priors at the image level, such as semantic segmentation maps, face segmentation maps, etc. These semantic images often contain only limited semantic information and cannot provide sufficient assistance for the super-resolution reconstruction process.
[0005] In addition, due to the ill-posedness of the image super-resolution task, given a low-resolution image, there are many different high-resolution image solutions. However, for a low-resolution image, existing super-resolution methods can only reconstruct a single high-resolution image. Summary of the invention
[0006] In order to overcome the deficiencies of the prior art, the object of the present invention is to provide a natural image super-resolution reconstruction method that utilizes prior knowledge of text description to reconstruct realistic and semantically correct text guidance.
[0007] The text-guided natural image super-resolution reconstruction method provided by the present invention is mainly aimed at the problem that low-resolution images contain limited image information, which limits the restoration of high-resolution image content in large-scale super-resolution reconstruction tasks. The present invention assists in generating realistic, accurate and beautiful natural images by combining text descriptions.
[0008] The text-guided natural image super-resolution reconstruction method provided by the present invention comprises a design of an image super-resolution reconstruction network model, wherein the image super-resolution reconstruction network model comprises a low-frequency network and a high-frequency network (see Figure 1 As shown), for the n-fold super-resolution reconstruction network, the low-frequency network and the high-frequency network each include 2×log2 n convolutional layers, log2 n residual blocks and log2 n deconvolution layers; each residual block contains 2 convolution layers, 1 relu activation layer and 1 skip connection layer; the low-frequency network includes a text perception module (see Figure 2 As shown in Figure 2, the text perception module is used to combine image features with text features t; the specific steps of reconstruction are as follows:
[0009] (1) Use a text encoder to extract text features t from the text description.
[0010] First, the text description of the image is input into the text encoder to extract the text features t. The text encoder uses a recurrent neural network (Long Short-Term Memory Network LSTM[1]).
[0011] (2) Using the image super-resolution reconstruction model, low-frequency and high-resolution images are generated according to the guidance of text features t.
[0012] First, the low-resolution image I LR Input low-frequency network, which uses a convolutional layer and a residual block to extract the features F of the low-resolution image im Each residual block contains 2 convolutional layers, 1 relu activation layer and 1 skip connection layer.
[0013] The feature F of the low-resolution image im Together with the text feature t, it is input into the text perception module. The text perception module uses a 1×1 convolution layer to transform the text feature t of size 256×a into a vector j of size b×a, and reshapes the image feature of size H×W×b into a vector i of size HW×b. The image vector i and the text vector j are multiplied by matrix multiplication to obtain a vector k of size HW×a, which is then transformed into a feature map of size a×H×W as a text perception map M. attn At the same time, vector k is matrix-multiplied with vector j and transformed into a feature map of size H×W×b to obtain feature F attn .
[0014] The feature F attn With the original image feature F im After cascading, a convolutional layer and a residual block are input respectively, and then a deconvolution layer is used to amplify the features by 2 times to obtain Fim 2. Finally, a 3×3 convolutional layer is used to output a 2x super-resolved RGB image I lowSR ×2 .
[0015] F im 2 and the text feature t are input into the text perception module to obtain the feature F attn 2. Set feature F attn 2 and image feature F im After 2 cascades, a convolution layer and a residual block are input respectively, and then a deconvolution layer is used to further amplify the features by 2 times based on the features that have been amplified by 2 times to obtain a 4-fold amplified feature F. im 3. Finally, a 3×3 convolutional layer is used to output a 4x super-resolved RGB image I lowSR ×4 .
[0016] F im 3 and the text feature t are input into the text perception module to obtain the feature F attn 3. Set feature F attn 3 and image feature F im After 3 cascades, a convolution layer and a residual block are input respectively, and then a deconvolution layer is used to further amplify the features by 2 times based on the features that have been amplified by 4 times to obtain the features F amplified by 8 times. im 4. Finally, a 3×3 convolutional layer is used to output the 8-fold super-resolved RGB image I lowSR ×8 .
[0017] The low-frequency network outputs low-frequency super-resolution images of different multiples {I lowSR ×2 ,I lowSR ×4 ,I lowSR ×8}.
[0018] The reason for this is that restoring super-resolution images scale by scale through a pyramid-like network structure can reduce the difficulty of network training and reduce information loss. Text information usually only describes the general characteristics of an object (such as color, outline, background and other attributes), so compared with texture details, text information is more helpful in generating global image information containing low frequencies.
[0019] (3) Generate high-frequency and high-resolution images with accurate texture
[0020] The low-resolution image I LR And low-frequency super-resolution images of different scales generated by the low-frequency network {I lowSR ×2 ,I lowSR ×4 ,IlowSR ×8} Input the high-frequency network of the image super-resolution reconstruction model. The high-frequency network uses a convolutional layer and a residual block to extract the low-resolution image features F im , and then use a convolutional layer to extract the features of low-frequency super-resolution images of different scales {F lowSR ×2 ,F lowSR ×4 ,F lowSR ×8}; Use a deconvolution layer to transform F im Zoom in 2 times to get feature F SR ×2 , and compare it with F lowSR ×2 Cascade, using a 3×3 convolutional layer to output a 2x super-resolution high-resolution image I SR ×2 .
[0021] The feature F SR ×2 A convolutional layer, a residual block, and a deconvolution layer are used to obtain the feature F that is further magnified by 2 times based on the feature that has been magnified by 2 times. SR ×4 . SR ×4 With F lowSR ×4 Cascade, using a 3×3 convolutional layer to output a 4x super-resolution high-resolution image I SR ×4 .
[0022] The feature F SR ×4 A convolutional layer, a residual block, and a deconvolutional layer are used to obtain the feature F that is further amplified by 2 times based on the feature that has been amplified by 4 times. SR ×8 . SR ×8 With F lowSR ×8 Cascade, using a 3×3 convolutional layer to output 8 times higher resolution image I SR ×8 .
[0023] The high-frequency network finally generates multiple images with different magnifications {I SR ×2 ,I SR ×4 ,I SR ×8}.
[0024] The reason for this is that since the low-frequency network directly predicts image information from the text, some unrealistic textures will be generated. The high-frequency network combines the output of the low-frequency sub-network stage by stage to repair the errors in the low-frequency super-resolution reconstructed image and predict the high-resolution image based on the internal image pixels, which can generate more accurate high-frequency details and maintain the real image information.
[0025] (4) Furthermore, in the training of the image super-resolution reconstruction network model, a semantic-based discriminator is introduced to constrain the optimization of the image super-resolution reconstruction network model; the discriminator is a convolutional neural network, which is used to extract features from the image, and then generates a one-dimensional output by adding a fully connected layer and a sigmoid function to determine whether the extracted features belong to a specific category; the semantic-based discriminator is used as a binary classifier to distinguish between real images and fake images; the text feature t and the super-resolution reconstructed image are input into the semantic-based discriminator to obtain a score for judging whether the image is real or fake and whether the image conforms to the text description at the semantic level;
[0026] (5) Training and parameter optimization of the image super-resolution reconstruction network model, including:
[0027] The gradient of the image super-resolution reconstruction model parameters is calculated through the reconstruction loss between the reconstructed super-resolution image and the real high-resolution image; the gradient of the semantic-based discriminator parameters is calculated through the real high-resolution image and the generated high-resolution image, and the network parameters are updated through gradient backpropagation;
[0028] Assume that for each image x in the training set i , all correspond to the text description t i , then define the training set X as:
[0029] X={(x1,t1),(x2,t2),…,(x n ,t n )}.
[0030] The image super-resolution reconstruction model is constructed by i Downsample to get a low-resolution image y i , using the corresponding text description t i information to generate a super-resolution reconstructed image z i During training, the overall loss function of the network optimization update is defined as:
[0031] L=λ CGAN L CGAN +λ TIC L TIC +λ TAR L TAR , (1)
[0032] Among them, LCGAN is the semantic-based discriminator loss, defined as:
[0033]
[0034] z i ,r i ,t i Represent the reconstructed image, the real high-resolution image and the text features respectively. D represents the discriminator in the present invention. The super-resolution reconstructed image is input into the image encoder to extract the image features. The image encoder uses the convolutional neural network Inception-v3[2]. The text features and image features are used to calculate the image-text matching loss L TIC , defined as:
[0035] L TIC =-(∑ i log(P(TE(t i )|IE(z i )))+∑ i log(P(IE(z i )|TE(t i )))), (3)
[0036] TE and IE are text encoder and image encoder respectively.
[0037] L TAR is the text-aware reconstruction loss, i.e., the text-aware map M output by the text-aware module attn Calculate the weighted pixel-wise L2 distance loss:
[0038] L TAR =M attn |z i ,LowPass(r i )|, (4)
[0039] λ CGAN , TIC and λ TAR To balance the weights of different losses.
[0040] In the present invention, the text-guided image super-resolution model uses an end-to-end training method, and the image super-resolution reconstruction network model and the discriminator are trained alternately.
[0041] During testing, only the trained image super-resolution reconstruction network model is needed.
[0042] The beneficial effects of the present invention are as follows: the present invention can generate high-resolution images that are truly high-definition and semantically accurate, effectively solving the problem that low-resolution images are insufficient to provide sufficient information for predicting high-resolution image information in large-scale super-resolution reconstruction tasks. The present invention can effectively utilize the information describing the characteristics of image objects (such as color, contour, background and other attributes) contained in text information to generate accurate global image information. The present invention utilizes a high-frequency network to restore realistic high-frequency textures and details. At the same time, the present invention can also manipulate the color, shape, texture, background and other features of the super-resolution reconstructed image by controlling the input text description, achieving a diverse and beautiful and natural effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 This is a network framework diagram of the present invention.
[0044] Figure 2 This is a network framework diagram of the text perception module of the present invention.
[0045] Figure 3 This is the evaluation result of the 8x super-resolution reconstructed image of the present invention.
[0046] Figure 4 This is the 8x super-resolution reconstruction effect produced by using different text descriptions in the present invention. DETAILED DESCRIPTION
[0047] For a low-resolution image, you need to generate a high-resolution image. You can use Figure 1 The shown network performs text-guided super-resolution reconstruction.
[0048] The specific steps are:
[0049] (1) Extract text features t from the input text description;
[0050] First, the text description of the image is input into a text encoder to extract text features t.
[0051] (2) Generate low-frequency high-resolution images using text guidance lowSR
[0052] Combine the text feature t with the low-resolution image I LR Input into the low-frequency network together. Use a convolutional layer and a residual block to extract image features F im , the residual block contains 2 convolutional layers, 1 relu activation layer and 1 skip connection layer.
[0053] The feature F of the low-resolution image im Enter together with the text feature t as Figure 2In the text perception module shown, a 1×1 convolutional layer transforms the text feature t of size 256×a into a vector j of size b×a. The image feature of size H×W×b is reshaped into a vector i of size HW×b. The image vector i and the text vector j are multiplied by matrix multiplication to obtain a vector k of size HW×a, which is then transformed into a feature map of size a×H×W as a text perception map M. attn At the same time, vector k is matrix-multiplied with vector j and transformed into a feature map of size H×W×b to obtain the image feature F fused with text information. attn .
[0054] The feature F attn With the original image feature F im After cascading, a convolutional layer and a residual block are input respectively, and then a deconvolution layer is used to amplify the features by 2 times to obtain F im 2. Finally, a 3×3 convolutional layer is used to output a 2x super-resolved RGB image I lowSR ×2 . im 2 and the text feature t are input into the text perception module to obtain the output feature F attn 2. Set feature F attn 2 and image feature F im After 2 cascades, a convolution layer and a residual block are input respectively, and then a deconvolution layer is used to further amplify the features by 2 times based on the features that have been amplified by 2 times to obtain a 4-fold amplified feature F. im 3. Finally, a 3×3 convolutional layer is used to output a 4x super-resolved RGB image I lowSR ×4 . im 3 and the text feature t are input into the text perception module to obtain the output feature F attn 3. Set feature F attn 3 and image feature F im After 3 cascades, a convolution layer and a residual block are input respectively, and then a deconvolution layer is used to further amplify the features by 2 times based on the features that have been amplified by 4 times to obtain the features F amplified by 8 times. im 4. Finally, a 3×3 convolutional layer is used to output the 8-fold super-resolved RGB image I lowSR ×8 The low-frequency network outputs low-frequency super-resolution images of different multiples {I lowSR ×2 ,I lowSR ×4 ,I lowSR ×8}.
[0055] (3) Generate high-frequency and high-resolution images with accurate texture I SR
[0056] The low-resolution image I LR And low-frequency super-resolution images of different scales generated by the low-frequency network {I lowSR ×2 ,I lowSR ×4 ,I lowSR ×8} Input high-frequency network. First, a convolutional layer and a residual block extract low-resolution image features F im , and another convolutional layer extracts features of low-frequency super-resolution images of different scales {F lowSR ×2 ,F lowSR ×4 ,F lowSR ×8}. Use a deconvolution layer to convert F im Zoom in 2 times to get feature F SR ×2 , and compare it with F lowSR ×2 Cascade, using a 3×3 convolutional layer to output a 2x super-resolution high-resolution image I SR ×2 .
[0057] The feature F SR ×2 A convolutional layer, a residual block, and a deconvolution layer are used to obtain the feature F that is further magnified by 2 times based on the feature that has been magnified by 2 times. SR ×4 . SR ×4 With F lowSR ×4 Cascade, using a 3×3 convolutional layer to output a 4x super-resolution high-resolution image I SR ×4 .
[0058] The feature F SR ×4 A convolutional layer, a residual block, and a deconvolutional layer are used to obtain the feature F that is further amplified by 2 times based on the feature that has been amplified by 4 times. SR ×8 . SR ×8 With F lowSR ×8 Cascade, using a 3×3 convolutional layer to output 8 times higher resolution image I SR ×8 .
[0059] The high-frequency network finally generates multiple images with different magnifications {I SR ×2 ,ISR ×4 ,I SR ×8}.
[0060] (4) Update the parameters of the image super-resolution reconstruction model and discriminator by optimizing the loss function
[0061] During training, the loss functions used by the text-guided natural image super-resolution network are semantic-based discriminator loss, image-text matching loss, and text-aware reconstruction loss to train the image super-resolution reconstruction model to learn the text-guided super-resolution reconstruction task. 8855 images from the CUB dataset, 7,034 images from the Oxford-102 dataset, 162,770 images from the CelebA dataset, and 80,000 images from the COCO dataset were used for training, and each image has a corresponding text description as a reference. During testing, the network only needs to use the image super-resolution reconstruction model;
[0062] Figure 3 The results of image super-resolution reconstruction guided by the text of the present invention are shown in the first to fourth columns. The images reconstructed using bicubic interpolation, EDSR[3], ESRGAN[4], and SPSR[5] super-resolution reconstruction methods are shown in the first to fourth columns. The fifth column is the result of reconstruction by the present invention, and the sixth column is the real high-resolution image. It can be seen that the reconstructed images of the present invention are superior to the existing image super-resolution methods in terms of authenticity and aesthetics.
[0063] Figure 4 The different image super-resolution reconstruction effects produced by manipulating text descriptions in the present invention. The first and second columns are the input low-resolution image and the real high-resolution image, respectively. The third and fourth columns are the super-resolution reconstructed images obtained by using the text descriptions in (a) and (b) as references, respectively. It can be seen that the present invention can effectively extract the semantic features in the text information and use them to restore the corresponding image content, obtaining diverse and natural super-resolution reconstruction results.
[0064] References
[0065] [1] M.Schuster and KKPaliwal, "Bidirectional recurrent neural networks," in IEEE Transactions on Signal Processing, vol.45, no.11, pp.2673-2681, Nov.1997.
[0066] [2]Christian Szegedy,Vincent Vanhoucke,Sergey Ioffe,Jonathon Shlens,and Zbigniew Wojna.Rethinking the inception architecture for computervision.2016IEEE Conference on Computer Vision and Pattern Recognition.
[0067] [3]B.Lim,S.Son,H.Kim,S.Nah and K.M.Lee,"Enhanced Deep ResidualNetworks for Single Image Super-Resolution,"2017IEEE Conference on ComputerVision and Pattern Recognition Workshops(CVPRW),Honolulu,HI,2017,pp.1132-1140,doi:10.1109 / CVPRW.2017.151.
[0068] [4]B.Lim,S.Son,H.Kim,S.Nah and K.M.Lee,"Enhanced Deep ResidualNetworks for Single Image Super-Resolution,"2017IEEE Conference on ComputerVision and Pattern Recognition Workshops(CVPRW),Honolulu,HI,2017,pp.1132-1140,doi:10.1109 / CVPRW.2017.151.
[0069] [5]X.Wang,K.Yu,C.Dong and C.Change Loy,"Recovering Realistic Texturein Image Super-Resolution by Deep Spatial Feature Transform,"2018IEEE / CVFConference on Computer Vision and Pattern Recognition,Salt Lake City,UT,2018,pp.606-615。
Claims
1. A text-guided natural image super-resolution reconstruction method, It is characterized in that Design an image super-resolution reconstruction network model, which includes a low-frequency network and a high-frequency network. For the n-fold super-resolution reconstruction network, the low-frequency network and the high-frequency network each include 2×log 2 n convolutional layers, log 2 n residual blocks and log 2 n deconvolution layers; each residual block contains 2 convolution layers, 1 relu activation layer and 1 skip connection layer; the low-frequency network includes a text perception module to combine image features with text features t; the specific steps of reconstruction are as follows: (1) using a text encoder to extract text features t from the text description; the text encoder is a recurrent neural network; (2) Using the image super-resolution reconstruction model, low-frequency and high-resolution images are generated according to the guidance of text features t; the specific process is as follows: First, the low-resolution image I LR Input low-frequency network, which uses a convolutional layer and a residual block to extract the features F of the low-resolution image im ; The residual block includes 2 convolutional layers, 1 relu activation layer and 1 skip connection layer; The feature F of the low-resolution image im Together with the text feature t, it is input into the text perception module; the text perception module uses a 1×1 convolution layer to transform the text feature t of size 256×a into a vector j of size b×a, and reshapes the image feature of size H×W×b into a vector i of size HW×b; matrix multiplication is used to multiply the image vector i and the text vector j to obtain a vector k of size HW×a, and then transforms it into a feature map of size a×H×W as a text perception map M attn ; At the same time, vector k is matrix-multiplied with vector j and transformed into a feature map of size H×W×b to obtain feature F attn ; The feature F attn With the original image feature F im After cascading, a convolutional layer and a residual block are input respectively, and then a deconvolution layer is used to amplify the features by 2 times to obtain F im 2 Finally, a 3×3 convolutional layer is used to output a 2x super-resolved RGB image I lowSR ×2 ; F im 2 Input the text perception module with the text feature t to obtain the feature F attn 2 ; The feature F attn 2 With image feature F im 2 After cascading, a convolution layer and a residual block are input respectively, and then a deconvolution layer is used to further amplify the features by 2 times based on the features that have been amplified by 2 times to obtain a 4-fold amplified feature F. im 3 Finally, a 3×3 convolutional layer is used to output a 4x super-resolved RGB image I lowSR ×4 ; F im 3 Input the text perception module with the text feature t to obtain the feature F attn 3 ; The feature F attn 3 With image feature F im 3 After cascading, a convolution layer and a residual block are input respectively, and then a deconvolution layer is used to further amplify the features by 2 times based on the features that have been amplified by 4 times to obtain the features F amplified by 8 times. im 4 Finally, a 3×3 convolutional layer is used to output an 8-fold super-resolved RGB image I lowSR ×8 ; The low-frequency network outputs low-frequency super-resolution images of different multiples {I lowSR ×2 ,I lowSR ×4 ,I lowSR ×8 }; (3) Generate high-frequency and high-resolution images with accurate texture. The specific process is as follows: The low-resolution image I LR And low-frequency super-resolution images of different scales generated by the low-frequency network {I lowSR ×2 ,I lowSR ×4 ,I lowSR ×8 } Input the high-frequency network of the image super-resolution reconstruction model; the high-frequency network uses a convolutional layer and a residual block to extract the low-resolution image features F im , and then use a convolutional layer to extract the features of low-frequency super-resolution images of different scales {F lowSR ×2 ,F lowSR ×4 ,F lowSR ×8 }; Use a deconvolution layer to transform F im Zoom in 2 times to get feature F SR ×2 , and compare it with F lowSR ×2 Cascade, using a 3×3 convolutional layer to output a 2x super-resolution high-resolution image I SR ×2 ; The feature F SR ×2 A convolutional layer, a residual block, and a deconvolution layer are used to obtain the feature F that is further amplified by 2 times based on the feature that has been amplified by 2 times. SR ×4 ; F SR ×4 With F lowSR ×4 Cascade, using a 3×3 convolutional layer to output a 4x super-resolution high-resolution image I SR ×4 ; The feature F SR ×4 A convolutional layer, a residual block, and a deconvolutional layer are used to obtain the feature F that is further amplified by 2 times based on the feature that has been amplified by 4 times. SR ×8 ; F SR ×8 With F lowSR ×8 Cascade, using a 3×3 convolutional layer to output 8 times higher resolution image I SR ×8 ; The high-frequency network finally generates multiple images with different magnifications {I SR ×2 ,I SR ×4 ,I SR ×8 }.
2. The text-guided natural image super-resolution reconstruction method according to claim 1, It is characterized in that In the training of the image super-resolution reconstruction network model, a semantic-based discriminator is introduced to constrain the optimization of the image super-resolution reconstruction network model; the discriminator is a convolutional neural network, which is used to extract features from the image, and then generates a one-dimensional output by adding a fully connected layer and a sigmoid function to determine whether the extracted features belong to a specific category; the semantic-based discriminator is used as a binary classifier to distinguish between real images and fake images; the text feature t and the super-resolution reconstructed image are input into the semantic-based discriminator to obtain a score for judging whether the image is true or false and whether the image conforms to the text description at the semantic level.
3. The text-guided natural image super-resolution reconstruction method according to claim 2, It is characterized in that The training and parameter optimization of the image super-resolution reconstruction network model include: The gradient of the image super-resolution reconstruction model parameters is calculated through the reconstruction loss between the reconstructed super-resolution image and the real high-resolution image; the gradient of the semantic-based discriminator parameters is calculated through the real high-resolution image and the generated high-resolution image, and the network parameters are updated through gradient backpropagation; the specific process is: Assume that for each image x in the training set i , all correspond to the text description t i , define the training set X as: X={(x 1 ,t 1 ),(x 2 ,t 2 ),…,(x n ,t n )}; The image super-resolution reconstruction model is constructed by i Downsample to get a low-resolution image y i , using the corresponding text description t i information to generate a super-resolution reconstructed image z i ; During training, the overall loss function of the network optimization update is defined as: L=λ CGAN L CGAN +λ TIC L TIC +λ TAR L TAR , (1) Among them, L CGAN is the semantic-based discriminator loss, defined as: z i ,r i ,t i Represent the reconstructed image, the real high-resolution image and the text feature t respectively; D represents the discriminator; Use an image encoder to extract image features from the super-resolution reconstructed image; the image encoder is a convolutional neural network Inception-v3; use text features t and image features to calculate the image-text matching loss L TIC , defined as: L TIC =-(∑ i log(P(TE(t i )|IE(z i )))+∑ i log(P(IE(z i )|TE(t i )))), (3) TE and IE are text encoder and image encoder respectively; L TAR is the text-aware reconstruction loss, i.e., the text-aware map M output by the text-aware module attn Calculate the weighted pixel-wise L2 distance loss: L TAR =M attn |z i ,LowPass(r i )|, (4) λ CGAN , TIC and λ TAR To balance the weights of different losses; The text-guided image super-resolution model uses an end-to-end training method, and the image super-resolution reconstruction network model and the discriminator are trained alternately.
4. The text-guided natural image super-resolution reconstruction method according to claim 3, It is characterized in that Use the trained image super-resolution reconstruction network model for testing.
