Image Super-Resolution Model Training Method, Device, Storage Medium and Equipment

Abstract

A method, device, storage medium and equipment for training an image super-resolution model. The method includes: obtaining a super-resolution data set and training a super-resolution backbone network using the super-resolution data set; obtaining a target super-resolution image currently output by the super-resolution backbone network during training, and calculating the loss between the target super-resolution image and a super-resolution template image to obtain a first loss term; respectively inputting the target super-resolution image and the super-resolution template image into a pre-trained semantic segmentation network to output corresponding target segmentation images and segmentation template images; calculating the loss between the target segmentation image and the segmentation template image to obtain a second loss term; and iteratively training the super-resolution backbone network according to the first loss term and the second loss term. By proposing a super-resolution training method based on semantic segmentation constraints, the present invention significantly improves the semantic segmentation index of the super-resolution image while ensuring comparable super-resolution indexes.

Description

Technical Field

[0001] The present invention belongs to the technical field of image super-resolution model training, and particularly relates to an image super-resolution model training method, device, storage medium and equipment. Background Art

[0002] More than 80% of the various information received by people from the outside world is obtained through vision. Vision is the most important perception for humans to obtain external information. Enabling a computer to "understand" human vision and endowing the computer with the ability to process human visual perception is also one of the most convenient ways for humans and computers to work together.

[0003] Image super-resolution reconstruction (hereinafter referred to as super-resolution) refers to a technology that analyzes digital image signals and uses software algorithms to reconstruct and restore higher-resolution images or videos from one or multiple frames of images without improving optical imaging devices. At present, image super-resolution technology has been widely used in fields such as medical imaging, surveillance and security, remote sensing measurement and control, and audio-visual entertainment.

[0004] However, current mainstream image super-resolution only focuses on the visual comfort of images for the human eye and does not fully consider the ability of super-resolved images to be understood by computers. The most intuitive value of super-resolved images in subsequent computer vision tasks lies in the contribution they can make to improving the accuracy of subsequent semantic segmentation. Therefore, the performance of image segmentation based on super-resolved images is currently not ideal. Summary of the Invention

[0005] Embodiments of the present invention provide an image super-resolution model training method, device, storage medium and equipment, aiming to solve the technical problem that the performance of image segmentation based on super-resolved images is currently not ideal.

[0006] An image super-resolution model training method according to an embodiment of the present invention includes:

[0007] Obtain a super-resolution data set and use the super-resolution data set to train a super-resolution backbone network;

[0008] Obtain a target super-resolution image currently output by the super-resolution backbone network during training, and calculate the loss between the target super-resolution image and a super-resolution template image to obtain a first loss term;

[0009] Input the target super-resolution image and the super-resolution template image into a pre-trained semantic segmentation network respectively. The target super-resolution image generates a target segmentation image through the semantic segmentation network, and the super-resolution template image generates a segmentation template image through the semantic segmentation network;

[0010] Calculate the loss between the target segmentation image and the segmentation template image to obtain a second loss term;

[0011] Determine the current total loss function of the super-resolution backbone network according to the first loss term and the second loss term, and perform iterative training on the super-resolution backbone network according to the current total loss function to train an image super-resolution model.

[0012] Preferably, after the step of obtaining the target super-resolution image currently output by the super-resolution backbone network, it further includes:

[0013] Input the target super-resolution image and the super-resolution template image into a pre-trained VGG network to output the high-dimensional features of the target super-resolution image and the super-resolution template image;

[0014] Calculate the difference in data distribution between the high-dimensional features of the target super-resolution image and the super-resolution template image to obtain a third loss term;

[0015] Wherein, the current total loss function is further determined according to the third loss term.

[0016] Preferably, after the step of inputting the target super-resolution image and the super-resolution template image into a pre-trained VGG network to output the high-dimensional features of the target super-resolution image and the super-resolution template image, it further includes:

[0017] Calculate the distribution difference between the target super-resolution image and the super-resolution template image after making Gram matrices of their high-dimensional features to obtain a fourth loss term;

[0018] Wherein, the current total loss function is further determined according to the fourth loss term.

[0019] Preferably, the current total loss function satisfies the following conditional formula:

[0020] L total = α × L vis + β × L feat + γ × L sem + δ × L text

[0021] In the formula, L total represents the current total loss function, L vis represents the first loss term, L sem represents the second loss term, L feat represents the third loss term, L text represents the fourth loss term, and α, β, γ, and δ respectively represent L vis , L feat , L sem and Ltext weight.

[0022] Preferably, α is 1, β is 10 -3 -10 -1 between, γ is 10 -2 -1.0, δ is 10 -6 -10 -4 between.

[0023] Preferably, the semantic segmentation network is a bypass of the super-resolution backbone network.

[0024] The embodiment of the present invention also provides an image super-resolution model training device, which includes:

[0025] An image training module, configured to obtain a super-resolution data set and train a super-resolution backbone network using the super-resolution data set;

[0026] A first loss calculation module, configured to obtain a target super-resolution image output by the super-resolution backbone network during current training, and calculate the loss between the target super-resolution image and a super-resolution template image to obtain a first loss term;

[0027] An image input module, configured to input the target super-resolution image and the super-resolution template image into a pre-trained semantic segmentation network respectively. The target super-resolution image generates a target segmentation image through the semantic segmentation network, and the super-resolution template image generates a segmentation template image through the semantic segmentation network;

[0028] A second loss calculation module, configured to calculate the loss between the target segmentation image and the segmentation template image to obtain a second loss term;

[0029] The image training module is further configured to determine a current total loss function of the super-resolution backbone network according to the first loss term and the second loss term, and perform iterative training on the super-resolution backbone network according to the current total loss function to train an image super-resolution model.

[0030] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the image super-resolution model training method as described above.

[0031] The embodiment of the present invention also provides an image super-resolution model training device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the image super-resolution model training method as described above.

[0032] The beneficial effects achieved by the present invention are as follows: By proposing an image super-resolution model training method based on semantic segmentation constraints, taking the semantic segmentation loss as a constraint term for super-resolution model training, thus considering the super-resolution problem and the segmentation problem as a whole. The trained super-resolution model can improve the accuracy of semantic segmentation of the super-resolution result while ensuring comparable super-resolution metrics. Without adding training parameters, it significantly improves the semantic segmentation metrics of the super-resolution image, enabling ideal performance for image segmentation based on the super-resolution image. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flowchart of the image super-resolution model training method in Embodiment 1 of the present invention;

[0034] Figures 2 - 3 is a flowchart of the image super-resolution model training method in Embodiment 2 of the present invention;

[0035] Figure 4 is a structural block diagram of the image super-resolution model training device in Embodiment 3 of the present invention;

[0036] Figure 5 is a structural block diagram of the image super-resolution model training device in Embodiment 4 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0038] Example 1

[0039] Please refer to Figure 1 , which shows the image super-resolution model training method in Embodiment 1 of the present invention. The method specifically includes steps S01 - step S05.

[0040] Step S01, obtain a super-resolution data set, and use the super-resolution data set to train a super-resolution backbone network.

[0041] Among them, the super-resolution backbone network can be, but is not limited to, any one of the currently four mainstream super-resolution backbone networks SRResNet, EDSR, CARN, and RCAN.

[0042] Step S02, obtain the target super-resolution image currently output by the super-resolution backbone network, and calculate the loss between the target super-resolution image and the super-resolution template image to obtain a first loss term.

[0043] Among them, the first loss term is used to measure the pixel-level difference between the target super-resolution image and the super-resolution template image. The first loss term usually can choose L1 loss (absolute loss function) or L2 loss (square loss function).

[0044] Step S03: Input the target super-resolution image and the super-resolution template image into a pre-trained semantic segmentation network respectively. The target super-resolution image passes through the semantic segmentation network to generate a target segmentation image, and the super-resolution template image passes through the semantic segmentation network to generate a segmentation template image.

[0045] Among them, the semantic segmentation network can be, but is not limited to, any one of networks such as DeepLabV3, U-net, SegNet, PSPNet, etc.

[0046] Step S04: Calculate the loss between the target segmentation image and the segmentation template image to obtain a second loss term.

[0047] Among them, the second loss term Lsem, also known as the semantic loss term, is used to measure the difference between the target super-resolution image and the super-resolution template image in the segmentation images after passing through DeepLabV3. Specifically, the difference between the two can be characterized by calculating the cross-entropy quantization distance of the two images.

[0048] Step S05: Determine the current total loss function of the super-resolution backbone network according to the first loss term and the second loss term, and perform iterative training on the super-resolution backbone network according to the current total loss function to train an image super-resolution model.

[0049] In some optional embodiments of this embodiment, the current total loss function can be calculated based on the weighted sum of the first loss term and the second loss term. During the training process, by continuously calculating the current total loss function of the network and performing iterative training on the super-resolution backbone network according to the current total loss function, the global optimum is found to complete the model training. Specifically, the small batch stochastic gradient descent optimization iteration method can be adopted for this loss function, and the iteration process is as follows: a total of 900 epochs (generations), and all training data sets are traversed in each epoch. Each time, 2 training pictures are read in as a batch. One iteration trains a batch of samples. The loss on the training set is calculated once in each iteration, and the gradients backpropagated are calculated using this loss to update the training parameters of the network. The PSNR, SSIM, PA, and mIoU performance metrics are calculated once on the test set after each epoch, and the network model of this training with the optimal value of the average of the performance metrics of these 900 epochs is saved to obtain the finally trained model.

[0050] In summary, the image super-resolution model training method in this embodiment proposes a method for training an image super-resolution model based on semantic segmentation constraints, and uses the semantic segmentation loss as a constraint term for super-resolution model training. Thus, the super-resolution problem and the segmentation problem are considered as a whole. The trained super-resolution model can stably and significantly improve semantic segmentation metrics such as mIoU (Mean Intersection over Union), FWIoU (Frequency Weighted Intersection over Union), PA (Pixel Accuracy), and MPA (Mean Pixel Accuracy) while ensuring that metrics such as PSNR (Peak Signal noise ratio) and SSIM (Structural Similarity) are comparable to the results trained by conventional loss functions, enabling ideal performance for image segmentation based on super-resolution images.

[0051] Example 2

[0052] Please refer to Figure 2 , which shows the image super-resolution model training method in Embodiment 2 of the present invention. The method specifically includes steps S11 - step S18.

[0053] Step S11: Obtain a super-resolution dataset and use the super-resolution dataset to train a super-resolution backbone network.

[0054] Step S12: Obtain the target super-resolution image currently output by the super-resolution backbone network during training, and calculate the loss between the target super-resolution image and the super-resolution template image to obtain the first loss term L vis .

[0055] Among them, the first loss term L vis is used to measure the pixel-level difference between the target super-resolution image and the super-resolution template image. The first loss term usually can choose L1 loss (absolute loss function) or L2 loss (square loss function). Currently, L1 loss is usually used as the per-pixel loss term in super-resolution tasks. Its definition is:

[0056]

[0057] Among them, Ⅰ is the super-resolution template image, is the SR image, and H, W, and C are the height, width, and number of channels of the image respectively. L vis measures the average difference of the pixel points at (i, j, k) between the SR and the super-resolution template image.

[0058] Step S13: Input the target super-resolution image and the super-resolution template image into a pre-trained semantic segmentation network respectively. The target super-resolution image passes through the semantic segmentation network to generate a target segmentation image, and the super-resolution template image passes through the semantic segmentation network to generate a segmentation template image.

[0059] It should be noted that before training the super-resolution backbone network, the semantic segmentation network needs to be trained first. For the training stage of the semantic segmentation network, a segmentation network needs to be trained with a segmentation dataset. In theory, any meaningful semantic segmentation neural network can be adopted. Without loss of generality, in this embodiment, the stable DeepLabV3 is selected as the semantic segmentation network for training. Through the training of this process, semantic segmentation prior knowledge is learned from the segmentation dataset. In other embodiments, DeepLabV3 can also be replaced by networks such as U-net, SegNet, PSPNet, etc.

[0060] Subsequently, during the training of the super-resolution backbone network, the super-resolution dataset is used to train the super-resolution backbone network. During this process, the previously pre-trained semantic segmentation network needs to be added to the overall model as a bypass of the super-resolution backbone network to calculate the semantic segmentation loss term for the result generated by the randomly initialized super-resolution backbone network. Conventional super-resolution algorithms do not have such a segmentation bypass, and the loss function only has the L1 loss term. Their learning parameters mainly converge in the direction of minimizing the per-pixel loss from random initialization, which will inevitably lead to the learned model only satisfying the HVS (Human Visual System) and ignoring the perception of computer vision. It should be noted that during the training stage of the super-resolution backbone network, the parameters of DeepLabV3 do not participate in the learning. Since it comes from the previous segmentation task, it can provide a very intuitive semantic prior for the training of the super-resolution network, providing additional system information compared to the super-resolution backbone network without semantic segmentation ability.

[0061] Step S14: Calculate the loss between the target segmentation image and the segmentation template image to obtain the second loss term L sem 。

[0062] Among them, the second loss term Lsem, also known as the semantic loss term, is used to measure the cross-entropy quantization distance (representing the difference) between the segmentation images of the target super-resolution image and the super-resolution template image after passing through DeepLabV3. Its definition is as follows:

[0063]

[0064] Among them, M is the total number of segmentation categories, γ is the output result of DeepLabV3, which is a feature vector with height, width, and number of channels being H, W, and M respectively. is a one-hot vector of length M at the pixel point at position (h, w). The elements on the vector only take two values, 0 and 1. If the pixel point category at the coordinate (h, w) of the feature vector generated after the target super-resolution image passes through DeepLab is the same as the pixel point category at the pixel point (h, w) of the super-resolution template image after passing through DeepLab, it takes 1, otherwise it is 0. represents the probability that the pixel point at the coordinate point (h, w) belongs to the category c after the SR image passes through the feature mapping of DeepLab and undergoes softmax calculation.

[0065]

[0066] Step S15: Input the target super-resolution image and the super-resolution template image into a pre-trained VGG network to output the high-dimensional features of the target super-resolution image and the super-resolution template image.

[0067] In addition, it is found in the training research that adding a perceptual loss term by introducing the VGG network can better ensure the PSNR and has an improvement effect on the semantic segmentation index.

[0068] Step S16: Calculate the difference in data distribution in the high-dimensional features of the target super-resolution image and the super-resolution template image to obtain the third loss term L feat .

[0069] Among them, the third loss term L feat is also called the perceptual loss, which measures the difference between the two images in the high-level global information by calculating the Euclidean distance of the VGG high-dimensional features of the target super-resolution image and the super-resolution template image. Its definition is:

[0070]

[0071] Among them represents the output feature vector of the super-resolution template image at the j-th layer of the VGG network.

[0072] Step S17: Calculate the distribution difference between the two after making the Gram matrix of the high-dimensional features of the target super-resolution image and the super-resolution template image to obtain the fourth loss term L text .

[0073] Among them, the fourth loss term Ltext is also called the texture loss. The texture loss further adds the Gram matrix in the high-dimensional vector space defined by the perceptual loss to extract texture-related features. During the training process, the network learns to generate a target super-resolution image with the same local texture as the super-resolution template image. The texture loss is defined as:

[0074]

[0075] Among them, the Gram matrix

[0076] Step S18: Calculate the current total loss function of the super-resolution backbone network according to the first loss term, the second loss term, the third loss term, and the fourth loss term, and perform iterative training on the super-resolution backbone network according to the current total loss function to train an image super-resolution model.

[0077] Among them, the current total loss function satisfies the following conditional formula:

[0078] L total = α × L vis + β × L feat + γ × L sem + δ × L text

[0079] In the formula, L total represents the current total loss function, L vis represents the first loss term, L sem represents the second loss term, L feat represents the third loss term, L text represents the fourth loss term, and α, β, γ, and δ respectively represent the weights of L vis , L feat , L sem and L text . In some specific cases of this embodiment, α is taken as 1, β is taken as 10 -3 -10 -1 between, γ is taken as 10 -2 -1.0 between, δ is taken as 10 -6 -10 -4 between.

[0080] That is, the total loss is the weighted sum of the above losses. By setting the visual loss coefficient α, the perceptual loss coefficient β, the semantic loss coefficient γ, and the texture loss coefficient δ to regulate the weights of the four, in the experiment, the weights of β and γ can be set to 0 in the way of controlling variables to observe their effects on the super-resolution result and the segmentation result.

[0081] In addition, after the model is trained, in the inference stage (i.e., actually using the trained model to perform super-resolution on actual images), the segmentation network and the VGG network are no longer needed. The input original image can obtain a super-resolution result that can improve the segmentation performance through the super-resolution backbone network. This improvement is obtained without adding any learnable parameters. The processing speed of the network is the same as that of the super-resolution network trained conventionally, but it has a higher segmentation accuracy. Moreover, since the processing logic of the segmentation constraint during training is independent of the processing logic of the super-resolution backbone network, these two parts can be designed independently.

[0082] Specifically, please refer to Figure 3 , which shows the model training flowchart in this embodiment. During training, in the forward propagation of this iteration, the super-resolution backbone network (SR backbone) generates the current target super-resolution image (hereinafter referred to as SR). This SR result will be used to calculate the loss function terms in three paths. The first path is to calculate the L1 loss between SR and the super-resolution template image (hereinafter referred to as HR), obtaining the L vis loss term. The second path inputs SR into the DeepLabV3 network to obtain the segmentation map SR_SEG of SR (i.e., the target segmentation image). At the same time, HR is also input into DeepLabV3 and processed to obtain HR_SEG (the segmentation template image). This HR_SEG is used as the training label of SR_SEG, and they are together used to calculate the cross-entropy. Here is a design processing detail. The result output by the DeepLabV3 network is a tensor in the form of [N, M, H, W], where N is the batch size, M is the total number of segmentation categories, H is the height of the image read by this network, and W is the width. Perform the torch.max() operation on the output result of DeepLabV3 for HR in the dimension where M is located to record the index of the maximum value. This index will fall within the range of [0, M - 1], and the value represents the labels of each category. The forward(input, target) method of the CrossEntropyLoss() class in Pytorch contains two positional parameters input and target. The feature vector of SR after passing through DeepLabV3 is passed in as input, and HR_SEG is passed in as target. Inside CrossEntropyLoss, the input will be mapped to the probability space through the softmax operation, and at the same time, the category labels of the target will be processed by one-hot encoding, and then the cross-entropy is calculated. In this way, the difference between the probability distribution of SR in the DeepLabV3 feature space and the data distribution using HR_SEG as the segmentation label can be measured, thereby obtaining L semLoss term. The third path inputs SR and HR into the VGG network respectively to obtain the high-dimensional features of SR and HR, and measures the difference in the data distribution of SR and HR in the VGG feature space by comparing their Euclidean distances, thereby obtaining L feat Loss term.

[0083] In addition, in some experiments, a fourth path loss term is also set. The fourth path loss term is an upgrade of the third path loss term. It also needs to input SR and HR into the VGG feature space respectively. However, the difference is that it does not directly calculate the Euclidean distance of this vector, but calculates the distribution difference between the two after making the Gram matrix of the two VGG feature vectors, and obtains L text . The Gram matrix is a matrix composed of the inner products of any k vectors in the n-dimensional Euclidean space in pairs, and it is considered to be able to measure the characteristics of the vector itself and the relationship between each pair of vectors in each dimension. The Gram matrix is widely used in the style transfer task and is considered to have a good ability to describe the hidden connection between the shallow feature vectors for extracting texture detail features and the deep feature vectors for extracting contours and sizes. It also has certain applications in the super-resolution vision task. However, the experimental results show that this loss term has no obvious enhancement effect on both the human eye visual perception index and the semantic segmentation index of computer vision. Therefore, this term is discarded in the finally selected L total This term is discarded.

[0084] Regarding the selection of the training dataset, since the algorithm is divided into two stages, each stage corresponds to different models and visual tasks, so the dataset is also divided into two sets, namely the super-resolution dataset and the segmentation dataset. Since the training of the segmentation network needs to provide semantic priors for the super-resolution network, there needs to be a certain correlation between the objects of segmentation and super-resolution. To facilitate the verification of the effectiveness of our theory, segmentation and super-resolution are carried out for outdoor scenes. SFTGAN created a dataset OutdoorSeg for outdoor scene segmentation, which contains 9900 images, of which 8447 images are from the ADE dataset, 899 mountain images are from Flickr, and 554 animal images are from the COCO dataset. OutdoorSeg also contains the corresponding manually annotated segmentation labels, covering 7 categories, namely sky, water area, building, grassland, vegetation, animal, mountain. The image parts in each segmentation image that do not belong to the above categories are uniformly marked as other categories. At the same time, they also provided a test dataset OutdoorSceneTest300 (OST300) containing 300 outdoor scenes and their segmentation images, which can be used for super-resolution and segmentation tasks. We use OutdoorSeg for training in the training stage of the DeepLabV3 network and use OST300 for testing.

[0085] For the training of the super-resolution network, we separately selected 457 and 39 high-resolution images related to the outdoors from DIV2K as the training set and validation set of the HR samples of the super-resolution network, and the corresponding 4×bicubic interpolated downsampled images as the LR samples; the HR in the super-resolution test set comes from the aforementioned OST300, and the same downsampling operation is performed on OST300 to obtain the LR test samples. In this way, in addition to the HR and the manually annotated segmentation map HR_Seg corresponding to the HR in our OST300 dataset, there is also a corresponding low-resolution dataset LR, which meets the requirements of segmentation and super-resolution and can be used to test the gap between the results obtained by SR after passing through the segmentation network and the manually annotated labels.

[0086] To verify the effectiveness of the semantic loss term and its compatibility with different network structures, tests were actually carried out on four mainstream super-resolution backbone networks (SRResNet, EDSR, CARN, RCAN), which are often directly used or indirectly form part of complex models in super-resolution tasks. Specifically, experiments were conducted under different loss term weight coefficients, and the semantic segmentation improvement ability of the super-resolution results was measured using the PA, MPA, FWIoU, and mIoU metrics, and the human eye visual comfort of the super-resolution results was measured using the PSNR and SSIM metrics. Since the test set contains manually annotated segmentation labels, we will measure the segmentation metrics of the segmentation map of SR with the segmentation map of HR and the manually annotated segmentation map (ground truth) respectively. In the experiment, we set the coefficient α of L vis to 1.0, and we set several different orders of magnitude of base weight for each of the coefficients β of L feat , the coefficient γ of L sem , and the coefficient δ of L text , which are β1 = 10 -1 , β2 = 10 -2 , β3 = 10 -3 , γ1 = 1.0, γ2 = 10 -1 , γ3 = 10 -2 , δ1 = 10 -4 , δ2 = 10 -5 , δ3 = 10 -6 , and conducted combined experiments on loss terms with different weights. We selected the best settings considering both metrics for each combination under the test of multiple groups of weight coefficient combinations. The weight coefficients are not shown in the two tables, and the specific weights are: 1×L vis , 1×L vis + 10 -5 ×L text , 1×L vis + 10 -2 ×L feat , 1×Lvis +10 -2 ×L sem , 1×L vis +10 -2 ×L sem +10 -3 ×L feat .

[0087] Tables 1 and 2 below show the results of the best weight combination on the EDSR super-resolution backbone network. The bold numbers represent the best loss item combination in the corresponding indicator, and the underlined numbers are second.

[0088] Table 1 (SR's segmentation / ground truth):

[0089]

[0090] Table 2 (SR's segmentation / HR's segmentation):

[0091]

[0092] Table (2)

[0093] In addition, the following table 3- Figure 4 Using L on multiple models vis + Lsem +L feat The indicators obtained from the loss combination are consistent with L vis The comparison results, the bold numbers represent the optimal combination in the indicator project, the numbers on the left represent the use of L vis +L sem +L feat The result of the loss, the number on the right indicates the use of L vis results.

[0094] Table 3 (SR's segmentation / ground truth):

[0095]

[0096]

[0097] Table 4 (SR's segmentation / HR's segmentation):

[0098]

[0099] It can be seen that by using 4 mainstream and stable super-resolution models as the backbone network for training, and setting multiple groups of loss terms and loss term weight coefficients for comparative experiments, we have confirmed that the loss function designed by us has the following characteristics: under the condition of minimizing the negative impact on the human eye visual perception index, it significantly improves the semantic segmentation related indexes. At the same time, this loss has good compatibility and can be used in the design of mainstream loss functions.

[0100] Example 3

[0101] On the other hand, the present invention also proposes an image super-resolution model training device. Please refer to Figure 4 , which shows the image super-resolution model training device provided in the third embodiment of the present invention. The device includes:

[0102] An image training module 11, configured to obtain a super-resolution data set and use the super-resolution data set to train a super-resolution backbone network;

[0103] A first loss calculation module 12, configured to obtain the target super-resolution image output by the current training of the super-resolution backbone network, and calculate the loss between the target super-resolution image and the super-resolution template image to obtain a first loss term;

[0104] An image input module 13, configured to input the target super-resolution image and the super-resolution template image into a pre-trained semantic segmentation network respectively. The target super-resolution image generates a target segmentation image through the semantic segmentation network, and the super-resolution template image generates a segmentation template image through the semantic segmentation network;

[0105] A second loss calculation module 14, configured to calculate the loss between the target segmentation image and the segmentation template image to obtain a second loss term;

[0106] The image training module 11 is further configured to determine the current total loss function of the super-resolution backbone network according to the first loss term and the second loss term, and perform iterative training on the super-resolution backbone network according to the current total loss function to train an image super-resolution model.

[0107] Preferably, in some optional embodiments of the present invention, the image super-resolution model training device further includes:

[0108] An image processing module, configured to input the target super-resolution image and the super-resolution template image into a pre-trained VGG network to output the high-dimensional features of the target super-resolution image and the super-resolution template image;

[0109] A third loss calculation module, configured to calculate the difference in data distribution in the high-dimensional features of the target super-resolution image and the super-resolution template image to obtain a third loss term;

[0110] Among them, the current total loss function is also determined according to the third loss term.

[0111] Preferably, in some optional embodiments of the present invention, the image super-resolution model training device further includes:

[0112] A fourth loss calculation module, configured to calculate the distribution difference between the Gram matrices of the high-dimensional features of the target super-resolution image and the super-resolution template image, so as to obtain a fourth loss term;

[0113] Among them, the current total loss function is also determined according to the fourth loss term.

[0114] Preferably, the current total loss function satisfies the following conditional formula:

[0115] L total = α × L vis + β × L feat + γ × L sem + δ × L text

[0116] In the formula, L total represents the current total loss function, L vis represents the first loss term, L sem represents the second loss term, L feat represents the third loss term, L text represents the fourth loss term, and α, β, γ, and δ respectively represent the weights of L vis , L feat , L sem and L text .

[0117] Preferably, α takes a value of 1, β takes a value between 10 -3 -10 -1 γ takes a value between 10 -2 -1.0, and δ takes a value between 10 -6 -10 -4 .

[0118] Preferably, the semantic segmentation network is a bypass of the super-resolution backbone network.

[0119] The functions or operation steps implemented when the above-mentioned modules and units are executed are substantially the same as those in the above method embodiment, and will not be elaborated here.

[0120] In summary, the image super-resolution model training device in this embodiment proposes a method for training an image super-resolution model based on semantic segmentation constraints, using the semantic segmentation loss as a constraint term for super-resolution model training. Thus, the super-resolution problem and the segmentation problem are considered as a whole. The trained super-resolution model can improve the accuracy of semantic segmentation of the super-resolution result while ensuring comparable super-resolution metrics. Without adding training parameters, the semantic segmentation metrics of the super-resolved images are significantly improved, enabling ideal performance in image segmentation based on super-resolved images.

[0121] Example 4

[0122] Please refer to Figure 5 , Embodiment 4 of the present invention proposes an image super-resolution model training device, including a processor 10, a memory 20, and a computer program 30 stored on the memory and executable on the processor. When the processor 10 executes the program 30, it implements the image super-resolution model training method as described above.

[0123] Among them, the processor 10 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments, and is used to run the program code stored in the memory 20 or process data, such as executing an access restriction program.

[0124] Among them, the memory 20 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 20 can be an internal storage unit of the image super-resolution model training device in some embodiments, such as the hard disk of the image super-resolution model training device. The memory 20 can also be an external storage device of the image super-resolution model training device in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the image super-resolution model training device. Preferably, the memory 20 can also include both the internal storage unit and the external storage device of the image super-resolution model training device. The memory 20 can be used not only to store application software and various data installed in the image super-resolution model training device, but also to temporarily store data that has been output or will be output.

[0125] It should be noted that Figure 5The structures shown do not constitute a limitation on the image super-resolution model training device. In other embodiments, the image super-resolution model training device may include fewer or more components than those shown, or combine certain components, or have different component arrangements.

[0126] In summary, for the image super-resolution model training device in this embodiment, by proposing an image super-resolution model training method based on semantic segmentation constraints and using the semantic segmentation loss as a constraint term for super-resolution model training, the super-resolution problem and the segmentation problem are considered as a whole. The trained super-resolution model can improve the accuracy of semantic segmentation of the super-resolution result while ensuring comparable super-resolution metrics. Without adding training parameters, the semantic segmentation metrics of the super-resolution image are significantly improved, enabling ideal performance in image segmentation based on the super-resolution image.

[0127] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the image super-resolution model training method as described above.

[0128] Those skilled in the art can understand that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0129] More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion (electronic device) having one or more wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other suitable processing as necessary, and then stored in a computer memory.

[0130] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following technologies well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0131] In the description of this specification, the description referring to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples", etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0132] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for training an image super-resolution model, characterized in that: The method comprises: Obtaining a super-resolution dataset, and using the super-resolution dataset to train a super-resolution backbone network; Obtaining a target super-resolution image output by the current training of the super-resolution backbone network, and calculating the loss between the target super-resolution image and the super-resolution template image to obtain a first loss term; The target super-resolution image and the super-resolution template image are respectively input into a pre-trained semantic segmentation network, the target super-resolution image is passed through the semantic segmentation network to generate a target segmentation image, and the super-resolution template image is passed through the semantic segmentation network to generate a segmentation template image; Calculating the loss between the target segmented image and the segmented template image to obtain a second loss term, wherein the second loss term is used to measure the cross entropy quantization distance between the target super-resolved image and the super-resolved template image after the segmentation image passes through the semantic segmentation network; Determining a current total loss function of the super-resolution backbone network according to the first loss term and the second loss term, and iteratively training the super-resolution backbone network according to the current total loss function to train an image super-resolution model; After obtaining the target super-resolution image output by the current training of the super-resolution backbone network, the method further includes: Inputting the target super-resolution image and the super-resolution template image into a pre-trained VGG network to output high-dimensional features of the target super-resolution image and the super-resolution template image; Calculating the difference in data distribution between the high-dimensional features of the target super-resolved image and the super-resolved template image to obtain a third loss term; Wherein, the current total loss function is also determined based on the third loss term; In addition, after the step of inputting the target super-resolution image and the super-resolution template image into a pre-trained VGG network to output high-dimensional features of the target super-resolution image and the super-resolution template image, the method further includes: Calculate the distribution difference between the high-dimensional features of the target super-resolution image and the super-resolution template image after making a Gram matrix, and obtain a fourth loss term; Wherein, the current total loss function is also determined based on the fourth loss term; Specifically, the current total loss function satisfies the following conditional formula: Where, L total represents the current total loss function, L vis represents the first loss term, L sem represents the second loss term, L feat represents the third loss term, L text represents the fourth loss term, α, β, γ and δ represent L vis 、 L feat 、 L sem and L text The weight of .

2. The image super-resolution model training method according to claim 1, wherein: α is 1, β is 10 -3 -10 -1 The value of γ is 10 -2 -1.0, δ is 10 -6 -10 -4 between.

3. The image super-resolution model training method according to any one of claims 1 to 2, characterized in that: The semantic segmentation network is a bypass of the super-resolution backbone network.

4. An image super-resolution model training device, characterized in that: The device comprises: An image training module is used to obtain a super-resolution dataset and use the super-resolution dataset to train a super-resolution backbone network; A first loss calculation module is used to obtain a target super-resolved image output by the current training of the super-resolved backbone network, and calculate the loss between the target super-resolved image and the super-resolved template image to obtain a first loss term; An image input module is used to input the target super-resolved image and the super-resolved template image into a pre-trained semantic segmentation network respectively, wherein the target super-resolved image is passed through the semantic segmentation network to generate a target segmentation image, and the super-resolved template image is passed through the semantic segmentation network to generate a segmentation template image; A second loss calculation module is used to calculate the loss between the target segmented image and the segmented template image to obtain a second loss term, wherein the second loss term is used to measure the cross entropy quantization distance between the target super-resolved image and the super-resolved template image after the segmentation image passes through the semantic segmentation network; The image training module is further configured to determine a current total loss function of the super-resolution backbone network according to the first loss term and the second loss term, and iteratively train the super-resolution backbone network according to the current total loss function to obtain an image super-resolution model; Wherein, the image super-resolution model training device further includes: An image processing module is used to input the target super-resolution image and the super-resolution template image into a pre-trained VGG network to output high-dimensional features of the target super-resolution image and the super-resolution template image; A third loss calculation module is used to calculate the difference in data distribution in the high-dimensional features of the target super-resolved image and the super-resolved template image to obtain a third loss term; Wherein, the current total loss function is also determined based on the third loss term; In addition, the image super-resolution model training device also includes: a fourth loss calculation module, configured to calculate a distribution difference between the target super-resolution image and the super-resolution template image after the high-dimensional features are converted into a Gram matrix, thereby obtaining a fourth loss term; The current total loss function is also determined based on the fourth loss term to obtain Specifically, the current total loss function satisfies the following conditional formula: Where, L total represents the current total loss function, L vis represents the first loss term, L sem represents the second loss term, L feat represents the third loss term, L text represents the fourth loss term, α, β, γ and δ represent L vis 、 L feat 、 L sem and L text The weight of .

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image super-resolution model training method as described in any one of claims 1 to 3 is implemented.

6. An image super-resolution model training device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the image super-resolution model training method as described in any one of claims 1 to 3 is implemented.

Images