A super-resolution reconstruction method

Abstract

The present invention provides a super-resolution reconstruction method, comprising: establishing an image dataset; constructing a neural network structure, the neural network structure being used to extract features of the image dataset during neural network training; establishing a loss function for the neural network structure, the loss function being used to guide neural network training; training the image dataset to obtain a neural network model; and reconstructing images using the neural network model, inputting low-resolution images and outputting high-resolution images. The SRGAN (Super-Resolution Generative Adversarial Network) has been improved by modifying the network structure and loss function of the generative network (G-NET). Due to the improved G-NET, the generative network extracts more precise features, resulting in better super-resolution reconstruction results. Consequently, better results can be achieved when performing detection, recognition, and semantic segmentation.

Description

Technical field

[0001] The present invention relates to the technical field of image processing, and in particular to a super-resolution reconstruction method. [Background Technology]

[0002] Super-resolution involves increasing the resolution of an image through hardware or software. The process of reconstructing a high-resolution image from a series of low-resolution images is called super-resolution reconstruction. High resolution means a high pixel density within the image, providing more detail, which is essential in many practical applications.

[0003] Current super-resolution reconstruction methods include interpolation-based methods, reconstruction-based methods, and learning-based methods. The interpolation-based method is the most classic, including nearest neighbor interpolation and cubic spline interpolation. The reconstruction effect has oscillating and jagged over-smoothed images. The performance of the reconstruction-based method is heavily dependent on the prior knowledge of the applied high-resolution image, which can easily lead to over-smoothing and loss of important high-frequency details. The learning-based method is based on machine learning theory, especially deep learning. In recent years, some methods have emerged, the more representative of which are SRCNN (Super-Resolution Convolutional Neural Network) and SRGAN (Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network). Although these methods make up for the shortcomings of the other two methods to a certain extent and improve the clarity of the image in terms of visual perception, they cannot control the appearance of some unnecessary lines in details. [Summary of the invention]

[0004] The technical problem to be solved by the present invention is to provide a super-resolution reconstruction method, which improves the SRGAN (Super-Resolution Generative Adversarial Network) by changing the network structure of the generative network G-NET and improving the loss function. After the improvement, the generative network G-NET extracts more accurate features, making the super-resolution reconstruction effect better, so that better results can be obtained when performing detection, recognition, and semantic segmentation.

[0005] To solve the above technical problems, one embodiment of the present invention provides a super-resolution reconstruction method, including: establishing an image dataset; building a neural network structure, which is used to extract features of the image dataset during the neural network training process; establishing a loss function of the neural network structure, which is used to guide the neural network training; training the image dataset to obtain a neural network model, and outputting a high-resolution image dataset.

[0006] Preferably, the image dataset is trained to obtain a neural network model, and outputting a high-resolution image dataset includes: assigning initial network weights to the neural network structure; inputting a small batch of image datasets into the neural network structure, and updating the network weights after calculation; when the loss function converges to a convergence threshold, saving the neural network structure and network weights to obtain a neural network model, and outputting a high-resolution image dataset.

[0007] Preferably, the image data set is a folder.

[0008] Preferably, the image data set image formats include jpg, png, jpeg, and tiff.

[0009] Preferably, before training the image dataset to obtain a neural network model, the method includes setting network hyperparameters.

[0010] Preferably, the step of training the image data set to obtain a neural network model includes: starting a training program to perform training.

[0011] Preferably, the initial network weights satisfy the standard normal distribution.

[0012] Preferably, the convergence threshold is 0-0.01.

[0013] Preferably, the network hyperparameters include: learning rate, number of times the training set is trained once, number of data sets used in one iteration of training, and normalization range.

[0014] Preferably, the training program includes three neural networks, namely a generative network G-NET, an adversarial network D-NET and a VGG network VGG-NET.

[0015] Preferably, the training includes: reading training set images, disrupting the order of the training set; and saving the images obtained after the network is generated.

[0016] Preferably, the initial value of the learning rate is set to 0.001-0.01.

[0017] Preferably, the number of times the training set is trained once is set to 300-700.

[0018] Preferably, the normalization range is set to 0-1.

[0019] Preferably, the number of data sets used in one iteration of training is set to 64-128.

[0020] Preferably, the generative network and the adversarial network are trained alternately, that is, the adversarial network is first trained with the loss function of the adversarial network, and then the generative network is trained with the loss function of the generative network; the VGG network is used when calculating the loss function of the generative network.

[0021] Preferably, the first mean square error and the second mean square error are used as the loss function of the generated network G-NET.

[0022] Preferably, the learning rate is updated to 0.5 to 0.8 times the current learning rate every time 10 training set iterations are completed.

[0023] Preferably, the first mean square error and the second mean square error are both mean square errors of feature vectors output after the input three-channel image and the real data real label pass through the VGG network VGG-NET respectively.

[0024] Compared with the existing technology, the above technical solution has the following advantages: SRGAN (Super-ResolutionGenerative Adversarial Network) is improved, in which the network structure of the generative network G-NET is changed and the loss function is improved. Since the improved generative network G-NET extracts more accurate features, the super-resolution reconstruction effect is better, so better results can be obtained when performing detection, recognition, and semantic segmentation.

Brief Description of the Drawings

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 This is a flow chart of the model training phase of the super-resolution reconstruction method of the present invention.

[0027] Figure 2 This is a flow chart of the image reconstruction stage of the super-resolution reconstruction method of the present invention.

[0028] Figure 3 Schematic diagram of the residual module structure of the super-resolution reconstruction method of the present invention.

[0029] Figure 4Schematic diagram of the network structure of the adversarial network D-NET of the super-resolution reconstruction method of the present invention.

[0030] Figure 5 Schematic diagram of the VGG-16 network structure of the super-resolution reconstruction method of the present invention.

[0031] Figure 6 This is a schematic diagram of the loss calculation after extracting deep information in the super-resolution reconstruction method of the present invention. [Specific implementation method]

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0033] Example 1

[0034] The present invention provides a super-resolution reconstruction method, comprising the steps of: establishing a picture data set; building a neural network structure, the neural network structure is used to extract features of the picture data set during the neural network training process; establishing a loss function of the neural network structure, the loss function is used to guide the neural network training; training the picture data set to obtain a neural network model; using the neural network model to reconstruct the picture, inputting a low-resolution picture, and outputting a high-resolution picture after the neural network model calculates. Neural network model: includes a neural network interface and a neural network weight. Neural network structure: represents the connection relationship of the neural network. Before training, there is only the neural network structure. During training, the weights are obtained, and then the neural network model is obtained. During the super-resolution reconstruction process, the neural network structure is first built, and the neural network weights are trained to obtain the neural network model. During use, feature extraction and image reconstruction are performed by using the neural network model.

[0035] Training an image dataset to create a neural network model involves assigning initial network weights to the neural network structure, which must adhere to a standard normal distribution. A small batch of image datasets is fed into the neural network structure, and the network weights are updated after computation. When the loss function converges to a threshold, the neural network structure and weights are saved to create the neural network model, which then outputs a high-resolution image dataset. In practice, the threshold is set between 0 and 0.01.

[0036] The image dataset is a folder. In practice, a certain number of subfolders can be set under the folder. The number refers to the number of people in the training set. Each subfolder stores all the images of one person, and different subfolders represent different people.

[0037] The image dataset image formats include but are not limited to jpg, png, jpeg, and tiff.

[0038] Before training an image dataset to obtain a neural network model, the following steps are required: Network hyperparameters are set. These include the learning rate, the number of training passes, the number of datasets used in one training iteration, and the normalization range. The initial learning rate is set to 0.001 to 0.01. In practice, the initial learning rate value can be set to 0.01. The number of training passes is set to 300 to 700. After every 10 training pass iterations, the learning rate is updated to 0.5 to 0.8 times the current learning rate. The normalization range is set to 0 to 1. The number of datasets used in one training iteration is set to 64 to 128.

[0039] Training the image dataset to obtain a neural network model includes: starting the training program and performing training. The training program includes three neural networks, namely the generative network G-NET, the adversarial network D-NET, and the VGG network VGG-NET. Training includes: reading the training set images, disrupting the order of the training set; and saving the images obtained after passing through the generative network. The generative network and the adversarial network are trained alternately, that is, the adversarial network is first trained with the loss function of the adversarial network, and then the generative network is trained with the loss function of the generative network; the VGG network is used when calculating the loss function of the generative network. The first mean square error and the second mean square error are used as the loss functions of the generative network G-NET. The first mean square error and the second mean square error are both the mean square errors of the feature vectors output after the input three-channel image and the real data true label pass through the VGG network VGG-NET respectively.

[0040] Example 2

[0041] Figure 1 This is a flow chart of the model training phase of the super-resolution reconstruction method of the present invention. Figure 1 This is a flowchart of the training phase, which is used when training the model. The purpose is to obtain the model parameters of the generative network G-NET after training. It is the first step in super-resolution reconstruction. The method includes:

[0042] S10: The training dataset. A dataset is a folder containing high-resolution images in formats such as jpg, png, jpeg, and tiff. Different datasets are used depending on the scenario.

[0043] For example: if you want to improve the image resolution of a certain camera, then collect the clear pictures taken by the camera. If it is divided into different time periods, you should also collect pictures from different time periods.

[0044] It should be noted here that since the generative network G-NET does not contain a fully connected layer, the image sizes of the training set and the test set can be inconsistent. Because when calculating the difference loss, each image is compared with the feature vector output by the network, and the sizes of the two are consistent.

[0045] Therefore, this is also an advantage of the present invention, in that pictures of different sizes can be used as network input, avoiding the loss of picture information caused by scaling the input pictures to a uniform size.

[0046] S20: represents the generative network. This is one of the networks that needs to be trained during the training process. During the image reconstruction phase, it extracts image features and generates the reconstructed image. The input is an n×n low-resolution image, and the output is a 4n×4n image, meaning the image is magnified 4 times in both length and width.

[0047] The reference standard (ground truth) is correctly labeled data. The loss function is used to measure the difference between the network output and the reference standard. During training, it is necessary to measure the difference (loss) between the features that generate the network output and the reference standard (ground truth) at the pixel level. The mean-square error (MSE) can measure this difference and is defined as follows:

[0048]

[0049] where y i is the reference standard (ground truth), The prediction results of the fitted data, the sample set size is m.

[0050] However, for the feature vectors (including three-channel images) output by the network for both real data and fitted data, the process of calculating the mean square error (MSE) is to sum the squares of the pixel differences at the corresponding positions of the corresponding channels and finally calculate the average. First, for the single-channel feature vector (feature maps), the mean square error (MSE) is defined as follows:

[0051]

[0052] where y ij is the groundtruth pixel value in row i and column j of the true feature maps. It is the pixel value of the i-th row and j-th column in the fitted feature vector (feature maps). The size of the feature vector (feature maps) is an n×m matrix.

[0053] For the mean square error (MSE) of multi-channel feature maps, the process of obtaining it is to obtain the mean square error (MSE) of each channel separately, and finally add them up to get the average.

[0054] S30: Adversarial network, used only in training. During the training phase, it uses the output image of the generative network as input and then determines whether the image is a true high-definition image, rather than the output image of the generative network. Then, a generative loss and an adversarial loss are generated. These two losses are used to optimize the model parameters during training. They are defined as follows:

[0055] When optimizing adversarial networks:

[0056]

[0057] Wherein D represents the function representation of the adversarial network; G represents the function representation of the generative network; z represents the input of the adversarial network, usually referring to white noise, and the present invention is a low-resolution image with pixel normalization from 0 to 1; G(z) represents the characteristic vector output by the generative network with z as input, and the present invention is a high-resolution three-channel image output by the generative network (pixel normalization from 0 to 1); D(G(z) represents the output characteristic vector obtained by the generative network with z as input, and then the secondary characteristic vector is input to the adversarial network, and the output value is obtained by calculation, and the output value is a value from 0 to 1; x represents the image of the real reference input of the adversarial network, and the present invention is a real high-definition resolution image, and is normalized from 0 to 1; Pdata(x) represents the distribution of the real training set, and the present invention represents the distribution of the high-resolution training set; Pz(z) represents the distribution of white noise, and the present invention is the distribution of the low-resolution training set made of high resolution; E z～Pz(z) [log(1-D(G(z)))] represents the loss of the discriminant network on the low-resolution training set; D(x) represents the output value obtained by the discriminant network with the high-resolution training set as input, and the output value is a value between 0 and 1; E x～Pdata(x) [log(D(x))] represents the loss of the discriminative network on the real high-definition training set; This means that for the two networks D and V, the weights of network D are optimized by maximizing the loss function V(D, G). This formula states that the adversarial network's loss function is V(D, G). During model training, the adversarial network's loss is increased. Ultimately, when the loss reaches its maximum, the adversarial network can most accurately identify whether the input image is real or generated by the generative network. At this point, the adversarial network's model parameters are optimized. This is the process and purpose of optimizing the adversarial network.

[0058] Conventional methods optimize the generated network:

[0059]

[0060] Wherein: D and G represent the functional representation of the adversarial network and the generative network. For example, in D(z), z represents the input of the adversarial network, and D(z) represents the output of the adversarial network. The output result is generally "1" or "0" to judge true or false. G(z) represents the characteristic vector output by the generative network with z as input; D(G(z)) represents that the adversarial network is the output result, and the input is G(z), which is the characteristic vector output by the generative network. V(D,G) represents the loss function of the network structure composed of the adversarial network D and the generative network D. Pz(z) represents the distribution of white noise, and the present invention uses high resolution to make the distribution of low resolution training set.

[0061] The purpose of the generative network is to generate an image that is indistinguishable from a real image from an input image. For example, a low-resolution image is input and a high-resolution image is generated, and this high-resolution image is consistent with the real image in nature, rather than a fake one. For example, if a low-resolution face image is input, the generative network outputs a high-resolution face image, rather than a high-resolution landscape image or a high-resolution image with special textures but meeting the requirements.

[0062] The above is the optimization method of conventional generative adversarial networks. However, this method adds four more mean square error losses when optimizing the generative network, namely the generative network loss:

[0063] g_loss=mse1+mse2+mse1_vgg+mse2_vgg+V(D,G)

[0064] In the formula, mse1 represents the mean squared error (MSE) between the intermediate output of the generative network, Image1, and the true high-resolution image; mse2 represents the mean squared error (MSE) between the final output of the generative network, Output images, and the true high-resolution image; mse1_vgg represents the mean squared error (MSE) between Image1 and the true high-resolution image after feature extraction through the VGG network; and mse2_vgg represents the mean squared error between Output images and the true high-resolution image after feature extraction through the VGG network. V(D, G) represents the loss function for the generative and adversarial networks. When optimizing the generative network, the generative network loss g_loss is minimized, replacing the conventional loss function. The ultimate goal is to achieve better super-resolution reconstruction results for the generative network. The optimization method used is stochastic gradient descent (SGD).

[0065] The model training adopts the stochastic gradient descent method to minimize the loss function g_loss of the generated network and the loss function V(D,G) of the adversarial network. That is, the loss function is allowed to find the minimum value along the fastest direction of gradient descent, so as to adjust the initialization weight of the model network and obtain the optimal weight value of the network model.

[0066] S40: VGG network. It is a tool for extracting deep features from images and is used only during training. It extracts key information that represents the key features of the image, effectively mitigating the effects of factors such as lighting and noise.

[0067] During training, it is necessary to measure the difference (loss) between the features of the generated network output and the reference standard (ground truth) at the pixel level. VGG is used to extract their underlying key features, and then the mean square error can be used to more accurately measure this difference.

[0068] In order to obtain a clearer reconstruction effect, the invention also uses two mean square errors as the loss function of the generation network: mse1_vgg and mse2_vgg.

[0069] mse1 is the mean squared error between the output of the G-NET network layer, Image1, and the reference standard, which is an image that is 1 / 2 the size of the input high-definition image. Similarly, mse2 is the mean squared error between the output of the G-NET network layer, Outputs images, and the reference standard, which is the input high-definition image.

[0070] In specific implementation, after starting training, a mini-batch of images is taken from the training set, i.e., the input images. After being compressed to 0.25 times and pixel normalized to [-1.1], they can be input into the G-NET network.

[0071] G-NET has two outputs, one is Image1 output by the middle network layer, and the other is the output image Output Images output by the last network layer. Then, the loss of the loss generation network and the loss of the adversarial network can be calculated. The process is as follows: the two outputs are respectively compared with the corresponding reference standards to obtain two mean square error losses; the two outputs are respectively compared with the corresponding reference standards and the deep features are extracted by VGG-NET to obtain two more mean square error losses; the output image Output Images is input to D-NET to calculate an adversarial loss and a generation loss;

[0072] Epoch: The number of times the entire training set is trained once. n epochs means training the entire training set n times. Mini-batch: The number of datasets used in one training iteration. An epoch consists of multiple mini-batches. After the loss is determined, backpropagation can be performed. The G-Net is used to generate the loss, while the D-Net is used to counter the loss. These two backpropagations are performed sequentially. After completion, training can proceed to the next mini-batch, and so on. After training reaches the maximum number of epochs, the neural network model weights are saved and training terminates.

[0073] Figure 2 This is a flow chart of the image reconstruction stage of the super-resolution reconstruction method of the present invention. Figure 2 This is the process of generating the network G-NET module in the reconstruction phase. It is to load the model parameters obtained in the training phase into the program, reconstruct the input low-resolution image, and then output a high-resolution image. This process is the second step of super-resolution. The method includes:

[0074] 210: The input of the generative network G-NET module is a three-channel color image with equal length and width, but the size is not limited. It is assumed to be n×n.

[0075] 220: It is a convolutional layer of the generated network G-NET module. The meaning of Conv-BN-ReLU (kernel3-num64-stride1) is, Conv: represents the convolution layer; BN: represents a BN layer (Batch Normalization) connected after the convolution layer; ReLU: represents that the activation function used by the convolution layer is ReLU; kernel3: represents that the size of the convolution kernel is 3×3. Similarly, if it is kernel5, then it means that the convolution kernel size is 5×5; num64: represents 64, which is the number of convolution kernels. Similarly, if it is num128, then the number of convolution kernels is 128; stride1: represents that the step size of the convolution process is 1. Similarly, if stride2, then it means that the convolution process is 2. The structural parameters of the network layer are set based on experience.

[0076] 230: It is a residual module in the generative network G-NET module. The structure of the residual module is as follows Figure 3 As shown in the figure, the meaning of Residual Block (kernel3-num64-stride1) is: Residual Block: indicates that this is a residual module, kernel3: indicates that the size of the convolution kernel is 3×3; num64: indicates the number of convolution kernels 64; stride1: indicates that the step size of the convolution process is 1.

[0077] 240: This part contains 6 residual modules with exactly the same structure.

[0078] 250: represents a three-channel feature, which can be understood as a three-channel color image, that is, a picture slightly clearer than the input image, and the size is also 2n×2n;

[0079] 260: This is a deconvolution layer in the G-NET module, which amplifies the input features. Deconv-BN-ReLU (kernel3-num64-stride2), where Deconv indicates a deconvolution layer, and BN, ReLU, kernel3, num64, and stride2 have the same meanings as 220.

[0080] 270: Represents a three-channel color image, a super-resolution reconstructed image, also of size 4n × 4n. In implementation, the input is a three-channel image Inputimages of size n × n and pixel normalized to [-1.1] (a mini-batch of images during training). Then, according to the arrows, it passes through the Conv-BN-ReLU, Residual Block, and other network layers, outputting an intermediate feature vector Image1 and a final feature vector Outputimages. The intermediate feature vector Image1 is three-channel and presented in image form, as is the case with Outputimages.

[0081] In the specific implementation, image1 is the feature vector (feature maps) output by the network layer, with a size of 3×2n×2n (3 represents the number of channels); the image of the high-definition image of the training set is compressed by 1 / 2, with a size of 3×2n×2n; mse1 is the mean squared error between image1 and the reference standard (ground truth) image of the high-definition image compressed by 1 / 2;

[0082] Similarly, output images are feature maps output by the network layer, with a size of 3×4n×4n (3 represents the number of channels). The high-definition images in the training set are compressed by 1 / 2, with a size of 3×4n×4n. mse2 is the mean squared error between the output images and the input high-definition image as a reference standard.

[0083] During training, mse1 and mse2 are used to calculate the loss function of the generative network. However, experiments have shown that using only mse1 and mse2 produces images with good color richness but still lacks clarity. Therefore, the invention also uses the VGG network (a common deep learning network) to extract deep features from the image before measuring the difference.

[0084] Figure 3 Schematic diagram of the residual module structure of the super-resolution reconstruction method of the present invention. Figure 3 It is a residual block. Figure 2 The detailed structure of 230 is a common neural network structure.

[0085] 310: Represents the input of the residual module, which is usually the output features of the previous network layer. When the residual module is used as the first layer, the input is usually in the form of an image;

[0086] 320: Conv-ReLU (kernel3-num64-stride1) means, Conv: represents the convolution layer; ReLU: indicates that the activation function used in the convolution layer is ReLU; kernel3: indicates that the size of the convolution kernel is 3×3. Similarly, if it is kernel5, then the convolution kernel size is 5×5; num64: indicates the number of convolution kernels, 64. Similarly, if it is num128, then the number of convolution kernels is 128; stride1: indicates that the step size of the convolution process is 1. Similarly, if stride2, then it means that the convolution process is 2. The structural parameters of the network layer are set based on experience.

[0087] 330: Indicates that the input of the residual module is directly transmitted to the output of the residual module, and after being merged with the output of the network layer, it is used as the final output of the residual module. The purpose of this approach is mainly to prevent the gradient from disappearing during the training process, thereby making the entire network layer deeper.

[0088] 340: represents the output of the residual module.

[0089] When implementing it specifically, Figure 3 This is a schematic diagram of the residual module structure. The essence is the process of convolution calculation. The input x is a certain feature vector. Then, after two convolution calculations, the feature vector is obtained and added to the input x to get the output y of the residual module.

[0090] Figure 4 Schematic diagram of the network structure of the adversarial network D-NET of the super-resolution reconstruction method of the present invention. Figure 4The network structure of the adversarial network D-NET is used in training to identify whether the input image is a real training set image or an image output by the generated network G-NET network.

[0091] 410: Indicates that this network is input, which is a three-channel color image. The output of the generative network G-NET is directly used as the input of this network, as well as the high-definition image of the training set.

[0092] 420 represents the output of this network, which is a number between 0 and 1. It is essentially a probability. The larger the number, the greater the possibility that the adversarial network D-NET network determines that the input image is a real image.

[0093] When implementing it specifically, Figure 4 This is a schematic diagram of the network structure of D-NET. The process is that the input feature vector Inputs is calculated by the 6-layer convolutional network in succession, and finally outputs Outputs.

[0094] Figure 5 Schematic diagram of the VGG-16 network structure of the super-resolution reconstruction method of the present invention. Figure 5 It is the classic VGG-16 network structure, which aims to extract deep features of the input image and avoid interference from factors such as lighting and noise.

[0095] 510: represents the input, which is a three-channel image with pixels normalized to [-1.1];

[0096] 520: is a convolutional layer, "3×3conv,64" is decomposed into three parts, "3×3" represents the size of the convolution kernel, similarly 1×1 and 5×5; "conv" indicates that this is a convolutional layer; "64" represents the number of convolution kernels, similarly 128, 256, and 512 all represent the number of convolution kernels;

[0097] 530: Also represents a convolutional layer, but the convolutional layer has an additional pooling layer compared to the convolutional layer of 520. That is, "pool / 2" indicates that this is a pooling layer, and after the pooling layer, the feature map becomes half of its original size;

[0098] 540: Indicates a fully connected layer. “fc, 4096” is decomposed into two parts: “fc” means that this is a fully connected layer; “4096” indicates the number of fully connected nodes.

[0099] 550: Indicates the output of the network.

[0100] When implementing it specifically, Figure 5This is a schematic diagram of the VGG-16 network structure, which shows that a three-channel image with pixels normalized to [-1.1] is input. According to the direction of the arrow, it is calculated through multiple convolutional layers and fully connected layers to obtain the output feature vector y.

[0101] Figure 6 This is a schematic diagram of the calculation loss after extracting deep information in the super-resolution reconstruction method of the present invention, showing the calculation process of mse1_vgg and mse2_vgg. Figure 6 As shown, the calculation of mse1_vgg and mse2_vgg has an extra VGG feature extraction process. mse1_vgg is the feature vector image1 of G-NET and the reference standard is a high-definition picture reduced by 1 / 2. After the features are extracted by the VGG network, the feature vector is output and the mean square error is calculated; similarly, mse2_vgg is the output picture outputImages and the reference standard is a high-definition picture. After passing through VGG, the feature vector is output and the mean square error is calculated. It is worth noting that mse1_vgg and mse2_vgg are unique to the present invention and are also key operations. The purpose is to control the generation process of the generation network through these two loss functions, so that the generated pictures can gradually achieve a clearer effect.

[0102] The process uses a low-resolution input image (one-quarter the size of the HD image) and, after a network operation, generates image1, which is double the size. At this point, image1 must not differ significantly from the HD image, which is halved in size. Image1 then undergoes another network operation, and the final output image is reconstructed. mse1_vgg and mse2_vgg complement each other, jointly improving the reconstruction effect.

[0103] As can be seen from the above description, a super-resolution reconstruction method according to the present invention is used to improve SRGAN (Super-Resolution Generative Adversarial Network), in which the network structure of the generative network G-NET is changed and the loss function is improved. Since the improved generative network G-NET extracts more accurate features, the effect of super-resolution reconstruction is better, so better results can be obtained when performing detection, recognition, and semantic segmentation. It can effectively solve the problem that the current learning-based method is difficult to control the texture of image details, making the reconstructed image clearer. In applications such as detection and recognition, using this method can improve the resolution of the original image, thereby improving the algorithm effect.

[0104] The embodiments of the present invention are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A super-resolution reconstruction method, characterized in that: include: Create an image dataset; Building a neural network structure, wherein the neural network structure is used to extract features of the image dataset during the neural network training process; Establishing a loss function for a neural network structure, wherein the loss function is used to guide neural network training; Training the image dataset to obtain a neural network model; Training the image data set to obtain a neural network model includes: starting a training program to perform training; the training program includes three neural networks, namely a generative network G-NET, an adversarial network D-NET, and a VGG network VGG-NET; The generative network and the adversarial network are trained alternately, that is, the adversarial network is first trained with the loss function of the adversarial network, and then the generative network is trained with the loss function of the generative network; the VGG network is used when calculating the loss function of the generative network; The first mean square error and the second mean square error are used as the loss function of the generated network G-NET. The first mean square error is: the feature vector of G-NET and the reference standard are respectively passed through the VGG network to extract features, and the output feature vectors are used to calculate the mean square error; the second mean square error is: the output picture and the reference standard are respectively passed through the VGG network to calculate the mean square error of the feature vectors; Use a neural network model to reconstruct images, input low-resolution images, and output high-resolution images.

2. The super-resolution reconstruction method according to claim 1, wherein: The image dataset is trained to obtain a neural network model, and the output high-resolution image dataset includes: Assign initial network weights to the neural network structure; Input a small batch of image data sets into the neural network structure and update the network weights after calculation; When the loss function converges to the convergence threshold, the neural network structure and network weights are saved to obtain the neural network model, and a high-resolution image dataset is output.

3. The super-resolution reconstruction method according to claim 1, wherein: The image dataset is a folder.

4. The super-resolution reconstruction method according to claim 1, wherein: The image dataset image formats include jpg, png, jpeg, and tiff.

5. The super-resolution reconstruction method according to claim 1, wherein: Before training the image dataset to obtain a neural network model, the process includes setting network hyperparameters.

6. The super-resolution reconstruction method according to claim 2, wherein: The initial network weights satisfy the standard normal distribution.

7. The super-resolution reconstruction method according to claim 2, wherein: The convergence threshold is 0 to 0.

01.

8. The super-resolution reconstruction method according to claim 5, wherein: The network hyperparameters include: learning rate, number of times the training set is trained once, number of data sets used in one iteration of training, and normalization range.

9. The super-resolution reconstruction method according to claim 1, wherein: Training includes: reading the training set images, disrupting the order of the training set; and saving the images obtained after passing the generation network.

10. The super-resolution reconstruction method according to claim 8, characterized in that: The initial value of the learning rate is set to 0.001~0.

01.

11. The super-resolution reconstruction method according to claim 8, wherein: Set the number of training sets to be trained once between 300 and 700.

12. The super-resolution reconstruction method according to claim 8, wherein: Set the normalization range to 0 to 1.

13. The super-resolution reconstruction method according to claim 8, wherein: The number of datasets used for one iteration in training is set to 64 to 128.

14. The super-resolution reconstruction method according to claim 11, wherein: Set the learning rate to 0.5 to 0.8 times the current learning rate after every 10 training set iterations.

Images