Image super-resolution method and system based on structure and texture features

Abstract

The invention discloses an image super-resolution method and system based on structural and textural features, and the method comprises the steps: constructing a generative adversarial network with the structural and textural feature extraction capability, inputting a super-resolution image data set into the generative adversarial network, training and generating the weight of a corresponding model, and obtaining an image super-resolution model according to the weight, inputting the to-be-super-resolution image into the image super-resolution model, and generating a super-resolution high-definition image by the image super-resolution model. By fusing the structure and texture features of the image, the super-resolution recovery effect of the image is improved, the loss of details is avoided, and meanwhile, an efficient performance optimization and user-friendly interface is provided, so that the image processing process is more convenient.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image super-resolution method and system based on structure and texture features. Background Art

[0002] Image super-resolution (SR) method refers to the process of reversely restoring a high-resolution (HR) image from a low-resolution (LR) image. This technology meets the demand for image clarity in application scenarios by restoring image detail information and improving image quality, and is widely used in medical imaging, remote sensing imaging, security monitoring and other fields. For example, in medical imaging, it improves diagnostic accuracy; in remote sensing processing, it enhances the resolution of ground objects; in security monitoring, it optimizes image details and helps public safety. This technology has an important position and broad prospects in the field of modern image processing.

[0003] Image super-resolution methods can be mainly divided into traditional methods and deep learning-based convolutional neural network methods. Traditional methods mainly include interpolation methods and example methods. These methods need to rely on prior knowledge of the image. Interpolation methods such as nearest neighbor, bilinear and bicubic interpolation improve image resolution through simple mathematical models, but it is difficult to restore high-resolution details of the image. Example-based methods restore details by using similar blocks or feature information in images or image databases. For example, non-local means and sparse representation techniques supplement missing details by searching for areas similar to the target image in the image database. These methods can improve the restoration of image details to a certain extent, but due to the need to match and model prior information, their computational complexity is high and their generalization ability for large-scale complex scenes is limited.

[0004] Methods based on deep learning, especially convolutional neural network (CNN) methods, are increasingly being used in the field of image super-resolution. The advantage of deep learning methods is that they no longer rely on manually designed image features or prior knowledge, and can be trained with large amounts of data to automatically extract the most suitable features for the task and directly generate high-resolution images end-to-end. These methods can effectively recover more high-frequency details from low-resolution images by automatically learning the multi-level features and complex relationships of images.

[0005] Currently, although super-resolution models have been able to restore image clarity to a certain extent through technologies such as deep learning and convolutional neural networks (CNN), many methods are still unable to effectively cope with the challenges of multi-scale and complex backgrounds, resulting in the final generated images being blurred or lacking sufficient details, thus affecting the breadth and effectiveness of their application. Summary of the invention

[0006] The present invention provides an image super-resolution method and system based on structure and texture features, which are used to solve the technical problem that when existing deep learning models are expected to generate high-resolution images with reasonable structure and rich texture details, the images are generally blurred or lack sufficient details.

[0007] In a first aspect, the present invention provides an image super-resolution method based on structure and texture features, comprising: Preprocessing the image data set to obtain a super-resolution image data set; Constructing a generative adversarial network with structural and texture feature extraction capabilities, the generative adversarial network includes a generator and a discriminator, the generator includes a dense variable receptive field convolution block module, an upsampling module with a depth-separable deconvolution module, and the discriminator includes a U-NET type network architecture; Inputting the super-resolution image dataset into the generative adversarial network, training and generating weights of the corresponding model, and obtaining an image super-resolution model according to the weights; The image to be super-resolutioned is input into the image super-resolution model, and the image super-resolution model generates a high-definition image after super-resolution.

[0008] In a second aspect, the present invention provides an image super-resolution system based on structure and texture features, comprising: A preprocessing module is configured to preprocess the image data set to obtain a super-resolution image data set; A construction module configured to construct a generative adversarial network with structural and texture feature extraction capabilities, wherein the generative adversarial network includes a generator and a discriminator, wherein the generator includes a dense variable receptive field convolution block module, an upsampling module with a depth-separable deconvolution module, and the discriminator includes a U-NET type network architecture; A training module, configured to input the super-resolution image dataset into the generative adversarial network, train and generate weights of the corresponding model, and obtain an image super-resolution model according to the weights; The output module is configured to input the image to be super-resolution into the image super-resolution model, and the image super-resolution model generates a high-definition image after super-resolution.

[0009] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the image super-resolution method based on structural and texture features of any embodiment of the present invention.

[0010] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor executes the steps of the image super-resolution method based on structural and texture features of any embodiment of the present invention.

[0011] The image super-resolution method and system based on structure and texture features of the present application construct a generative adversarial network with the ability to extract structure and texture features, input a super-resolution image dataset into the generative adversarial network, train and generate weights for the corresponding model, obtain an image super-resolution model based on the weights, input the image to be super-resolution into the image super-resolution model, and the image super-resolution model generates a high-definition image after super-resolution, thereby improving the image super-resolution restoration effect and avoiding detail loss by fusing the structure and texture features of the image, while providing efficient performance optimization and a user-friendly interface, making the image processing process more convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 A flowchart of an image super-resolution method based on structure and texture features provided by an embodiment of the present invention; Figure 2 A structural block diagram of an image super-resolution system based on structure and texture features provided by an embodiment of the present invention; Figure 3 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] See also Figure 1 , which shows a flowchart of an image super-resolution method based on structure and texture features of the present application.

[0016] like Figure 1As shown, the image super-resolution method based on structure and texture features specifically includes the following steps: Step S101, preprocessing the image data set to obtain a super-resolution image data set.

[0017] In this step, in the training phase, the original image is a high-resolution image, and the high-resolution image is converted into a low-resolution image using the bicubic interpolation method. In the test phase, the image is the original low-resolution image. After obtaining the low-resolution image, the Canny edge detector is used to generate the edge structure image and grayscale image corresponding to the low-resolution image. This embodiment uses two public image datasets, CelebA-HQ and Paris Street View, in the experiment.

[0018] Step S102, constructing a generative adversarial network with structural and texture feature extraction capabilities, wherein the generative adversarial network includes a generator and a discriminator, wherein the generator includes a dense variable receptive field convolution block module and an upsampling module with a depth-separable deconvolution module, and the discriminator includes a U-NET type network architecture.

[0019] In this step, the input of the generator is a dual parallel branch structure with synchronized structural and texture features. Each branch structure is composed of multiple dense variable receptive field convolution block modules connected in series. Dual feature fusion is performed in the upsampling stage at the output end, and the two features are sent to the deep separable deconvolution module together to amplify the feature details and obtain a high-definition image as output.

[0020] The dense variable receptive field convolution block module is composed of three groups of dense convolution subunits with residual connections connected in series, wherein the first dense convolution subunit adopts a 1×1 convolution kernel, the second dense convolution subunit adopts a 3×3 convolution kernel, and the third dense convolution subunit adopts a 5×5 convolution kernel. The structure of the dense variable receptive field convolution block module is expressed as follows: , In the formula, It is the feature of the input dense variable receptive field convolution block module, It is the feature output by the dense variable receptive field convolution block module. , , and are weighting coefficients, is a dense convolution subunit; Each dense convolution subunit is composed of multiple corresponding n×n convolutional layers and ReLU activation functions, and features are transferred across layers through dense connections. Dense connections enable the network to learn features at different levels more effectively and avoid information loss. The output of each layer is not only used as the input of the next layer, but also directly passed to the subsequent layers, thereby promoting the efficient fusion of features of different scales. Finally, the dense convolution subunit integrates the outputs of all layers through a convolution operation to generate the final output of the dense convolution unit, providing richer feature representation for subsequent network modules. The expression of dense connection is: , In the formula, , , , and All are convolution features. Indicates that the convolution kernel is used The ordinary convolution operation, is the LeakyRelu activation function.

[0021] The convolution process of the depthwise separable deconvolution module includes depthwise deconvolution and pointwise convolution, wherein, in the depthwise deconvolution, each convolution kernel processes only one channel of the input feature map, and in the pointwise convolution, the feature map is integrated according to the pointwise convolution to generate a new feature map. The expression of the depthwise separable convolution of the depthwise separable deconvolution module is: , In the formula, is the feature of the input depth-wise separable deconvolution module, is the feature output by the depthwise separable deconvolution module, is point-wise convolution, To use the convolution kernel The depth deconvolution of Indicates that the convolution kernel is used The ordinary convolution operation, is the LeakyRelu activation function.

[0022] The discriminator consists of three layers of encoders, three layers of decoders and one layer of output convolution. It uses the skip connection between the encoder and the decoder to enhance feature transfer, and maps the discriminant features to a two-dimensional plane by gradually extracting features. The discriminant features are normalized to the 0-1 range through the Sigmoid activation function.

[0023] Step S103: input the super-resolution image data set into the generative adversarial network, train and generate weights of the corresponding model, and obtain an image super-resolution model according to the weights.

[0024] In this step, the combined loss function of the image super-resolution model includes a pixel loss sub-function, a perceptual loss sub-function, and a relative adversarial loss sub-function; The expression for calculating the combined loss function is: , In the formula, is the combined loss function, is the weight of the pixel loss sub-function, is the pixel loss sub-function, is the weight of the perceptual loss sub-function, is the perceptual loss subfunction, is the weight of the relative adversarial loss sub-function, is the relative adversarial loss sub-function; Among them, the expression for calculating the pixel loss subfunction is: , In the formula, is the mathematical expectation, The super-resolution image generated by the generator, is the real image of the corresponding size; The expression for calculating the perceptual loss subfunction is: , In the formula, For the VGG-19 network The activation map of the pooling layer, To use the VGG network Extract features, To use the VGG network Extract features; The expression for calculating the relative adversarial loss subfunction is: , In the formula, For the generator, is the discriminator, Generate an image discriminator for real images. To generate an image relative to the real image discriminator, For Find the mathematical expectation, For Find the mathematical expectation.

[0025] Step S104: input the image to be super-resolutioned into the image super-resolution model, and the image super-resolution model generates a high-definition image after super-resolution.

[0026] In summary, the method of the present application constructs a generative adversarial network with the ability to extract structural and texture features, inputs a super-resolution image dataset into the generative adversarial network, trains and generates weights for the corresponding model, obtains an image super-resolution model based on the weights, inputs the image to be super-resolution into the image super-resolution model, and the image super-resolution model generates a high-definition image after super-resolution, thereby improving the image super-resolution restoration effect and avoiding detail loss by fusing the structure and texture features of the image, while providing efficient performance optimization and a user-friendly interface, making the image processing process more convenient.

[0027] In a specific embodiment, the simulation experiment of the present application is implemented under PYTORCH, and the NVIDIA(R) Tesla(R) V100 GPU is used for training and testing. For the convenience of comparison, all low-resolution images are adjusted to 128×128 pixels and the image super-resolution is enlarged to 512×512 pixels by ×4 times, the batch size is set to 4, and the Adam optimizer is used. The initial learning rate is 0.0002, and the fine-tuning learning rate is 0.0001.

[0028] The superior performance of the proposed method is verified by comparing it with three other representative image super-resolution algorithms of the same type. The three algorithms are SRGAN, ESRGAN, and Real-ESRGAN. For image quality evaluation indicators, several common indicators in image processing tasks are used: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learning-perceptual image patch similarity (LPIPS). The experimental results on the CelebA-HQ and Paris Street View datasets are compared in Table 1.

[0029] Table 1: Comparison of experimental results of data sets , It can be seen from the image super-resolution index results given in Table 1 that in the image super-resolution enhancement process, the method of the present invention is superior to the comparison method. At the same time, the error between the pixel values ​​of the experimental image and the real image obtained is the smallest among all experimental methods, and the image super-resolution effect is the best.

[0030] See also Figure 2 , which shows a structural block diagram of an image super-resolution system based on structure and texture features of the present application.

[0031] like Figure 2 As shown, the image super-resolution system 200 includes a pre-processing module 210 , a construction module 220 , a training module 230 and an output module 240 .

[0032] Among them, the preprocessing module 210 is configured to preprocess the image data set to obtain a super-resolution image data set; the construction module 220 is configured to construct a generative adversarial network with structural and texture feature extraction capabilities, the generative adversarial network includes a generator and a discriminator, the generator includes a dense variable receptive field convolution block module, an upsampling module with a depth-separable deconvolution module, and the discriminator includes a U-NET type network architecture; the training module 230 is configured to input the super-resolution image data set into the generative adversarial network, train and generate weights of the corresponding model, and obtain an image super-resolution model according to the weights; the output module 240 is configured to input the image to be super-resolution into the image super-resolution model, and the image super-resolution model generates a high-definition image after super-resolution.

[0033] It should be understood that Figure 2 Modules and references documented in Figure 1 Therefore, the operations and features described above for the method and the corresponding technical effects are also applicable to Figure 2 The modules in it will not be described in detail here.

[0034] In some other embodiments, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor executes the image super-resolution method based on structure and texture features in any of the above method embodiments; As an implementation mode, the computer-readable storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are configured as follows: Preprocessing the image data set to obtain a super-resolution image data set; Constructing a generative adversarial network with structural and texture feature extraction capabilities, the generative adversarial network includes a generator and a discriminator, the generator includes a dense variable receptive field convolution block module, an upsampling module with a depth-separable deconvolution module, and the discriminator includes a U-NET type network architecture; Inputting the super-resolution image dataset into the generative adversarial network, training and generating weights of the corresponding model, and obtaining an image super-resolution model according to the weights; The image to be super-resolutioned is input into the image super-resolution model, and the image super-resolution model generates a high-definition image after super-resolution.

[0035] The computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the image super-resolution system based on structure and texture features, etc. In addition, the computer-readable storage medium may include a high-speed random access memory, and may also include a memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the image super-resolution system based on structure and texture features via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0036] Figure 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, the device includes: a processor 310 and a memory 320. The electronic device may also include: an input device 330 and an output device 340. The processor 310, the memory 320, the input device 330 and the output device 340 may be connected via a bus or other means. Figure 3 In the example, the connection through the bus is taken as an example. The memory 320 is the above-mentioned computer-readable storage medium. The processor 310 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 320, that is, the image super-resolution method based on structure and texture features of the above-mentioned method embodiment is implemented. The input device 330 can receive input digital or character information, and generate key signal input related to user settings and function control of the image super-resolution system based on structure and texture features. The output device 340 may include display devices such as display screens.

[0037] The electronic device can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0038] As an implementation mode, the electronic device is applied to an image super-resolution system based on structure and texture features, and is used for a client, and includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: Preprocessing the image data set to obtain a super-resolution image data set; Constructing a generative adversarial network with structural and texture feature extraction capabilities, the generative adversarial network includes a generator and a discriminator, the generator includes a dense variable receptive field convolution block module, an upsampling module with a depth-separable deconvolution module, and the discriminator includes a U-NET type network architecture; Inputting the super-resolution image dataset into the generative adversarial network, training and generating weights of the corresponding model, and obtaining an image super-resolution model according to the weights; The image to be super-resolutioned is input into the image super-resolution model, and the image super-resolution model generates a high-definition image after super-resolution.

[0039] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image super-resolution method based on structural and texture features, characterized in that: include: Preprocessing the image data set to obtain a super-resolution image data set; Constructing a generative adversarial network with structural and texture feature extraction capabilities, the generative adversarial network includes a generator and a discriminator, the generator includes a dense variable receptive field convolution block module, an upsampling module with a depth-separable deconvolution module, and the discriminator includes a U-NET type network architecture; Inputting the super-resolution image dataset into the generative adversarial network, training and generating weights of the corresponding model, and obtaining an image super-resolution model according to the weights; The image to be super-resolutioned is input into the image super-resolution model, and the image super-resolution model generates a high-definition image after super-resolution.

2. The image super-resolution method based on structure and texture features according to claim 1, characterized in that: The combined loss function of the image super-resolution model includes a pixel loss sub-function, a perceptual loss sub-function and a relative adversarial loss sub-function; The expression for calculating the combined loss function is: ， In the formula, is the combined loss function, is the weight of the pixel loss sub-function, is the pixel loss sub-function, is the weight of the perceptual loss sub-function, is the perceptual loss subfunction, is the weight of the relative adversarial loss sub-function, is the relative adversarial loss sub-function; Among them, the expression for calculating the pixel loss subfunction is: ， In the formula, is the mathematical expectation, The super-resolution image generated by the generator, is the real image of the corresponding size; The expression for calculating the perceptual loss subfunction is: ， In the formula, For the VGG-19 network The activation map of the pooling layer, To use the VGG network Extract features, To use the VGG network Extract features; The expression for calculating the relative adversarial loss subfunction is: ， In the formula, For the generator, is the discriminator, Generate an image discriminator for real images. To generate an image relative to the real image discriminator, For Find the mathematical expectation, For Find the mathematical expectation.

3. The image super-resolution method based on structure and texture features according to claim 1, characterized in that: The input end of the generator is a dual parallel branch structure with synchronized structural texture features. Each branch structure is composed of multiple dense variable receptive field convolution block modules connected in series. Dual feature fusion is performed in the upsampling stage at the output end, and the two features are sent to the depthwise separable deconvolution module to amplify the feature details, and a high-definition image is output.

4. The image super-resolution method based on structure and texture features according to claim 1, characterized in that: The dense variable receptive field convolution block module is composed of three groups of dense convolution subunits with residual connections connected in series, wherein the first dense convolution subunit adopts a 1×1 convolution kernel, the second dense convolution subunit adopts a 3×3 convolution kernel, and the third dense convolution subunit adopts a 5×5 convolution kernel. The structure of the dense variable receptive field convolution block module is expressed as follows: ， In the formula, It is the feature of the input dense variable receptive field convolution block module, It is the feature output by the dense variable receptive field convolution block module. , , and are weighting coefficients, is a dense convolution subunit; Each dense convolution subunit consists of multiple corresponding n×n convolutional layers and ReLU activation functions, and features are transferred across layers through dense connections. The expression of dense connections is: ， In the formula, , , , and They are all convolution features. Indicates that the convolution kernel is used The ordinary convolution operation, is the LeakyRelu activation function.

5. The image super-resolution method based on structure and texture features according to claim 1, characterized in that: The convolution process of the depthwise separable deconvolution module includes depthwise deconvolution and pointwise convolution, wherein, in the depthwise deconvolution, each convolution kernel processes only one channel of the input feature map, and in the pointwise convolution, the feature map is integrated according to the pointwise convolution to generate a new feature map. The expression of the depthwise separable convolution of the depthwise separable deconvolution module is: ， In the formula, is the feature of the input depth-wise separable deconvolution module, is the feature output by the depthwise separable deconvolution module, is point-wise convolution, To use the convolution kernel The depth deconvolution of Indicates that the convolution kernel is used The ordinary convolution operation, is the LeakyRelu activation function.

6. The image super-resolution method based on structure and texture features according to claim 1, characterized in that: The discriminator includes three layers of encoders, three layers of decoders and one layer of output convolution. The skip connection between the encoder and the decoder is used to enhance feature transfer. The discriminant features are mapped to a two-dimensional plane by gradually extracting features, and the discriminant features are normalized to the range of 0-1 by the Sigmoid activation function.

7. An image super-resolution system based on structure and texture features, characterized in that: include: A preprocessing module is configured to preprocess the image data set to obtain a super-resolution image data set; A construction module configured to construct a generative adversarial network with structural and texture feature extraction capabilities, wherein the generative adversarial network includes a generator and a discriminator, wherein the generator includes a dense variable receptive field convolution block module, an upsampling module with a depth-separable deconvolution module, and the discriminator includes a U-NET type network architecture; A training module, configured to input the super-resolution image dataset into the generative adversarial network, train and generate weights of the corresponding model, and obtain an image super-resolution model according to the weights; The output module is configured to input the image to be super-resolution into the image super-resolution model, and the image super-resolution model generates a high-definition image after super-resolution.

8. An electronic device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images