Image super-resolution processing method based on feature enhancement and adversarial learning

Abstract

The invention discloses an image super-resolution processing method based on feature enhancement and adversarial learning, and the method comprises the steps: carrying out the key feature enhancement extraction of an input low-resolution weapon equipment image, preliminarily extracting features through a series of convolution layers, focusing a key target region through an attention mechanism module, and obtaining the enhanced key features; then, constructing a generator adopting a progressive up-sampling and residual connection combined structure, and generating a high-resolution image by using the enhanced key features; thirdly, the improved discriminator adopts a multi-scale and multi-level discrimination structure, and carries out adversarial training with the generator in combination with a training algorithm with a gradient penalty term; then, constructing a composite loss function, synthesizing pixel loss, perception loss and adversarial loss, comprehensively measuring the difference between a generated image and a real high-resolution image, and guiding a training process; in order to improve the image quality, post-processing operations of adaptive filtering and contrast adaptive stretching are carried out on the generated high-resolution image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and computer vision, and relates to an image super-resolution processing method based on feature enhancement and adversarial learning. The aim is to improve the resolution of the images collected in the security monitoring scenario, and provide a high-quality and detail-rich image basis for subsequent tasks such as target recognition and event analysis. Background Art

[0002] In the field of security monitoring, the continuous operation of surveillance cameras is essential for ensuring the safety of public areas. However, in practice, various factors severely affect the quality of surveillance images. On the one hand, to control costs, the hardware performance of some cameras is limited, resulting in poor imaging quality. At the same time, the installation locations may not be able to achieve comprehensive coverage and ideal conditions, leading to poor shooting effects in some areas. Moreover, the monitoring environment is complex and variable. Insufficient lighting at night makes the picture dim, and bad weather such as rain, snow, and fog further reduces the image clarity. In addition, to save data transmission bandwidth, images are often compressed during transmission, which also causes a reduction in image resolution and a large loss of details. For example, in the monitoring of crowded places such as shopping malls and stations, the low-resolution images make the facial features of people blurred, making it impossible to accurately identify suspicious persons and difficult to make timely and accurate judgments on potential security threats, greatly weakening the effectiveness of security and traffic.

[0003] Early image super-resolution techniques mainly relied on traditional interpolation algorithms such as bilinear interpolation and bicubic interpolation. These algorithms increase the number of image pixels through simple pixel value calculations to achieve image magnification. Although the calculation process is simple and the execution efficiency is high, they are only simple operations based on local pixels and cannot effectively restore the lost high-frequency detail information. The edges of the processed images are prone to jaggedness, and the overall picture remains blurred. In the application scenario of security monitoring, they cannot meet the requirements for precise analysis of targets.

[0004] With the development of deep learning, super-resolution methods based on convolutional neural networks (CNNs) have gradually emerged. These methods construct deep neural networks to learn the mapping relationship between low-resolution images and high-resolution images, and can restore the details of images to a certain extent. However, when processing surveillance images, the CNN model is difficult to distinguish the importance differences between key target areas and background areas in the image, and treats all areas equally, resulting in an unsatisfactory detail restoration effect for key targets.

[0005] The application of Generative Adversarial Networks (GANs) in the field of image super-resolution has also achieved some results. Through the adversarial training mechanism of the generator and the discriminator, GANs can generate high-resolution images with high realism. However, there are many problems in the training process of GANs, such as the mode collapse phenomenon, that is, the images generated by the generator lack diversity; the training process is unstable, which easily leads to artifacts or incorrect details in the generated images. In the scenario of surveillance image analysis, these incorrect information may seriously mislead the decision-making.

[0006] The deficiencies of the above existing technologies all affect the execution efficiency of tasks such as personnel identity recognition and abnormal event judgment in security surveillance, and cannot meet the requirements of real-time monitoring scenarios for accurate presentation of details in key target areas while ensuring high-precision image processing quality. Summary of the Invention

[0007] The present invention aims to overcome the defects of the existing technology and proposes an image super-resolution processing method based on feature enhancement and adversarial learning. By strengthening key feature extraction, optimizing the structure of the generative adversarial network, and improving the training strategy, this method effectively improves the resolution and image quality of security surveillance images, enhances the clarity of image details, and adapts to complex and changing real-world scenarios.

[0008] To achieve the above object, the present invention adopts the following technical solutions: An image super-resolution processing method based on feature enhancement and adversarial learning, which includes the following steps:

[0009] Step 1: Design a feature extraction network specifically for security surveillance images. Use the convolutional layer to perform preliminary feature extraction on low-resolution weapon equipment images to obtain a basic feature map. On this basis, introduce an attention mechanism module, which calculates the importance weights of features in different regions to strengthen the expression of key features and obtain more representative key features;

[0010] Step 2: Construct a generator network based on key features. The generator adopts a structure that combines progressive upsampling and residual connection. During the progressive upsampling process, gradually increase the image resolution, and at the same time use the residual connection to transfer the underlying feature information to the high layer to avoid feature loss;

[0011] Step 3: Design an improved discriminator, which adopts a multi-scale and multi-level discrimination structure. It not only judges the authenticity of the generated image and the real high-resolution image from the overall image level, but also discriminates at different resolution scales and feature levels. In the adversarial training, adopt an improved training algorithm and introduce a gradient penalty term to stabilize the training process, so that the generator and the discriminator can be stably co-trained, and promote the generator to generate images closer to the real high-resolution;

[0012] Step 4: Optimize and train the network using a composite loss function, which consists of pixel loss, perceptual loss, and adversarial loss. The pixel loss is calculated based on the mean squared error to ensure the similarity of pixel values between the generated image and the real image. The perceptual loss is calculated by extracting the semantic features of the image through a pre-trained neural network to measure the similarity between the generated image and the real image at the semantic level. The adversarial loss prompts the generator to generate more realistic images and enhances the visual effect of the image. By adjusting the weights of different loss terms, the optimization objectives in various aspects are balanced;

[0013] Step 5: Perform post-processing operations on the generated high-resolution image. First, use an adaptive filtering algorithm to remove the noise in the image, and then adopt the contrast adaptive stretching technology to automatically adjust the contrast of the image according to the brightness distribution of the image itself, making the edges and textures of the target area clearer and further improving the visual quality of the image.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] Through the key feature enhanced extraction, it can accurately focus on the key parts of the surveillance image, significantly improve the recovery effect of key information. The optimized generative adversarial network structure and stable training strategy ensure the generation of high-quality high-resolution images. The composite loss function and image post-processing technology further improve the overall quality and visual effect of the image. The method of the present invention has good processing effect and strong stability, can greatly improve the analysis value of security surveillance images, and provides strong support for the efficient implementation of security work in real scenarios. Description of the Drawings

[0016] Figure 1 is the implementation flowchart of the image super-resolution processing method based on feature enhancement and adversarial learning of the present invention;

[0017] Figure 2 is the schematic diagram of the key feature extraction network structure of the present invention;

[0018] Figure 3 is the training flowchart of the generative adversarial network. Detailed Embodiments

[0019] The following further describes the embodiments and effects of the present invention in detail with reference to the drawings.

[0020] As Figure 1 、 Figure 2 shown, the image super-resolution processing method based on feature enhancement and adversarial learning of the present invention includes the following steps:

[0021] Step 1: In the image super-resolution processing flow, the key feature enhanced extraction is a crucial starting link. Assume that the input low-resolution image is I LR, first, through a series of convolutional layers C1, C2, …, C n perform preliminary feature extraction on it to obtain the basic feature map F0. The specific expression formula is:

[0022] F0 = C n (…(C2(C1(I LR ))))

[0023] Then, input the basic feature map F0 into the attention mechanism module. The role of the attention mechanism module is to enable the network to automatically focus on the key target areas in the image, enhance the feature expression of the key areas, and suppress the feature responses of secondary areas such as the background. In the attention mechanism module, first perform global average pooling and global max pooling on the basic feature map F0. Global average pooling calculates the average value of the entire feature map in the spatial dimension to obtain a feature vector F avg , which can reflect the average intensity information of each channel in the feature map. Global max pooling takes the maximum value in the spatial dimension F max , which highlights the most significant feature information of each channel in the feature map. Then, process F avg and F max through two multi-layer perceptrons (MLPs) with shared parameters respectively to obtain the intermediate features M avg and M max . Their specific mathematical expressions are as follows:

[0024] M avg = MLP(F avg ), M max = MLP(F max )

[0025] Add M avg and M max and process them through the activation function σ to obtain the channel attention weight W c . The specific expression formula is:

[0026] W c = σ(M avg + M max )

[0027] Next, perform average pooling and max pooling on the basic feature map F0 in the channel dimension to obtain F s-avg and F s-max , and splice them and then process them through the convolutional layer C s and obtain the spatial attention weight W s through the activation function σ. The specific expression formula is:

[0028] W s = σ(C s ([Fs-αvg ; F s-max ))

[0029] Finally, multiply the channel attention weights and the spatial attention weights element - by - element to obtain the final attention weight W = The enhanced key feature F att = F0 × W;

[0030] Step 2: Optimize the generator architecture. The generator adopts a structure that combines progressive upsampling and residual connections. Let the initial upsampling layer be U1, and the subsequent upsampling layers be U2, …, U m , when performing the first - layer upsampling, input the enhanced key feature F att into the initial upsampling layer U1 to obtain the feature map F1 = U1(F att ). At the same time, use the residual connection to add F att and F1 to get the fused first - layer feature F′1 = F1 + F att . In the subsequent upsampling process, for the i - th layer (i = 2, …, m), the upsampling operation is based on the fused feature F′ i-1 of the previous layer, that is, F i = U i (F′ i-1 ), and then use the residual connection again to add F i and F att to obtain the fused feature F′ i = F i + F att (i = 2, …, m). In this way, each layer of upsampling can make full use of the information of the key feature, gradually restore the details and textures of the image, and finally generate the high - resolution image I SR ;

[0031] Step 3: As Figure 3 shown, in order to make the generated image closer to the real high - resolution image and improve the quality and credibility of the image, improve the discriminator and conduct adversarial training. The discriminator adopts a multi - scale and multi - level discriminant structure, including discriminant sub - networks D1, D2, …, D k at different resolution levels and discriminant modules at different feature levels. During the training process, in order to enable the generator and the discriminator to be stably trained and avoid problems such as mode collapse, a training algorithm with a gradient penalty term is used. For the generator G and the discriminator D, the adversarial loss of the generator is set as:

[0032]

[0033] Among them, D(.) represents the discriminator's judgment result on the generated image, E(.) represents the expectation of the judgment result, and the loss L of the discriminator D is set to:

[0034]

[0035] where I HR is the real high-resolution image, is the gradient penalty term, λ is the gradient penalty coefficient, x is an image sampled from the real data distribution or the generated data distribution, represents the gradient of the discriminator D with respect to x. By alternately optimizing the parameters of the generator and the discriminator, that is, first fixing the parameters of the discriminator and optimizing the generator to make the generator generate more realistic images, and then fixing the parameters of the generator and optimizing the discriminator to make the discriminator more accurately judge the authenticity of the image, and so on in a cycle, gradually improving the quality of the images generated by the generator;

[0036] Step 4: To more precisely guide the training process of the generator and the discriminator, the present invention constructs a total loss function L total , which is composed of a combination of multiple different types of loss functions and can measure the difference between the generated image I SR and the real high-resolution image I HR from multiple perspectives. The expression of the total loss function is:

[0037]

[0038] where the total loss function L total includes the pixel loss L MSE , the perceptual loss L perceptual and the adversarial loss α and β are weight coefficients for balancing different losses. By weighted combination of different losses, comprehensively considering the differences in the pixel level, semantic level, and visual realism level of the image. The pixel loss L MSE measures the difference between the generated image and the real image from the perspective of pixel values, and its calculation formula is:

[0039]

[0040] where N is the number of pixels in the image, and represent the pixel values of the generated image and the real image at the i-th pixel position respectively. The perceptual loss L perceptual measures the difference between the generated image and the real image from the semantic level, and it is calculated using the features φ l of the l-th layer of the pre-trained VGG network. Its calculation formula is:

[0041]

[0042] Among them, M is the number of VGG network layers selected, and N l is the number of pixels of the feature at the l-th layer;

[0043] Step 5: The generated high-resolution image I SR Although there is a certain improvement in resolution and visual effect, it may be affected by noise. Therefore, a series of post-processing operations need to be performed on it to further improve the image quality. First, perform adaptive filtering on the high-resolution image I SR to obtain the image I after noise removal filtered = AF(I SR ), where AF is the adaptive filtering function. Then, use the contrast adaptive stretching technique to process the denoised image I filtered . According to the brightness distribution of I filtered , calculate the appropriate stretching parameters to adjust the contrast of the image. Assume that the brightness value of a certain pixel point in the image is I(x, y), and the brightness value after contrast adaptive stretching is I′(x, y). The stretching is achieved through a specific mapping function, and finally the enhanced high-quality image I final is obtained, making the target details in the image clearer.

[0044] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. Obviously, for professionals in the field, after understanding the content and principle of the present invention, various modifications and changes in form and details may be made without departing from the principle and structure of the present invention. However, these modifications and changes based on the idea of the present invention are still within the protection scope of the claims of the present invention.

Claims

1. An image super-resolution processing method based on feature enhancement and adversarial learning, characterized in that It includes the following steps: Step 1: In the image super-resolution processing flow, the extraction of key feature enhancement is a crucial starting link. Assume that the input low-resolution image is I LR , first, through a series of convolutional layers C1, C2, …, C n to perform preliminary feature extraction on it, obtaining the basic feature map F0. The specific expression formula is: F0 = C n (…(C2(C1(I LR )))) Then, the basic feature map F0 is input into the attention mechanism module. The role of the attention mechanism module is to enable the network to automatically focus on the key target areas in the image, enhance the feature expression of the key areas, and suppress the feature responses of secondary areas such as the background. In the attention mechanism module, first, global average pooling and global max pooling are performed on the basic feature map F0. Global average pooling calculates the average value of the entire feature map in the spatial dimension to obtain a feature vector F avg , which can reflect the average intensity information of each channel in the feature map. Global max pooling, on the other hand, takes the maximum value in the spatial dimension F max , which highlights the most significant feature information of each channel in the feature map. Then, through two multi-layer perceptrons (MLPs) with shared parameters, F avg and F max are processed respectively to obtain intermediate features M avg and M max . Their specific mathematical expressions are as follows: M avg = MLP(F avg ), M max = MLP(F max ) Add M avg and M max and process through the activation function σ to obtain the channel attention weight W c , and the specific expression formula is: W c = σ(M avg + M max ) Next, average pooling and max pooling are performed on the basic feature map F0 in the channel dimension to obtain F s-avg and F s-max , and after they are concatenated, they are processed through the convolutional layer C s , and the spatial attention weight W s is obtained through the activation function σ. The specific expression formula is: W s =σ(C s ([F s-αvg ; F s-max )) Finally, multiply the channel attention weights and the spatial attention weights element by element to obtain the final attention weights The enhanced key feature F att = F0 × W; Step 2: Optimize the generator architecture. The generator adopts a structure that combines progressive upsampling and residual connections. Let the initial upsampling layer be U1, and the subsequent upsampling layers be U2, …, U m , during the first upsampling, the enhanced key feature F att is input into the initial upsampling layer U1 to obtain the feature map F1 = U1(F att ). At the same time, using the residual connection, F att is added to F1 to obtain the fused first-layer feature F′1 = F1 + F att . During the subsequent upsampling process, for the i-th layer (i = 2, …, m), the upsampling operation is based on the fused feature F′ i-1 of the previous layer, that is, F i = U i (F′ i-1 ). Then, using the residual connection again, F i is added to F att to obtain the fused feature F′ i of the i-th layer = F i + F att (i = 2, …, m). In this way, each upsampling layer can make full use of the information of the key feature, gradually restore the details and textures of the image, and finally generate the high-resolution image I SR ; Step 3: To make the generated image closer to the real high-resolution image and improve the quality and credibility of the image, the discriminator is improved and adversarial training is carried out. The discriminator adopts a multi-scale and multi-level discriminant structure, including discriminant sub-networks D1, D2, …, D at different resolution levels k and discriminant modules at different feature levels. During the training process, in order to enable the generator and the discriminator to be stably trained and avoid the problem of mode collapse, the present invention adopts a training algorithm with a gradient penalty term. For the generator G and the discriminator D, the adversarial loss of the generator is set as: L Gadv = -E[D(I SR )] Among them, D(.) represents the discriminator's judgment result on the generated image, E(.) represents the expectation of the judgment result, and the loss L of the discriminator D is set to: Among them, I HR is the real high-resolution image, is the gradient penalty term, λ is the gradient penalty coefficient, x is the image sampled from the real data distribution or the generated data distribution, represents the gradient of the discriminator D with respect to x. By alternately optimizing the parameters of the generator and the discriminator, that is, first fixing the parameters of the discriminator and optimizing the generator to make the generator generate more realistic images, and then fixing the parameters of the generator and optimizing the discriminator to make the discriminator more accurately judge the authenticity of the images, and so on, the quality of the images generated by the generator is gradually improved; Step 4: To more precisely guide the training process of the generator and discriminator, the present invention constructs a total loss function L total , which is composed of a combination of multiple loss functions of different types and can measure the generated image I SR and the real high-resolution image I HR The difference between them, the expression of the total loss function is: Among them, the total loss function L total includes the pixel loss L MSE , the perceptual loss L perceptual and the adversarial loss L Gadv . α and β are weight coefficients for balancing different losses. Through the weighted combination of different losses, the differences in the pixel level, semantic level, and visual realism level of the image are comprehensively considered. The pixel loss L MSE measures the difference between the generated image and the real image from the perspective of pixel values, and its calculation formula is: where N is the number of image pixels, and represent the pixel values of the generated image and the real image at the i-th pixel position respectively; the perceptual loss L perceptual measures the difference between the generated image and the real image from the semantic level, and it uses the feature φ of the l-th layer of the pre-trained VGG network l to calculate; its calculation formula is: Among them, M is the number of VGG network layers selected, and N l is the number of pixels of the feature at the l-th layer; Step 5: The generated high-resolution image I SR Although there has been a certain improvement in resolution and visual effects, it may be affected by noise. Therefore, a series of post-processing operations need to be performed on it to further improve the image quality. First, for the high-resolution image I SR perform adaptive filtering to obtain the denoised image I filtered = AF(I SR ), where AF is the adaptive filtering function. Then, use the contrast adaptive stretching technique to process the denoised image I filtered . According to the brightness distribution of I filtered , calculate the appropriate stretching parameters and adjust the contrast of the image. Assume that the brightness value of a certain pixel point in the image is I(x, y), and the brightness value after contrast adaptive stretching is I′(x, y). The stretching is achieved through a specific mapping function, and finally, the enhanced high-quality image I final is obtained, making the target details in the image clearer.

Images