An unmanned aerial vehicle remote sensing video blind motion deblurring method based on adversarial learning and contrastive learning

Abstract

The application provides a UAV remote sensing video blind motion blur removing method based on adversarial learning and contrast learning, a characteristic model ALCLDeblur includes a generator with a CAM attention mechanism, a discriminator with a CAM attention mechanism and a registerer.The method comprises the following steps: collecting data, constructing a plurality of groups of UAV remote sensing image blur image-clear image pairs (x, y) to form a training data set; training a model, training the characteristic model ALCLDeblur according to the training data set; testing the model, inputting the blur image of the UAV remote sensing video to be processed into the trained generator to obtain the corresponding deblurring image.The generator comprises a generator CAM attention layer, the discriminator comprises a discriminator CAM attention layer, and the main problem to be solved is the inaccurate focusing and poor generation quality when generating the deblurring image.The registerer extracts the shape and color gamut distribution information of the image, obtains the difference between the generated image and the original image, guides the generator to retain the spatial and color gamut information of the image during the generation process, and improves the quality of the generated image.Finally, the network model is optimized through the generative adversarial loss, the registerer contrast loss and the identity content loss.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing based on deep learning, and in particular to a method for blind motion deblurring of unmanned aerial vehicle remote sensing video based on adversarial learning and contrastive learning. BACKGROUND

[0002] In recent years, with the rapid development of unmanned aerial vehicle technology, the cost of unmanned aerial vehicle remote sensing image shooting equipment has gradually decreased, and remote sensing images are no longer limited to satellite shooting. Although the coverage of unmanned aerial vehicle shooting is becoming wider and wider, due to the limitations of unmanned aerial vehicle flight motion, shooting object motion, unmanned aerial vehicle, camera lens shaking and lens aperture, etc., the imaging of remote sensing images exists in the case of object target motion blur. The existence of such motion blur will reduce the application value of unmanned aerial vehicle remote sensing images and cannot meet the subsequent tasks of other image processing fields, such as target detection, target tracking, etc. Therefore, there is an urgent need for a method for motion deblurring of unmanned aerial vehicle remote sensing images.

[0003] At present, most image motion deblurring methods adopt the framework based on neural network or deep model, mainly multi-scale convolutional neural network, generative adversarial network, etc. Such methods have two limitations: 1) the process of generating deblurred images exists in the problem of inaccurate focusing and poor generation quality; 2) the color domain loss problem existing in the traditional network structure. In addition, the shooting method of the images in the existing data set is mostly handheld camera shooting, and there are few motion deblurring algorithms for unmanned aerial vehicle remote sensing images. There is no large remote sensing image data set for image motion deblurring at present, which also increases the difficulty of this task.

[0004] Based on this, the present application proposes a method for blind motion deblurring of unmanned aerial vehicle remote sensing video based on adversarial learning and contrastive learning. The present application provides an image motion deblurring method for the characteristics of unmanned aerial vehicle remote sensing video, which can maximize the restoration of the details of blurred images. This provides a foundation for better visual task application of unmanned aerial vehicle remote sensing video, so that the deblurred remote sensing images can have utilization value in other computer tasks or other scenarios. In addition, the research uses DJI Air 2S unmanned aerial vehicle to simulate flight and collect data, and uses the obtained 5.4K / 30fps high-definition video to construct unmanned aerial vehicle remote sensing image motion blur image / clear image pairs to form a remote sensing image motion deblurring data set.

[0005] At present, the deep learning computing platform mostly uses Nvidia processor, in order to promote the application of full nationalization, the application proposes a name as MobiUS Ascend Core, which is an airborne embedded chip based on Huawei Atlas 200 Ascend 310 processor. Atlas 200 is a high-performance low-power AI acceleration module developed by Huawei, based on Huawei Ascend 310 AI processor, the size is only half of a credit card, the power consumption is only 9.5 watts, can provide 22TOPS INT8 computing power, supports 20-way real-time analysis of high-definition video (1080P 25FPS), can be deployed on devices such as cameras, drones and robots, supports milliwatt sleep and millisecond wake-up, typical power consumption is only 5.5W, enabling edge AI application.

[0006] Based on the above method, data set and computing platform, a completely autonomous and controllable domestic unmanned aerial vehicle remote sensing video blind motion deblurring system is constructed, and the unmanned aerial vehicle remote sensing video motion deblurring task is completed. SUMMARY

[0007] In order to solve the above-mentioned defects existing in the prior art, the purpose of the application is to provide an unmanned aerial vehicle remote sensing video blind motion deblurring method based on adversarial learning and contrast learning, which aims to reduce or eliminate the problems of mode collapse and overfitting in the training process, and can generate clear images according to unmanned aerial vehicle remote sensing video images with motion blur.

[0008] According to the first aspect of the application, an unmanned aerial vehicle remote sensing video blind motion deblurring method based on adversarial learning and contrast learning is provided, characterized in that:

[0009] The feature model ALCLDeblur includes: 1 generator with CAM attention mechanism, 1 discriminator with CAM attention mechanism, and 1 register.

[0010] The method comprises:

[0011] Step 10: collect data, the unmanned aerial vehicle flies and collects high-definition video data, and constructs a plurality of blurred image-clear image pairs (x, y) of unmanned aerial vehicle remote sensing images according to the high-definition video data, to form a training data set.

[0012] Step 20: training the model, training the feature model ALCLDeblur according to the training data set, wherein the generator inputs the blurred image x, outputs the deblurred image G(x), the discriminator inputs the clear image y and the deblurred image G(x), and outputs the discrimination result, the discrimination result includes the main discrimination result G(y), D(G(x)), and the auxiliary discrimination result η(y), D(G(x)), the registration device inputs the deblurred image G(x) and the clear image y, and outputs the registration feature map R(G(x)) and R(y), and the generator is trained according to the generator total loss.

[0013] Step 30: test the model, input the blurred image of the unmanned aerial vehicle remote sensing video to be processed into the trained generator to obtain the corresponding deblurred image.

[0014] The generator includes a generator CAM attention layer, the input feature map of the generator CAM attention layer is processed through an adaptive mean pooling layer and then is spatially mapped through a full connection layer to obtain first average pooling weights, the input feature map of the generator CAM attention layer is processed through an adaptive maximum pooling layer and then is spatially mapped through a full connection layer to obtain first maximum pooling weights, the first average pooling weights and the first maximum pooling weights are respectively multiplied with the input feature map of the generator CAM attention layer to obtain two CAM response maps, and the two CAM response maps are spliced and then are transformed through a 1×1 convolution to obtain attention features of the generator CAM as the output of the generator CAM attention layer.

[0015] The discriminator includes a discriminator CAM attention layer, the input feature map of the discriminator CAM attention layer is processed through an adaptive mean pooling layer and then is spatially mapped through a full connection layer with InstanceNorm normalization to obtain second average pooling weights, the input feature map of the discriminator CAM attention layer is processed through an adaptive maximum pooling layer and then is spatially mapped through a full connection layer to obtain second maximum pooling weights, the second average pooling weights and the second maximum pooling weights are respectively multiplied with the input feature map of the discriminator CAM attention layer to obtain two CAM response maps, and the two CAM response maps are spliced and then are transformed through a 1×1 convolution with InstanceNorm normalization to obtain second discriminator attention features and auxiliary discrimination feature maps as the output of the discriminator CAM attention layer.

[0016] The registration device includes: 1 1×1 convolution layer ω, 1 per-patch full connection layer 1 MLP layer M, 1 full connection layer v.

[0017] The convolution layer ω is used for generating a feature map Y, which represents the color gamut information of the deblurred image G(x) and the clear image y.

[0018] The per-patch full connection layer For image block cutting, a feature map Y of size HxWxC is cut into n image blocks of size P1xP2, n=(H / P1)x(W / P2), H is a number that can be divided by P1, W is a number that can be divided by P2, to obtain a feature map after block division Using per-patch fully connected layer From Y r , the i-th image block Y i is randomly selected.

[0019] The MLP layer M is used for high-level nonlinear mapping of Y i , and then input to the fully connected layer v for processing to obtain the registration feature map R=v(Y i ), The blurred image G(x) and its corresponding clear image y have the same image block number i and the same registration network weight in the registration processor.

[0020] Generator total loss

[0021]

[0022] Registration loss

[0023]

[0024] Where, the registration feature map of the deblurred image is S=R(G(x)), the registration feature map of the clear image is r=T(y), ||·|| represents the length of the matrix, and τ represents the adjustment coefficient.

[0025] Adversarial loss of generator

[0026]

[0027] p d (x) represents the probability distribution that the blurred image data x obeys; c represents the value that the generator is determined to let the discriminator think that the deblurred image is a clear image, c=1.

[0028] Identity content loss of generator

[0029]

[0030] Where, μ G(x) represents the mean of G(x), μ y represents the mean of y; σ G(x) represents the variance of G(x), σ y represents the variance of y; σ G(x)y represents the covariance of G(x) and y; c1 and c2 are constants used to stabilize the loss function, c1=(K1L) 2 , c2=(K2L) 2, L represents the range of pixels, K1 = 0.001, K2 = 0.003.

[0031] Further, the unmanned aerial vehicle remote sensing video blind motion blur removal method based on the adversarial learning and the contrast learning provided by the application has the characteristics that the generator is constructed in a feature pyramid network mode and includes a fifteen-layer structure.

[0032] The first layer and the fourteenth layer of the generator are pooling layers; the second layer of the generator is a generator conv1 convolutional layer, which is used to obtain a first depth feature, has a representative convolutional kernel of 7, a padding of 0, uses an InstanceNorm2d function for normalization, and uses a ReLU function for activation; the third layer to the fourth layer of the generator are generator conv2 convolutional layers, which are used to obtain a second depth feature, have a convolutional kernel size of 3, a step of 2, and a padding of 1, use an InstanceNorm2d function for normalization, and use a ReLU function for activation; the fifth layer to the seventh layer and the ninth layer to the eleventh layer of the generator are residual blocks, which include a boundary padding layer with a padding value of 1, a convolutional layer with a convolutional kernel size of 3, a normalization and ReLU activation layer, and a Dropout layer with a parameter of 0.5; the eighth layer of the generator is a generator CAM attention layer; the twelfth layer to the thirteenth layer of the generator are Deconv up-sampling layers, which have an up-sampling multiple of 2, perform convolution processing once after each up-sampling, have a convolutional kernel size of 3, a step of 2, and a padding of 1, and then perform InstanceNorm normalization and ReLU activation function processing again; the third layer of the generator is connected to the thirteenth layer of the generator through 1x1 convolutional skip connection, and the fourth layer of the generator is connected to the twelfth layer of the generator through 1x1 convolutional skip connection; the fifteenth layer of the generator is a convolutional layer with a convolutional kernel of 7 and a padding of 0, and the output result after Tanh activation function processing is connected to the blur image x through skip connection to generate a deblurring image G(x).

[0033] Further, the unmanned aerial vehicle remote sensing video blind motion blur removal method based on the adversarial learning and the contrast learning provided by the application has the characteristics that the generator is constructed in a feature pyramid network mode and includes a fifteen-layer structure.

[0034] The first layer to the third layer of the discriminator are discriminator conv1 convolutional layers, which have a representative convolutional kernel size of 4, a step of 2, and a padding of 2; the fourth layer of the discriminator is a discriminator CAM attention layer; the fifth layer and the sixth layer of the discriminator are discriminator conv2 convolutional layers, which have a representative convolutional kernel size of 4, a step of 1, and a padding of 2, wherein the fifth layer of the discriminator has InstanceNorm normalization and a Leaky ReLU function, the sixth layer of the discriminator has a channel number of 1, and outputs a discrimination result.

[0035] Further, the unmanned aerial vehicle remote sensing video blind motion blur removal method based on the adversarial learning and the contrastive learning provided by the application has the characteristics that the discriminator is a multi-scale discriminator, and step 20 further comprises: for each deblurred image G(x) input to the discriminator, first input the clear image y to the discriminator to obtain a scale one discrimination result, then perform a scale 2 downsampling operation on the clear image y and input the clear image y to the discriminator to obtain a scale two discrimination result, and then perform a scale 3 operation on the clear image y to obtain a scale three discrimination result.

[0036] Further, the unmanned aerial vehicle remote sensing video blind motion blur removal method based on the adversarial learning and the contrastive learning provided by the application has the characteristics that the residual block adopts a Leaky ReLU function as an activation function, and the slope of all Leaky ReLU functions in the ALCLDeblur is 0.2.

[0037] Further, the unmanned aerial vehicle remote sensing video blind motion blur removal method based on the adversarial learning and the contrastive learning provided by the application has the characteristics that step 20 further comprises: training the discriminator according to the discriminator loss, and the discriminator loss comprises a main discriminator generative adversarial loss

[0038]

[0039] and an auxiliary discriminator generative adversarial loss

[0040]

[0041] p d (y) represents a probability distribution to which the clear image data y conforms; p d (x) represents a probability distribution to which the blurred image data x conforms; represents an expectation that the clear image data y sample is discriminated as true; represents an expectation that the deblurred image G(x) sample is discriminated as false; a and b respectively represent the deblurred image G(x) sample code and the clear image data y sample code, c represents a value defined by the generator in order to make the discriminator discriminate the deblurred image G(x) as true, and b=c=1 and a=0 are set.

[0042] Further, the unmanned aerial vehicle remote sensing video blind motion blur removal method based on the adversarial learning and the contrastive learning provided by the application has the characteristics that step 20 further comprises: gradient regularization is introduced, and the generator, the discriminator and the aligner are subjected to regularization processing.

[0043] Further, the unmanned aerial vehicle remote sensing video blind motion blur removal method based on the adversarial learning and the contrastive learning provided by the application has the characteristics that step 20 further comprises: fixing the discriminator and the registration device, training the generator and updating the parameters, fixing the generator, training the discriminator and the registration network.

[0044] Further, the unmanned aerial vehicle remote sensing video blind motion blur removal method based on the adversarial learning and the contrastive learning provided by the application has the characteristics that step 20 further comprises: using the Adam method to train the feature model ALCLDeblur, setting n = 256, P1 = P2 = 16, and τ = 0.007; and the learning rate of the generator and the registration device is selected as 0.0001, and the learning rate of the discriminator is selected as 0.0002.

[0045] According to a second aspect of the application, a computer device is provided, comprising: a memory for storing instructions; and a processor for calling the instructions stored in the memory to execute the unmanned aerial vehicle remote sensing video blind motion blur removal method based on the adversarial learning and the contrastive learning of the first aspect.

[0046] According to a third aspect of the application, a computer readable storage medium is provided, which stores instructions, and the instructions are executed by a processor to execute the unmanned aerial vehicle remote sensing video blind motion blur removal method based on the adversarial learning and the contrastive learning of the first aspect.

[0047] Compared with the prior art, the above technical solutions conceived by the application have at least the following beneficial effects:

[0048] 1. The self-attention layer of the prior art cannot filter out irrelevant noise points, so a large amount of calculation is required, and the self-attention layer of the application can reduce the calculation amount during work. The global max pooling and fully connected layer mechanism are used in the self-attention layer, which helps to prevent overfitting. Adding the CAM attention layer structure in the generator improves the feature extraction capability, and adding the CAM attention layer in the discriminator can help the model to extract the target information in the blurred image, so as to better represent the blurred target, strengthen the feature representation of the target, and improve the deblurring effect and restore more image details.

[0049] 2. In the adversarial learning of the prior art, although the generator can effectively guide the generation direction of the generator, the original information of the image will be destroyed in the image generation process, so that the generated image has a large number of noise points and serious information loss. The registration device is designed to obtain the difference between the generated image and the original image, and to guide the generator to retain the spatial and color gamut information of the image during the generation process. The registration device designed by the application mainly maps the block information and color gamut information of the image into the feature space, aiming to extract the shape and color gamut distribution information of the image, rather than using a deep network to extract the texture and contour information of the image.

[0050] 3、The prior art identity content loss based on L1 norm constraint considers the overall pixel constraint, but does not consider the local color gamut loss and spatial position loss, and the application proposes a metric method based on contrast learning, which uses image blocks as the input of the registration device, effectively measures the distance between local features and overall features, thereby ensuring the accuracy and speed of the model during the training process.

[0051] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings incorporated in and forming a part of the specification illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0053] Figure 1 is a feature model schematic diagram of an unmanned aerial vehicle remote sensing video blind motion deblurring method based on adversarial learning and contrast learning according to an exemplary embodiment.

[0054] Figure 2 is a basic flow schematic diagram of an unmanned aerial vehicle remote sensing video blind motion deblurring method based on adversarial learning and contrast learning according to an exemplary embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0056] The purpose of the application is to provide an unmanned aerial vehicle remote sensing video blind motion deblurring method based on adversarial learning and contrast learning, which aims to reduce or eliminate the problems of mode collapse and overfitting in the training process, and can generate clear images according to unmanned aerial vehicle remote sensing video images with motion blur.

[0057] The application provides an unmanned aerial vehicle remote sensing video blind motion deblurring method based on adversarial learning and contrast learning, which comprises a feature model ALCLDeblur, as shown in Figure 1

[0058] The feature model ALCLDeblur comprises: 1 generator with CAM attention mechanism, 1 discriminator with CAM attention mechanism, and 1 registration device.

[0059] ​Through the training of the feature model ALCLDeblur, the deblurring quality of the unmanned aerial vehicle remote sensing video image is improved, and good pre-sequence support is provided for subsequent image processing tasks such as target detection, target segmentation, target tracking, scene segmentation and the like, and the technical problems involved in the background technology are solved.

[0060] Among them, the generator with CAM attention mechanism and the discriminator mainly aim at the problems of inaccurate focusing and poor generation quality existing in the process of generating deblurred images by existing methods. In view of the color gamut loss problem existing in the traditional network structure, the aligner aims to map the block information and color gamut information of the image into the feature space, extract the shape and color gamut distribution information of the image, so as to obtain the difference between the generated image and the original image, and guide the generator to retain the spatial and color gamut information of the image in the generation process, thereby improving the quality of the generated image. Based on the above structure, further propose the generation adversarial loss, aligner contrast loss and identity content loss to optimize the network model.

[0061] The method comprises steps 10 to 30, as shown in Figure 2

[0062] Step 10: Collect data, unmanned aerial vehicle flight and collect high-definition video data, and construct a plurality of unmanned aerial vehicle remote sensing image blur-image pairs (x, y) according to the high-definition video data, to form a training data set.

[0063] In some embodiments, a DJI Air 2S unmanned aerial vehicle is used to simulate flight and collect data, and a blur-image pair is constructed using the obtained 5.4K / 30fps high-definition video to form a remote sensing image deblurring training data set.

[0064] Step 20: Train the model, and train the feature model ALCLDeblur according to the training data set, wherein the generator inputs the blurred image x and outputs the deblurred image G(x), the discriminator inputs the clear image y and the deblurred image G(x), and outputs the discrimination result, the discrimination result includes the main discrimination result D(y), D(G(x)), and the auxiliary discrimination result η(y), η(G(x)), the aligner inputs the deblurred image G(x) and the clear image y, and outputs the registration feature map R(G(x)) and R(y), and the generator is trained according to the total loss of the generator.

[0065] In some embodiments, the training model is specifically implemented through a pytorch framework, the model code and the training data set are deployed on a workstation and optimized for training, and then the feature model ALCLDeblur (adversarial learning and contrast learning deblur) is obtained, wherein ALCL represents adversarial learning and contrast learning, and Deblur represents deblurring.​

[0066] In the training process, the input of the generator is the blurred image in the blurred image-clear image pair, and the output is the generated deblurred image; the input of the discriminator is the deblurred image output by the generator and the clear image in the image pair, and the output is the discrimination result of the discriminator (false if the input image is a deblurred image, and true if the input image is a clear image), so as to obtain the trained generator model; the input of the aligner is the clear image in the image pair and the generated deblurred image, and the alignment loss value guides the training of the generator G.

[0067] Step 30: test the model, input the blurred image of the unmanned aerial vehicle remote sensing video to be processed into the trained generator to obtain the corresponding deblurred image.

[0068] In some embodiments, the blurred image of the remote sensing video to be processed, the trained feature model ALCLDeblur, is deployed on Atlas 200DK after conversion, and then the inference operation of the model is performed to test the effect of deblurring the remote sensing image.

[0069] The generator includes a generator CAM attention layer. The input feature map of the generator CAM attention layer is processed through an adaptive mean pooling layer and then through a full connection layer for spatial mapping to obtain a first average pooling weight. The input feature map of the generator CAM attention layer is processed through an adaptive maximum pooling layer and then through a full connection layer for spatial mapping to obtain a first maximum pooling weight. The first average pooling weight and the first maximum pooling weight are respectively multiplied with the input feature map of the generator CAM attention layer to obtain two CAM response maps. The two CAM response maps are spliced and then subjected to 1x1 convolution transformation to obtain the attention feature of the generator CAM as the output of the generator CAM attention layer.

[0070] The function of the generator CAM attention layer structure is to improve the feature extraction capability. The generator CAM attention layer can better represent the image information, automatically extract the target that needs to be deblurred in the image, and guide the generation through the attention mechanism point by point, accurately locate the target information, and improve the deblurring effect of the image.

[0071] The traditional self-attention layer cannot filter out irrelevant noise points, so a large amount of calculation is required. In order to solve this problem, we improved the self-attention layer, which can reduce the amount of calculation during work. We used global maximum pooling and full connection layer mechanism in the self-attention layer, which helps to prevent overfitting.

[0072] The discriminator includes a discriminator CAM attention layer, an input feature map of the discriminator CAM attention layer is processed through an adaptive mean pooling layer and then is processed through a fully connected layer with InstanceNorm normalization for spatial mapping to obtain a second average pooling weight, an input feature map of the generator CAM attention layer is processed through an adaptive maximum pooling layer and then is processed through a fully connected layer for spatial mapping to obtain a second maximum pooling weight, the second average pooling weight and the second maximum pooling weight are respectively multiplied with the input feature map of the discriminator CAM attention layer to obtain two CAM response maps, and the two CAM response maps are spliced and then are processed through a 1x1 convolution transformation with InstanceNorm normalization to obtain a second discriminator attention feature and an auxiliary discrimination feature map as an output of the discriminator CAM attention layer.

[0073] In the training process, the main discriminator result of the discriminator mainly extracts global and local features of the image, the auxiliary discrimination result mainly extracts self-attention features of the image, and finally outputs a self-attention feature map node. The discriminator CAM attention layer can help the model to extract target information in the blurred image, so as to better represent the blurred target and strengthen the feature representation of the target, thereby improving the deblurring effect.

[0074] In the adversarial learning, although the generator can effectively guide the generation direction of the generator, in the generation process of the image, the original information of the image is destroyed, so that the generated image has a large number of noise points and serious information loss. Some methods use identity loss to alleviate this phenomenon, but it can only alleviate this problem to a certain extent, and at the same time, it consumes additional memory and computing power, reducing the model training speed.

[0075] In view of the above problems, the aligner is designed to obtain the difference between the generated image and the original image, and to guide the generator to retain the spatial and gamut information of the image in the generation process. The aligner designed mainly maps the block information and gamut information of the image into the feature space, aiming to extract the shape and gamut distribution information of the image, rather than using a deep network to extract the texture and contour information of the image.

[0076] The aligner includes: 1 1x1 convolution layer omega, 1 per-patch fully connected layer 1 MLP layer Mu, and 1 fully connected layer v. The aligner aims to ensure the distribution between the original image and the generated image. The input is the deblurred image and the clear image, and the output is the contrast loss result of the aligner.

[0077] The convolution layer omega is used to generate a feature map Y, representing the gamut information of the deblurred image G(x) and the clear image y, so as to facilitate the aligner to better measure the difference between the blurred image and the original image.

[0078] The per-patch fully connected layer For image block cutting, aiming at local extraction of features after characterization, and then more accurate measurement from local to whole to generate effect of deblurring image and details of original image, a feature map Y with size H*W*C is cut into n image blocks with size P1*P2, n=(H / P1)*(W / P2), H is a number that can be divided by P1, W is a number that can be divided by P2, to obtain a feature map after block division Using per-patch fully connected layer Randomly select the i-th image block Y r from Y i .

[0079] MLP layer M is used for high-level nonlinear mapping of Y i , to ensure that the color distribution of the source domain and the target domain is consistent, and then input to the fully connected layer v to obtain the registration feature map R=v(Y i ), the blurred image G(x) and the corresponding clear image y select the same number i of image blocks in the registration processor, and use the same registration network weight.

[0080] Generator total loss

[0081]

[0082] Since the identity content loss based on L1 norm constraint considers the overall pixel constraint, but does not consider the local color domain loss and spatial position loss, aiming at this problem, the application proposes a metric method based on contrast learning, uses image blocks as the input of the registration network, effectively measures the distance between local features and overall features, so as to ensure the accuracy and rapidity of the model in the training process.

[0083] Registration loss

[0084]

[0085] Wherein, the registration feature map of the deblurred image is S=R(G(x)), the registration feature map of the clear image is T=R(y), ||·|| represents the length of the matrix, and τ represents the adjustment coefficient.

[0086] The function can effectively ensure that the generated image and the target image are in the same color domain and the color change is effectively controlled, and provide accurate gradient guidance for the generation of the image.

[0087] Generator generative adversarial loss

[0088]

[0089] p d(x) represents a probability distribution to which the blurred image data x is subjected; c represents a value determined by the generator in order for the discriminator to consider the deblurred image to be a clear image, c = 1.

[0090] Identity content loss of the generator

[0091]

[0092] wherein μ G(x) represents the mean of G(x), μ y represents the mean of y; σ G(x) represents the variance of G(x), σ y represents the variance of y; σ G(x)y represents the covariance of G(x) and y; c1, c2 are constants used to stabilize the loss function, c1 = (K1L) 2 , c2 = (K2L) 2 , L represents the range of pixels, K1 = 0.001, K2 = 0.003.

[0093] In some embodiments, the generator adopts Inception-ResNet-v2 as a backbone network, adopts a feature pyramid network (FPN) mode for construction, and includes a fifteen-layer structure:

[0094] The first layer and the fourteenth layer of the generator are pooling layers, which are used to supplement the images with insufficient sizes; the second layer of the generator is a generator conv1 convolutional layer, which is used to obtain first deep features, and the representative convolution kernel size is 7, the padding is 0, the InstanceNorm2d function is used for normalization, and the ReLU function is used for activation; the third layer to the fourth layer of the generator are generator conv2 convolutional layers, which are used to obtain second deep features, the convolution kernel size is 3, the step is 2, the padding is 1, the InstanceNorm2d function is used for normalization, and the ReLU function is used for activation; the fifth layer to the seventh layer and the ninth layer to the eleventh layer of the generator are residual blocks, and the residual block includes a boundary padding layer with a padding value of 1, a convolutional layer with a convolution kernel size of 3, a normalization and ReLU activation layer, and a Dropout layer with a parameter of 0.5; the eighth layer of the generator is a generator CAM attention layer; the twelfth layer to the thirteenth layer of the generator are Deconv up-sampling layers, the up-sampling multiple is 2, and convolution processing is performed once after each up-sampling, the convolution kernel size is 3, the step is 2, the padding is 1, and InstanceNorm normalization and ReLU activation function processing are performed again; the third layer of the generator is connected to the thirteenth layer through a 1*1 convolutional skip connection, and the fourth layer of the generator is connected to the twelfth layer through a 1*1 convolutional skip connection, so that the features with the same size in the down-sampling process and the up-sampling process are connected through a convolutional layer to prevent excessive loss of features; the fifteenth layer of the generator is a convolutional layer with a convolution kernel of 7 and a padding of 0, and the output result after Tanh activation function processing is connected to the fuzzy image x to generate a deblurring image G(x).

[0095] In some embodiments, the discriminator includes a six-layer structure.

[0096] The first layer to the third layer of the discriminator are discriminator conv1 convolutional layers, and the representative convolution kernel size is 4, the step is 2, and the padding is 2; the fourth layer of the discriminator is a discriminator CAM attention layer; the fifth layer and the sixth layer of the discriminator are discriminator conv2 convolutional layers, and the representative convolution kernel size is 4, the step is 1, and the padding is 2, wherein the fifth layer of the discriminator is provided with an InstanceNorm normalization and a Leaky ReLU function, the sixth layer of the discriminator has a channel number of 1, and outputs a discrimination result.

[0097] In some embodiments, the unmanned aerial vehicle remote sensing video blind motion blur removal method based on the adversarial learning and the contrast learning provided by the application is characterized in that the discriminator is a multi-scale discriminator (MultiScaleDiscriminator), and the selected discrimination head scale is 3.

[0098] Step 20 further comprises: for each input deblurred image G(x) of the discriminator, first input the clear image y into the discriminator to obtain a scale-one discrimination result, then input the clear image y into the discriminator after a scale-2 downsampling operation to obtain a scale-two discrimination result, and input the clear image y into the discriminator after a scale-3 downsampling operation to obtain a scale-three discrimination result.

[0099] In some embodiments, the residual block adopts a Leaky ReLU function as an activation function, and the slope of all Leaky ReLU functions in the ALCLDeblur is 0.2.

[0100] In some embodiments, step 20 further comprises: training the discriminator according to a discriminator loss, wherein the discriminator loss comprises a main discriminator generative adversarial loss

[0101]

[0102] and an auxiliary discriminator generative adversarial loss

[0103]

[0104] p d (y) represents a probability distribution to which the clear image data y conforms; p d (x) represents a probability distribution to which the blurred image data x conforms; represents an expectation that the clear image data y sample is discriminated as true; represents an expectation that the deblurred image G(x) sample is discriminated as false; a and b respectively represent the deblurred image G(x) sample code and the clear image data y sample code, and c represents a value defined by the generator for the discriminator to discriminate the deblurred image G(x) as true, wherein b=c=1 and a=0.

[0105] In some embodiments, step 20 further comprises: gradient regularization is introduced to regularize the generator, the discriminator and the registration network. The introduction of gradient regularization can ensure the stability of the model.

[0106] In some embodiments, step 20 further comprises: first fixing the discriminator and the registration network, training the generator and updating the parameters, then fixing the generator, training the discriminator and the registration network. This training method can reduce the interaction between modules and prevent the model from not converging.

[0107] In some embodiments, step 20 further comprises: training the feature model ALCLDeblur using an Adam method. Adam uses momentum and adaptive learning rate to accelerate the convergence speed, which is fast in convergence speed, easy to adjust parameters, and it can iteratively update the neural network weights based on the training data. Unlike the traditional stochastic gradient descent, which maintains the same learning rate, the Adam method calculates the adaptive learning rate of different parameters from the first and second moments of the gradient. Wherein beta1 represents the exponential decay rate of the first moment estimate, beta1 = 0.5, and beta2 represents the exponential decay rate of the second moment estimate, beta2 = 0.999.

[0108] Set n = 256, P1 = P2 = 16, tau = 0.007; the generator and the registration learning rate are selected as 0.0001, and the discriminator learning rate is selected as 0.0002.

[0109] In some embodiments, the Ascend computing power and the MindX SDK suite are selected as the application development suite for development, and the MindSpore framework is used for algorithm development and model training. The Ascend AI processor accelerated AI software development kit (SDK) provided by the MindX SDK and the related application programming interface (API) are used, and mxVision is also used for development.

[0110] To verify the technical effect of the present application, the SRN, DeblurGAN-v2, DMPHN, and MIMO-UNet methods are selected as the comparison methods for experiments.

[0111] The PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural SIMilarity) are used to measure the quality of the generated image.

[0112] The experimental results are as follows

[0113]

[0114]

[0115] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0116] It should be understood that the application is not limited to the precise construction which has been described above and which shown in the drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application should be limited only by the appended claims.

Claims

1. A blind motion blur removal method for UAV remote sensing videos based on adversarial learning and contrastive learning, characterized in that, The feature model ALCLDeblur includes: a generator with CAM attention mechanism, a discriminator with CAM attention mechanism, and a registration mechanism; The method includes: Step 10: Data Acquisition. The UAV flies and acquires high-definition video data. Based on the high-definition video data, multiple sets of blurred image-sharp image pairs (x,y) of UAV remote sensing images are constructed to form a training dataset. Step 20: Train the model. Train the feature model ALCLDeblur based on the training dataset. The generator takes a blurred image x as input and outputs a deblurred image G(x). The discriminator takes a sharp image y and a deblurred image G(x) as input and outputs a discrimination result, which includes the main discrimination result D(y) and D(G(x)) as well as the auxiliary discrimination result η(x) and η(G(x)). The register takes a deblurred image G(x) and a sharp image y as input and outputs registration feature maps R(G(x)) and R(y). Train the generator based on the generator's total loss. Step 30: Test the model by inputting the blurred image of the UAV remote sensing video to be processed into the trained generator to obtain the corresponding deblurred image; The generator includes a generator CAM attention layer. The input feature map of the generator CAM attention layer is processed by an adaptive average pooling layer and then spatially mapped by a fully connected layer to obtain the first average pooling weight. The input feature map of the generator CAM attention layer is processed by an adaptive max pooling layer and then spatially mapped by a fully connected layer to obtain the first max pooling weight. The first average pooling weight and the first max pooling weight are multiplied element-wise with the input feature map of the generator CAM attention layer. The two CAM response maps obtained are concatenated and then transformed by a 1×1 convolution to obtain the attention features of the generator CAM as the output of the generator CAM attention layer. The discriminator includes a discriminator CAM attention layer. The input feature map of the discriminator CAM attention layer is processed by an adaptive average pooling layer and then spatially mapped by a fully connected layer with InstanceNorm normalization to obtain the second average pooling weight. The input feature map of the generator CAM attention layer is processed by an adaptive max pooling layer and then spatially mapped by a fully connected layer to obtain the second max pooling weight. The second average pooling weight and the second max pooling weight are multiplied element-wise with the input feature map of the discriminator CAM attention layer, respectively. The two CAM response maps obtained are concatenated and then subjected to a 1×1 convolution transformation with InstanceNorm normalization to obtain the second discriminator attention feature map and the auxiliary discriminator feature map as the output of the discriminator CAM attention layer. The register consists of: one 1×1 convolutional layer ω and one per-patch fully connected layer. One MLP layer M, one fully connected layer v; The convolutional layer ω is used to generate a feature map Y, which represents the color gamut information of the deblurred image G(x) and the sharp image y; per-patch fully connected layer For image block segmentation, a feature map Y of size H×W×C is segmented into n image blocks of size P1×P2, where n = (H / P1)×(W / P2), H is a number divisible by P1, and W is a number divisible by P2, resulting in the segmented feature map. Use a per-patch fully connected layer From Y r Randomly select the i-th image patch Y from the middle i ; MLP layer M is used for Y i Perform high-level nonlinear mapping, then input it into a fully connected layer v for processing to obtain the registration feature map R = v(Y). i In the registration process, the blurred image G(x) and its corresponding sharp image y have the same image patch number i and use the same registration network weights. Total generator loss Registration loss Wherein, the registration feature map of the deblurred image is S=R(G(x)), the registration feature map of the clear image is T=R(y), ||·|| represents the matrix magnitude, and τ represents the adjustment coefficient; Generator's Generative Adversarial Loss p d (x) represents the probability distribution that the blurred image data x follows; c represents the value set by the generator to make the discriminator consider the deblurred image to be a clear image, c = 1; Generator identity content loss Where, μ G(x) Let μ represent the mean of G(x). y σ represents the mean of y; G(x) Let σ represent the variance of G(x). y σ represents the variance of y; G(x)y Let c1 and c2 represent the covariance between G(x) and y; c1 and c2 are constants used to stabilize the loss function, where c1 = (K1L). 2 c2 = (K2L) 2 L represents the range of pixels, K1 = 0.001, K2 = 0.

003.

2. The blind motion blur removal method for UAV remote sensing videos based on adversarial learning and contrastive learning according to claim 1, characterized in that, The generator is constructed using a feature pyramid network approach, consisting of a fifteen-layer structure: The first and fourteenth layers of the generator are pooling layers; The second layer of the generator is the generator conv1 convolutional layer, which is used to obtain the first depth feature, representing a convolutional kernel of 7 and padding of 0. It is normalized using the InstanceNorm2d function and activated using the ReLU function. The third and fourth layers of the generator are the generator conv2 convolutional layers, which are used to obtain the second depth features. The convolutional kernel size is 3, the stride is 2, the padding is 1, the InstanceNorm2d function is used for normalization, and the ReLU function is used for activation. The generator consists of layers 5 to 7 and layers 9 to 11 as residual blocks. Each residual block includes a boundary padding layer with a padding value of 1, a convolutional layer with a kernel size of 3, a normalized and ReLU activation layer, and a Dropout layer with a parameter of 0.

5. The eighth layer of the generator is the generator CAM attention layer; The twelfth and thirteenth layers of the generator are Deconv upsampling layers with an upsampling factor of 2. After each upsampling, a convolution process is performed with a kernel size of 3, a stride of 2, and padding of 1. Then, InstanceNorm normalization and ReLU activation function processing are performed. The third and thirteenth layers of the generator are connected by a 1×1 convolution jump, and the fourth and twelfth layers of the generator are connected by a 1×1 convolution jump. The fifteenth layer of the generator is a convolutional layer with a kernel of 7 and padding of 0. The output of the generator after being processed by the Tanh activation function is connected to the blurred image x in a skip connection to generate the deblurred image G(x).

3. The blind motion blur removal method for UAV remote sensing videos based on adversarial learning and contrastive learning according to claim 1, characterized in that, The discriminator consists of a six-layer structure: The first to third layers of the discriminator are discriminator conv1 convolutional layers, representing a kernel size of 4, a stride of 2, and padding of 2; The fourth layer of the discriminator is the discriminator CAM attention layer; The fifth and sixth layers of the discriminator are discriminator conv2 convolutional layers, representing a kernel size of 4, a stride of 1, and padding of 2. The fifth layer of the discriminator has InstanceNorm normalization and Leaky ReLU function, and the sixth layer of the discriminator has 1 channel, outputting the discrimination result. The discriminator is a multi-scale discriminator. Step 20 further includes: for each deblurred image G(x) input to the discriminator, first input the clear image y into the discriminator to obtain the discrimination result of scale one, then input the clear image y into the discriminator after performing a downsampling operation of scale 2 to obtain the discrimination result of scale two, and then perform a downsampling operation of scale 3 on the clear image y to obtain the discrimination result of scale three.

4. The blind motion blur removal method for UAV remote sensing videos based on adversarial learning and contrastive learning according to claim 1, characterized in that, The residual block uses the Leaky ReLU function as the activation function, and the slope of all Leaky ReLU functions in ALCLDeblur is 0.

2.

5. The blind motion blur removal method for UAV remote sensing videos based on adversarial learning and contrastive learning according to claim 1, characterized in that, Step 20 further includes: training a discriminator based on the discriminator loss, which includes the main discriminant generative adversarial loss. And auxiliary discrimination to generate adversarial loss p d (y) represents the probability distribution that the sharp image data y follows; p d (x) represents the probability distribution that the blurred image data x follows; This represents the expectation that a clear image data y sample will be classified as true; The expression represents the expectation that the deblurred image G(x) sample will be judged as false; a and b represent the encoding of the deblurred image G(x) sample and the encoding of the clear image data y sample, respectively; c represents the value set by the generator in order to make the discriminator judge the deblurred image G(x) as true, and b = c = 1 and a = 0 are set.

6. The blind motion blur removal method for UAV remote sensing videos based on adversarial learning and contrastive learning according to claim 1, characterized in that, Step 20 also includes introducing gradient regularization to regularize the generator, discriminator, and register.

7. The blind motion blur removal method for UAV remote sensing videos based on adversarial learning and contrastive learning according to claim 1, characterized in that, Step 20 also includes: first fixing the discriminator and register, training the generator and updating the parameters, then fixing the generator and training the discriminator and register network.

8. The blind motion blur removal method for UAV remote sensing videos based on adversarial learning and contrastive learning according to claim 1, characterized in that, Step 20 also includes: training the feature model ALCLDeblur using the Adam method, setting n=256, P1=P2=16, τ=0.007; the learning rate for the generator and register is set to 0.0001, and the learning rate for the discriminator is set to 0.0002.

9. A computer device, characterized in that, include: Memory, used to store instructions; as well as The processor is configured to invoke instructions stored in the memory to execute the blind motion blur removal method for UAV remote sensing videos based on adversarial learning and contrastive learning as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The device stores instructions that, when executed by a processor, perform the blind motion blur removal method for UAV remote sensing videos based on adversarial learning and contrastive learning as described in any one of claims 1-8.

Images