Unmanned aerial vehicle inspection image jitter correction method and device and storage medium

Abstract

The embodiment of the invention discloses an unmanned aerial vehicle inspection image jitter correction method and device and a storage medium. The method comprises the following steps: constructing an image jitter model; acquiring an inspection dithering image according to the image dithering model; training an image jitter correction model by using the inspection jitter image and the inspection clear image to obtain a trained image jitter correction model; inputting a to-be-corrected inspection jitter image into a first encoder, and outputting a first shared submerged space vector; performing domain migration on the first shared submerged space vector to obtain a first shared submerged space vector after domain migration; inputting the first shared submerged space vector after domain migration into a first Transform self-attention module, and outputting a first feature vector; and inputting the first feature vector into a first decoder, and outputting a target correction image. According to the method, the jitter of the inspection image of the unmanned aerial vehicle is corrected, and a clear and stable corrected image can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle inspection image processing, and in particular to a method, device and storage medium for unmanned aerial vehicle inspection image jitter correction. Background Art

[0002] With the continuous development of drone technology, more and more inspection tasks are carried out by drones. During the drone inspection process, image jitter often occurs due to the influence of the flight environment (such as wind, terrain changes, mechanical vibration, etc.). This jitter can cause the image to be blurred, distorted, and even difficult to identify key details. Through jitter correction, the clarity and stability of the image can be significantly improved, and the overall image quality can be improved. Therefore, it is essential to correct the jitter artifacts in drone inspection images.

[0003] At present, although self-supervised image shake correction technologies such as CycleGAN have made some progress in image quality restoration, they still have some defects. CycleGAN relies on generative adversarial networks (GANs), and its main limitation is that the training process of the generator and discriminator is unstable, which may lead to loss of details or residual artifacts in the generated images. In addition, CycleGAN usually requires a large amount of training data and computing resources when dealing with complex scenes and diverse artifacts. However, in drone power grid patrol missions, the scale of original images is often limited, which limits the training effect of CycleGAN, resulting in unsatisfactory image shake correction effects in drone inspections. Summary of the invention

[0004] Based on this, it is necessary to propose a method, device and storage medium for UAV inspection image jitter correction to address the above problems.

[0005] First aspect

[0006] A method for correcting jitter of an unmanned aerial vehicle inspection image, the method comprising:

[0007] Build a UAV image jitter model;

[0008] According to the drone image jitter model, obtaining a drone inspection jitter image;

[0009] Using the jittered image of the drone inspection and the clear image of the drone inspection, the image jitter correction model is trained to obtain a trained image jitter correction model; wherein the image jitter correction model includes an encoder module, a Transformer self-attention module and a decoder module, the encoder module includes a first encoder, a second encoder and a third encoder, the Transformer self-attention module includes a first Transformer self-attention module and a second Transformer self-attention module, and the decoder module includes a first decoder and a second decoder;

[0010] Inputting the unmanned aerial vehicle inspection jitter image to be corrected into a first encoder, and outputting a first shared latent space vector;

[0011] Performing domain migration on the first shared latent space vector to obtain a first shared latent space vector after domain migration;

[0012] Inputting the first shared latent space vector after the domain migration into a first Transformer self-attention module, and outputting a first feature vector;

[0013] The first feature vector is input into a first decoder, and a target corrected image is output.

[0014] Preferably, the step of training an image jitter correction model using the jittered image and the clear image to obtain a trained image jitter correction model specifically includes:

[0015] The jittered image of the drone inspection is input into the first encoder and the second encoder at the same time, and the first shared latent space vector and the second shared latent space vector are output respectively; wherein the first shared latent space vector is the output vector corresponding to the first encoder, and the second shared latent space vector is the output vector corresponding to the second encoder;

[0016] Inputting the clear image of the drone inspection into the third encoder and outputting the third shared latent space vector;

[0017] Summing the second shared latent space vector and the third shared latent space vector to obtain a fourth shared latent space vector;

[0018] Performing domain migration on the first shared latent space vector and the fourth shared latent space vector to obtain the first shared latent space vector and the fourth shared latent space vector after domain migration;

[0019] Inputting the first shared latent space vector after the domain migration into a first Transformer self-attention module, and outputting a first feature vector;

[0020] Inputting the fourth shared latent space vector after the domain migration into a second Transformer self-attention module, and outputting a second feature vector;

[0021] Inputting the first feature vector into a first decoder, and outputting a clear image corresponding to the jittered image of the unmanned aerial vehicle inspection to be corrected;

[0022] Input the second feature vector into a second decoder, and output a jittered image corresponding to the clear image of the unmanned aerial vehicle inspection to be corrected;

[0023] The above process is repeated until the loss value is less than the set threshold, and the cycle is stopped to obtain the trained image shake correction model.

[0024] Preferably, the loss functions adopted by the image shake correction model include: a cross-cycle consistency loss function, a feature adversarial loss function, a background consistency loss function and a semantic consistency loss function.

[0025] Preferably, the drone image jitter model simulates displacement effect and blur effect through a motion blur kernel.

[0026] Preferably, obtaining the drone inspection jitter image according to the drone image jitter model specifically includes:

[0027] Get clear images of drone inspections;

[0028] Summing the drone inspection clear image and the displacement vector to obtain the drone inspection displacement image;

[0029] The UAV inspection displacement image is convolved with the fuzzy matrix to obtain the UAV inspection jitter image.

[0030] Preferably, the encoder module comprises a convolutional layer, a Relu activation function and a batch normalization layer.

[0031] Preferably, the Transformer self-attention module includes: a linear layer, a multi-head attention layer and a normalization layer.

[0032] Preferably, the decoder module comprises a convolutional layer, a Relu activation function, a Sigmoid activation function and an upsampling layer.

[0033] Second aspect

[0034] A drone inspection image jitter correction device, the device comprising:

[0035] Image jitter model building module, used to build a UAV image jitter model;

[0036] A patrol jitter image acquisition module, used to acquire a drone patrol jitter image according to the drone image jitter model;

[0037] An image jitter correction model training module is used to train the image jitter correction model using the jittered image and the clear image of the drone inspection to obtain a trained image jitter correction model; wherein the image jitter correction model includes an encoder module, a Transformer self-attention module and a decoder module, the encoder module includes a first encoder, a second encoder and a third encoder, the Transformer self-attention module includes a first Transformer self-attention module and a second Transformer self-attention module, and the decoder module includes a first decoder and a second decoder;

[0038] A first shared latent space vector output module, used for inputting the unmanned aerial vehicle inspection jitter image to be corrected into a first encoder, and outputting a first shared latent space vector;

[0039] A domain migration module, configured to perform domain migration on the first shared latent space vector to obtain a first shared latent space vector after domain migration;

[0040] A first feature vector output module, configured to input the first shared latent space vector after domain migration into a first Transformer self-attention module, and output a first feature vector;

[0041] The target correction image output module is used to input the first feature vector into a first decoder and output a target correction image.

[0042] The third aspect

[0043] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the method described in the first aspect.

[0044] The embodiments of the present invention have the following beneficial effects:

[0045] This paper proposes an unsupervised drone inspection image jitter correction method based on the Transformer architecture self-attention mechanism. This method can effectively remove the image jitter caused by drone movement, restore the details and structure of the image, and achieve efficient correction through unsupervised learning strategies without relying on paired data, ultimately obtaining clear, stable, and high-quality image output. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] in:

[0048] Figure 1 Flow chart of a method for correcting jitter of an unmanned aerial vehicle inspection image in one embodiment;

[0049] Figure 2 is a structural diagram of an image shake correction model in one embodiment;

[0050] Figure 3 An example diagram of a data set in an embodiment;

[0051] Figure 4 A schematic diagram of training results of an image shake correction model in one embodiment;

[0052] Figure 5 This is a diagram showing the effect of image jitter correction in one embodiment;

[0053] Figure 6 The figure is a structural block diagram of a UAV inspection image jitter correction device in one embodiment. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] like Figure 1 As shown, in one embodiment, a method for correcting image jitter during inspection by a drone is provided. The method specifically includes steps 1100 to 1700:

[0056] Step 1100: construct a drone image jitter model.

[0057] In this embodiment, in order to effectively correct the jitter problem in the drone inspection image, it is first necessary to establish a mathematical model of image jitter to obtain a large amount of data.

[0058] In some examples, the drone image jitter model simulates displacement effects and blur effects through a motion blur kernel.

[0059] Image jitter is usually caused by factors such as airflow and vibration during the flight of the drone. To simplify the model building process, a simple motion blur kernel is used in this embodiment to simulate these effects.

[0060] Step 1200: Acquire a drone inspection jitter image according to a drone image jitter model.

[0061] In this model, we regard the jitter effect of an image as a combination of pixel displacement and blur in the image. The motion blur effect of an image can be simulated by applying a simple motion blur kernel, which generates a smear effect on the image similar to that caused by speed changes or vibrations during the flight of a drone, effectively simulating the image blur caused by the motion of a drone.

[0062] In step 1200, obtaining the jittered image of the drone inspection according to the drone image jitter model may further include steps 1201 to 1203:

[0063] Step 1201, obtaining a clear image of the drone inspection.

[0064] A clear drone inspection image means an image without jitter artifacts.

[0065] Step 1202: sum the drone inspection clear image and the displacement vector to obtain the drone inspection displacement image.

[0066] Displacement vector, used to represent the displacement effect caused by image jitter.

[0067] Step 1203, convolve the UAV inspection displacement image with the fuzzy matrix to obtain the UAV inspection jitter image.

[0068] Blur matrix, used to represent the blur effect caused by image jitter.

[0069] Exemplarily, let the clear image of the drone inspection be B(p), where p is the pixel position of the clear image of the drone inspection. Specifically, p = [a, b], a∈[0, H], b∈[0, W], where H and W are the height and width of the image, respectively, and the image contains three channels.

[0070] During the flight of the drone, due to the interference of the external environment, the image may have jitter artifacts. In order to simulate this phenomenon, let the displacement vector generated by the image jitter be D(p), which disturbs the original position of each pixel in the image, causing the image pixel to shift. At this time, the jitter artifact of the image can be expressed as:

[0071] I'(p)=B(p+D(p))

[0072] Among them, I'(p) is the displacement image of the UAV inspection, which means the jitter image considering only the displacement effect. The displacement vector D(p) represents the displacement of the image pixels caused by the jitter, which may cause the misalignment or overlap of some details in the image.

[0073] However, in addition to the displacement effect, jitter also induces a blurring effect. This blurring effect can be achieved by using a motion blur kernel M. b (p) to simulate. Let M b (p) is the motion blur kernel, which will produce a blur effect similar to smear in the image. Therefore, combining the displacement effect and blur effect, the image affected by the jitter artifact can be expressed as:

[0074] I(p)=I′(p)*M b (p) = B (p + D (p)) * M b (p)

[0075] Here, “*” indicates the convolution operation. I(p) is the final image affected by the jitter artifact. This model describes the jitter artifacts that appear in the image due to the displacement and blurring effects during the flight of the drone. It can be seen that the jitter artifact effect not only causes the pixel position in the image to shift, but also further reduces the clarity and detail of the image due to the blurring effect.

[0076] In the actual drone inspection process, jitter artifacts may cause different degrees of blur and displacement in the entire image range, thus affecting the validity of the image. The jitter artifact correction task is to restore the clear image B(p) from the given image I(p) affected by jitter artifacts.

[0077] Through the above steps, a large number of jittered images are generated using simulation methods. Specifically, we simulate different drone motion trajectories and shooting conditions to generate images with jitter effects, while retaining the corresponding non-jittered images as controls. In this way, we can train the model on large-scale data and improve its generalization ability.

[0078] Step 1300: train an image jitter correction model using the jittered image and the clear image of the drone inspection to obtain a trained image jitter correction model.

[0079] After clarifying the model of image jitter, the next step is to build a multi-stage network structure. Our network design is divided into two stages. In the first stage, the network removes the jitter effect in the image through the self-attention mechanism and generates a preliminary corrected image. In the second stage, the network further restores the details and texture of the image to improve the visual quality of the corrected image. The entire network structure is based on the Transformer architecture, which uses its powerful global information capture ability to effectively perform jitter correction.

[0080] Figure 2 Figure 1 is a structural diagram of an image shake correction model in one embodiment. Our goal is to learn an abstract intermediate representation from the shaken image input and reconstruct clear observations. The complete network structure diagram is shown in Figure 1. Figure 2 As shown, the image shake correction model includes an encoder module, a Transformer self-attention module and a decoder module.

[0081] In some examples, the encoder module includes a first encoder, a second encoder, and a third encoder. The encoder module includes a convolutional layer, a Relu activation function, and a batch normalization layer.

[0082] In some examples, the Transformer self-attention module includes a first Transformer self-attention module and a second Transformer self-attention module. The Transformer self-attention module includes: a linear layer, a multi-head attention layer, and a normalization layer.

[0083] In some examples, the decoder module includes a first decoder and a second decoder. The decoder module includes a convolutional layer, a Relu activation function, a Sigmoid activation function, and an upsampling layer.

[0084] In step 1300, the image jitter correction model is trained using the jittered image and the clear image of the drone inspection to obtain the trained image jitter correction model, which may further include steps 1301 to 1309:

[0085] Step 1301, input the jittered image of the drone inspection into the first encoder and the second encoder at the same time, and output the first shared latent space vector and the second shared latent space vector respectively.

[0086] In some examples, the first shared latent space vector is an output vector corresponding to the first encoder, and the second shared latent space vector is an output vector corresponding to the second encoder.

[0087] The first encoder corresponds Figure 2 W x , the second encoder corresponds to Figure 2 W x N .

[0088] Step 1302: input the clear image of the drone inspection into the third encoder, and output the third shared latent space vector.

[0089] The third encoder corresponds Figure 2 W y .

[0090] Step 1303 , sum the second shared latent space vector and the third shared latent space vector to obtain a fourth shared latent space vector.

[0091] Step 1304 , domain-transfer the first shared latent space vector and the fourth shared latent space vector to obtain the first shared latent space vector and the fourth shared latent space vector after domain transfer.

[0092] Step 1305: input the first shared latent space vector after domain migration into the first Transformer self-attention module, and output a first feature vector.

[0093] The first Transformer self-attention module corresponds to Figure 2 Transformer self-attention module in the clear domain.

[0094] Step 1306: Input the fourth shared latent space vector after domain migration into the second Transformer self-attention module, and output a second feature vector.

[0095] The second Transformer self-attention module corresponds to Figure 2 Transformer self-attention module in the dithered noise domain.

[0096] Step 1307: input the first feature vector into the first decoder, and output a clear image corresponding to the jittered image of the unmanned aerial vehicle inspection to be corrected.

[0097] The first decoder corresponds to Figure 2 R y .

[0098] Step 1308: input the second feature vector into the second decoder, and output the jittered image corresponding to the clear image of the drone inspection to be corrected.

[0099] The second decoder corresponds to Figure 2 R x .

[0100] Step 1309, loop the above process until the loss value is less than the set threshold, stop the loop, and obtain the trained image shake correction model.

[0101] For example, to some extent, unsupervised image generation can be viewed as a specific domain transfer problem, i.e., from the dithered noise domain to the clear domain.

[0102] For unsupervised domain transfer, samples (x, y) are drawn from the marginal distributions Px(x) and Py(y). In order to infer the joint distribution from the marginal samples, a shared latent space assumption is proposed, that is, there exists a shared latent code z in the shared latent space z, from which the two images can be recovered. Given a sample (x, y) from the joint distribution, the process is expressed as follows:

[0103] z=Wx(x)=Wy(y)

[0104] x=Rx(z)=Ry(z)

[0105] Wherein, W represents the encoding function and R represents the decoding function.

[0106] The key step is how to implement this shared latent space assumption. To this end, an effective strategy is to share high-level representations through a shared weight encoder that samples features from a uniform distribution. However, the latent representation only contains semantic features and is not suitable for image retrieval, which can lead to domain shift, blurred details and inconsistent background of the restored image. Therefore, we try to learn a more generalized representation from the input that contains richer texture and semantic features, that is, an invariant representation. To achieve this goal, an adversarial domain adaptation algorithm based on discrete representation learning and a self-supervised constraint model is introduced in this embodiment.

[0107] The goal of discrete representation is to compute a latent code z from the input, where z contains as much texture and semantic information as possible. To this end, we use two autoencoders to model {Wx, Rx} and {Wy, Ry} respectively. Given any unpaired sample 9x, y), where x∈X and y∈Y represent noisy samples and clean samples from different domains, respectively, the above formula is expressed as: x =Wx(x) and z y =Wy(y). In addition, image restoration can be expressed as F x→y (x) = Ry(z x ). However, since the dithering noise is always attached to the high-frequency signal, the size and type of the noise will vary, and it is difficult to directly reconstruct a clean image, which requires powerful domain generators and discriminators. Therefore, we introduce a disentangled representation into our architecture.

[0108] Decoupled representation: For dithered samples x, use an additional dither noise encoder to model the varying noise levels and types. The self-reconstruction formula is where zx=Wx(x), Assume that the shared space {z x , z y The hidden code z in}∈Z x and z y Following the same distribution, similar to image translation, unsupervised image restoration can be divided into two stages: forward translation and backward reconstruction.

[0109] Forward translation: We first start from the image pair (x, y) and the additional noise encoding Extract the representation {z x , z y}. Recovery and degradation can be expressed as:

[0110]

[0111] in, Represents the clear picture after restoration, Represents a degraded dither noise sample. Indicates the sum operation on the channel. Rx and Ry are considered as domain-specific generators.

[0112] Backward reconstruction: After completing the first translation, and To achieve reconstruction:

[0113]

[0114] in, and represents the reconstructed input.

[0115] We flatten the feature map into a sequence representation, and in the process of backward image reconstruction, the feature vector at each position is regarded as an element in the sequence and input into the encoder of the Transformer. The self-attention mechanism in the Transformer architecture can capture long-range dependencies in sequence data.

[0116] Self-attention mechanism: The core of the self-attention mechanism is to achieve attention to each element by adjusting the degree of association between each element (such as attention weight). Given an input feature sequence x∈X, the calculation process of the self-attention mechanism is:

[0117]

[0118] Among them, Q represents query, K represents key, and V represents value, and they form a matrix, which is obtained by linear transformation of input features:

[0119] Q=XW Q

[0120] K=XW K

[0121] V=XW V

[0122] Here, W Q , W K and W V is the learned weight matrix, and is the dimension of the feature vector. By calculating the attention weights and weighted summing them, the self-attention mechanism can dynamically aggregate feature information at different positions in the image, thereby achieving jitter correction.

[0123] Multi-head attention mechanism: In order to better capture different patterns in features, we introduce a multi-head attention mechanism. The input feature sequence is divided into multiple subspaces, and each subspace performs self-attention operations. The calculation process of multi-head attention is:

[0124] MultiHead(Q, K, V) = [head 1 ,head 2 ,…,head h ]W 0

[0125] Among them, the calculation of each attention head is:

[0126] head i =Attention(QW Qi ,KW Ki ,VW Vi )#(14)

[0127] Here, W Qi , W Ki , W Vi is the weight matrix of the ith attention head, W 0 is the output weight matrix. Through the multi-head attention mechanism, the model can learn different jitter correction modes from different feature subspaces to further improve the correction effect.

[0128] Position encoding: The Transformer architecture itself does not contain position information, so position encoding is added to the input sequence to capture the spatial relationship in the image. Position encoding is calculated using sine and cosine functions:

[0129]

[0130] Among them, pos represents the position in the sequence, i represents the index of the feature dimension, and d modelis the dimension of the input feature. After the position encoding is added to the input sequence, it can effectively preserve the spatial information of the image and contribute to more accurate jitter correction.

[0131] Decoder and image reconstruction:

[0132] Corrected feature map F corrected The image is reconstructed through the decoder network. The decoder consists of multiple deconvolution layers, each of which gradually restores the spatial size of the image and finally outputs the corrected image I corrected ∈R H×W×3 The decoder is implemented as follows:

[0133] I corrected =Decoder(F corrected )

[0134] During the decoding process, the decoder network gradually upsamples the rectified feature map to the original image size to restore the spatial details of the image. This process usually includes several deconvolution layers and batch normalization layers to gradually restore the high-frequency information of the image and repair the jitter blur in the image.

[0135] The decoder network consists of multiple deconvolution layers, each of which is responsible for gradually increasing the spatial resolution of the feature map. Specifically, each layer of the decoder can be expressed as:

[0136] F deconv =ConvTranspose(F corrected ,W deconv )

[0137] Among them, ConvTranspose represents the deconvolution operation, W deconv is the corresponding weight matrix. The decoder gradually restores the spatial resolution of the image through multiple deconvolution layers and converts the high-level feature map into the final rectified image.

[0138] We designed corresponding loss functions for the specific goals of the multi-stage network. The loss function of the first stage aims to minimize the displacement difference between the corrected image and the original image, thereby removing the effect of jitter. The loss function of the second stage includes two levels: pixel level and feature level, ensuring that the restored image is visually highly consistent with the clear image. The feature level loss ensures accurate restoration of details and textures by comparing high-level feature maps.

[0139] In some examples, the loss functions adopted by the image shake correction model include: cross-cycle consistency loss function, feature adversarial loss function, background consistency loss function, and semantic consistency loss function.

[0140] Exemplarily, the process of designing the loss function is as follows:

[0141] To enforce the constraints, we add a cross-cycle consistency loss L for the X and Y domains cc :

[0142]

[0143] Adversarial Domain Adaptation Loss: Another key factor is how to transform the latent representation z x and z y Embed into a shared space. Inspired by unsupervised domain adaptation, we achieve this through adversarial learning instead of sharing weight encoders. Our goal is to promote the representation of the input to follow a similar distribution while preserving the rich texture and semantic information of the input. Therefore, a representation discriminator D is used in our architecture. r We use this feature adversarial loss It is expressed as:

[0144]

[0145] Self-supervised loss: In the absence of effective supervision information, relying solely on the discriminability of feature regions will inevitably lead to domain bias. In order to speed up the convergence of the algorithm and improve the robustness of the algorithm, this project will also use context-based context consistency models and semantic consistency models to achieve effective control of the algorithm.

[0146] The goal of BCM is to make the converted image have the same background as the input image. This method is also used to automatically reconstruct images. The algorithm uses fuzzy operators such as Gaussian blur kernels and guided filters to smooth the input and output images, thereby achieving restrictions on the reconstructed image. Therefore, the background consistency loss L BC can be formulated as:

[0147]

[0148] Among them B σ (·) denotes the Gaussian blur operator with blur kernel σ, λ σ is a hyperparameter used to balance the errors of different Gaussian blur levels. χ and denote the original input and the translated output respectively, i.e. and According to the image denoising experiment, we set λ σ They are set to {0.25, 0.5, 1.0} corresponding to σ = {5, 9, 15} respectively.

[0149] In addition, inspired by the perceptual loss, the deep features of the pre-trained model only contain semantic information, which are noise-free or almost noise-free. Therefore, unlike the general feature loss, which aims to restore more delicate image texture details through the similarity between shallow features, we only extract the deep features of the damaged and restored images as semantic representations to maintain consistency, which is called the semantic consistency loss L SC . It can be formulated as:

[0150]

[0151] where φ(·) represents the lth th Layer features. In this example, we used the conv5-1 layer of the VGG-19 network pre-trained on ImageNet.

[0152] Before training the image shake correction model, we constructed a dataset such as Figure 3 As shown in Figure 1, the dataset includes clear images of drone inspection and jittered images of drone inspection.

[0153] The process of building the dataset is as follows:

[0154] Data generation: To meet the needs of unsupervised training, we used synthetic images to generate pairs of images affected by jitter. Specifically, we used an algorithm that simulates the jitter effect and applied different degrees of jitter perturbations to clear images to generate a set of images affected by jitter. This method of synthesizing data not only increases the diversity of the data, but also ensures that the scale of the dataset is large enough.

[0155] Dataset size: We generated 738 training images and 184 test images. The images include clear images and images affected by shaking. The training images are used for the model training process, while the test images are used to evaluate the performance of the model.

[0156] Dataset examples: Some examples of images included in the dataset are as follows Figure 3 As shown, the clear images and the blurred images are shown in different columns. The first and third columns are images affected by shaking, and the second and fourth columns are clear images.

[0157] During model training and validation, we took the following steps:

[0158] The constructed multi-stage network is trained end-to-end. During the training process, we use the loss function designed above to optimize the network parameters. After the training is completed, we verify the performance of the model on the test set and evaluate the shake correction effect of the model by comparing the images before and after correction. The experimental results show that our method can effectively remove image shake and significantly improve image quality.

[0159] We adopted an end-to-end training method and optimized the overall loss function. During the training process, the batch-size was set to 4 and a total of 1000 epochs were performed to gradually adjust the model parameters so that the model can better remove the jitter effect in the image.

[0160] Step 1400: input the unmanned aerial vehicle inspection jitter image to be corrected into a first encoder, and output a first shared latent space vector.

[0161] The first encoder corresponds Figure 2 W x .

[0162] Step 1500 , domain-transfer the first shared latent space vector to obtain the first shared latent space vector after domain transfer.

[0163] Step 1600 , input the first shared latent space vector after domain migration into the first Transformer self-attention module, and output a first feature vector.

[0164] Step 1700: input the first feature vector into a first decoder, and output a target corrected image.

[0165] The first decoder corresponds to Figure 2 R y .

[0166] In this embodiment, the process of outputting the target corrected image refers to the process of training the image shake correction model.

[0167] Quantitative evaluation of model performance: The trained model was evaluated on the test dataset, mainly using the following indicators:

[0168] SSIM (Structural Similarity Index): The best SSIM of the model on the test dataset is 0.923, showing the superior performance of the model in preserving the image structure.

[0169] FID (Frechet Inception Distance): The best FID value is 48.9443, indicating that the generated image is close to the real image in quality.

[0170] KID (Kernel Inception Distance): The best KID value is 0.0009, which shows that the quality of the generated images is superior and the distribution is close to the real images.

[0171] SWD (Squared Wasserstein Distance): SWD values ​​range from 118.8725 to 216.7240. Although there are some fluctuations, the overall trend is positive, indicating that the distribution of generated images is close to that of real images.

[0172] SE (Similarity Error): The best SE value is 6.7365, which means that the similarity error between the generated image and the real image is small and the quality of the generated image is high.

[0173] STD (Standard Deviation): The best STD value is 27.3272, indicating that the quality stability of the images generated by the model is good.

[0174] SF (Similarity Factor): The best SF value is 8.2642, which shows that the generated image has a high similarity with the real image.

[0175] CORR (Correlation): The best CORR value is 0.9789, indicating that the correlation between the generated image and the real image is high, and the image generated by the model can well reflect the characteristics of the real image.

[0176] MMD (Maximum Mean Discrepancy): The best MMD value is 0.0318, which indicates that the model performs well in generating image distribution.

[0177] Qualitative results of model performance: Figure 4 FIG. 1 is a schematic diagram of training results of an image shake correction model in one embodiment. Figure 4 The qualitative experimental results of the model on the test dataset are shown:

[0178] The first column shows a drone image affected by jitter, with obvious blur and jitter traces.

[0179] The second column shows the de-shaking results after being processed by the method proposed in this embodiment. These images show a significant improvement in clarity, and the background details are effectively restored.

[0180] The third column shows a reference image without jitter interference. By comparing the second column with the third column, it can be intuitively observed that the method of this embodiment has a significant effect in removing the jitter influence.

[0181] Figure 5The figure is an effect diagram of image jitter correction in one embodiment. The prior art usually relies on paired image data for jitter correction, that is, the model is trained through pre-prepared jitter and non-jitter image pairs. The limitation of this method is that it is difficult to obtain paired image data, and it can only be effectively applied in specific scenarios. In contrast, the method of this embodiment uses an unsupervised learning strategy to perform image jitter correction, getting rid of the dependence on paired image data. This difference makes this method more widely applicable, especially in actual inspection scenarios where data acquisition is difficult, and the generalization ability of the model can be significantly improved through autonomous learning.

[0182] Some current methods only use unsupervised generative adversarial networks (GANs) for image shake correction. Although they can generate high-quality corrected images to a certain extent, due to the lack of the ability to capture global information, the model may be insufficient when dealing with complex and diverse scenes. This invention introduces the self-attention mechanism of the Transformer architecture and combines it with the generation ability of GAN to effectively capture the global information in the image, thereby achieving more accurate shake correction in a wider range of scenarios. This difference significantly improves the quality of the corrected image, enabling it to cope with more complex scenes and environments in practical applications. The difference in results is shown in the figure below. Figure 5 .

[0183] Introducing the self-attention mechanism of the Transformer architecture: Most of the current image jitter correction technologies are mainly based on traditional convolutional neural networks (CNNs), which rely on local perception capabilities for correction and are difficult to effectively capture global information. The present embodiment adopts the self-attention mechanism of the Transformer architecture, which can capture global dependencies in images over a larger range, thereby more accurately identifying and correcting complex jitter phenomena. This makes the present embodiment more robust and accurate when dealing with image jitter problems in complex scenes. Traditional image jitter correction methods usually rely on a large amount of annotated training data, which is not only costly to obtain, but also has great limitations. The present embodiment adopts an unsupervised learning method to reduce the dependence on annotated data by automatically learning image jitter patterns from unannotated data. This feature not only reduces the cost of data acquisition, but also expands the applicability of the method, enabling it to be effectively applied in more practical scenarios.

[0184] This embodiment proposes an unsupervised drone inspection image jitter correction method based on the Transformer architecture self-attention mechanism. This method can effectively remove the image jitter caused by drone movement, restore the details and structure of the image, and achieve efficient correction through unsupervised learning strategies without relying on paired data, ultimately obtaining clear, stable, and high-quality image output.

[0185] The present application also discloses a drone inspection image jitter correction device, such as Figure 6 As shown, the device comprises:

[0186] The image jitter model building module 6100 is used to build a drone image jitter model.

[0187] The inspection jitter image acquisition module 6200 is used to acquire the drone inspection jitter image according to the drone image jitter model.

[0188] The image jitter correction model training module 6300 is used to train the image jitter correction model using the jittered images and clear images of the drone inspection to obtain a trained image jitter correction model.

[0189] In some examples, the image shake correction model includes an encoder module, a Transformer self-attention module and a decoder module, the encoder module includes a first encoder, a second encoder and a third encoder, the Transformer self-attention module includes a first Transformer self-attention module and a second Transformer self-attention module, and the decoder module includes a first decoder and a second decoder.

[0190] The first shared latent space vector output module 6400 is used to input the unmanned aerial vehicle inspection jitter image to be corrected into the first encoder and output the first shared latent space vector.

[0191] The domain migration module 6500 is used to perform domain migration on the first shared latent space vector to obtain the first shared latent space vector after domain migration.

[0192] The first feature vector output module 6600 is used to input the first shared latent space vector after domain migration into the first Transformer self-attention module and output the first feature vector.

[0193] The target corrected image output module 6700 is used to input the first feature vector into the first decoder and output the target corrected image.

[0194] In one embodiment, the image jitter correction model training module 6300 is specifically used to: input the jittered image of the drone inspection into the first encoder and the second encoder at the same time, and output the first shared latent space vector and the second shared latent space vector respectively; wherein the first shared latent space vector is the output vector corresponding to the first encoder, and the second shared latent space vector is the output vector corresponding to the second encoder; input the clear image of the drone inspection into the third encoder, and output the third shared latent space vector; sum the second shared latent space vector and the third shared latent space vector to obtain the fourth shared latent space vector; domain transfer the first shared latent space vector and the fourth shared latent space vector to obtain the domain transfer The first shared latent space vector and the fourth shared latent space vector after domain migration are input into the first Transformer self-attention module to output the first feature vector; the fourth shared latent space vector after domain migration is input into the second Transformer self-attention module to output the second feature vector; the first feature vector is input into the first decoder to output the clear image corresponding to the jittered image of the drone inspection to be corrected; the second feature vector is input into the second decoder to output the jittered image corresponding to the clear image of the drone inspection to be corrected; the above process is repeated until the loss value is less than the set threshold, the cycle is stopped, and a trained image jitter correction model is obtained.

[0195] In one embodiment, the loss functions used by the image shake correction model include: a cross-cycle consistency loss function, a feature adversarial loss function, a background consistency loss function, and a semantic consistency loss function.

[0196] In one embodiment, the drone image jitter model simulates displacement effects and blur effects through a motion blur kernel.

[0197] In one embodiment, the inspection jitter image acquisition module 6200 is specifically used to: obtain a clear image of the drone inspection; sum the clear image of the drone inspection with the displacement vector to obtain a displacement image of the drone inspection; perform a convolution operation on the displacement image of the drone inspection and the fuzzy matrix to obtain a jittered image of the drone inspection.

[0198] In one embodiment, the encoder module includes a convolutional layer, a ReLU activation function, and a batch normalization layer.

[0199] In one embodiment, the Transformer self-attention module includes: a linear layer, a multi-head attention layer, and a normalization layer.

[0200] In one embodiment, the decoder module includes a convolutional layer, a Relu activation function, a Sigmoid activation function, and an upsampling layer.

[0201] The above description of the embodiment of the apparatus for correcting image jitter during inspection by a drone is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiment of the apparatus for correcting image jitter during inspection by a drone of the present invention, please refer to the description of the method embodiment of the present invention for understanding.

[0202] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method in any of the above embodiments.

[0203] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0204] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0205] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for correcting image jitter during inspection by an unmanned aerial vehicle, characterized in that: The method comprises: Build a UAV image jitter model; According to the drone image jitter model, obtaining a drone inspection jitter image; Using the jittered image of the drone inspection and the clear image of the drone inspection, the image jitter correction model is trained to obtain a trained image jitter correction model; wherein the image jitter correction model includes an encoder module, a Transformer self-attention module and a decoder module, the encoder module includes a first encoder, a second encoder and a third encoder, the Transformer self-attention module includes a first Transformer self-attention module and a second Transformer self-attention module, and the decoder module includes a first decoder and a second decoder; Inputting the unmanned aerial vehicle inspection jitter image to be corrected into a first encoder, and outputting a first shared latent space vector; Performing domain migration on the first shared latent space vector to obtain a first shared latent space vector after domain migration; Inputting the first shared latent space vector after the domain migration into a first Transformer self-attention module, and outputting a first feature vector; The first feature vector is input into a first decoder, and a target corrected image is output.

2. The method for correcting image jitter during inspection by a drone according to claim 1, characterized in that: The method of training the image jitter correction model by using the jittered image and the clear image of the drone inspection to obtain the trained image jitter correction model specifically includes: The jittered image of the drone inspection is input into the first encoder and the second encoder at the same time, and the first shared latent space vector and the second shared latent space vector are output respectively; wherein the first shared latent space vector is the output vector corresponding to the first encoder, and the second shared latent space vector is the output vector corresponding to the second encoder; Inputting the clear image of the drone inspection into the third encoder and outputting the third shared latent space vector; Summing the second shared latent space vector and the third shared latent space vector to obtain a fourth shared latent space vector; Performing domain migration on the first shared latent space vector and the fourth shared latent space vector to obtain the first shared latent space vector and the fourth shared latent space vector after domain migration; Inputting the first shared latent space vector after the domain migration into a first Transformer self-attention module, and outputting a first feature vector; Inputting the fourth shared latent space vector after the domain migration into a second Transformer self-attention module, and outputting a second feature vector; Inputting the first feature vector into a first decoder, and outputting a clear image corresponding to the jittered image of the unmanned aerial vehicle inspection to be corrected; Input the second feature vector into a second decoder, and output a jittered image corresponding to the clear image of the unmanned aerial vehicle inspection to be corrected; The above process is repeated until the loss value is less than the set threshold, and the cycle is stopped to obtain the trained image shake correction model.

3. The method for correcting image jitter during inspection by a drone according to claim 1, characterized in that: The loss functions used by the image shake correction model include: a cross-cycle consistency loss function, a feature adversarial loss function, a background consistency loss function, and a semantic consistency loss function.

4. The method for correcting image jitter during inspection by a drone according to claim 1, characterized in that: The UAV image jitter model simulates the displacement effect and blur effect through the motion blur kernel.

5. The method for correcting image jitter during inspection by an unmanned aerial vehicle according to claim 4, characterized in that: The step of obtaining the jitter image of the drone inspection according to the drone image jitter model specifically includes: Get clear images of drone inspections; Summing the drone inspection clear image and the displacement vector to obtain the drone inspection displacement image; The UAV inspection displacement image is convolved with the fuzzy matrix to obtain the UAV inspection jitter image.

6. The method for correcting image jitter during inspection by an unmanned aerial vehicle according to claim 1, characterized in that: The encoder module includes a convolutional layer, a Relu activation function, and a batch normalization layer.

7. The method for correcting image jitter during inspection by an unmanned aerial vehicle according to claim 1, characterized in that: The Transformer self-attention module includes: a linear layer, a multi-head attention layer and a normalization layer.

8. The method for correcting image jitter during inspection by an unmanned aerial vehicle according to claim 1, characterized in that: The decoder module includes a convolutional layer, a Relu activation function, a Sigmoid activation function and an upsampling layer.

9. A drone inspection image jitter correction device, characterized in that: The device comprises: Image jitter model building module, used to build a UAV image jitter model; A patrol jitter image acquisition module, used to acquire a drone patrol jitter image according to the drone image jitter model; An image jitter correction model training module is used to train the image jitter correction model using the jittered image and the clear image of the drone inspection to obtain a trained image jitter correction model; wherein the image jitter correction model includes an encoder module, a Transformer self-attention module and a decoder module, the encoder module includes a first encoder, a second encoder and a third encoder, the Transformer self-attention module includes a first Transformer self-attention module and a second Transformer self-attention module, and the decoder module includes a first decoder and a second decoder; A first shared latent space vector output module, used for inputting the unmanned aerial vehicle inspection jitter image to be corrected into a first encoder, and outputting a first shared latent space vector; A domain migration module, configured to perform domain migration on the first shared latent space vector to obtain a first shared latent space vector after domain migration; A first feature vector output module, configured to input the first shared latent space vector after domain migration into a first Transformer self-attention module, and output a first feature vector; The target correction image output module is used to input the first feature vector into a first decoder and output a target correction image.

10. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 8.

Images