UAV image alignment method, electronic device and storage medium based on deep learning
Through the encoding-decoder network and channel attention mechanism based on deep learning, a fake perspective transformation label is generated, which solves the problem of poor alignment of drone images, and achieves high accuracy alignment on insufficient texture information and noise images , enhances the robustness of drone navigation.
Patent Information
- Application Number
- CN202210708805.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-22
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-06-22
AI Technical Summary
The existing drone image alignment methods are poorly robust on images with poor texture information, different lighting and noise, and are prone to matching point search errors, resulting in failure of perspective matrix calculations, especially when GPS fails, navigation errors accumulate, which makes it difficult to Accurately aligned.
Using a deep learning-based method, by building an encoding-decoder network, increasing the channel attention mechanism, designing residual modules, generating fake perspective transformation labels, using loss functions for regression prediction, enhancing the robustness of perspective transformation matrix, and through RANSAC filters outliers to improve alignment accuracy.
It realizes high robust alignment on insufficient texture information and noisy images, reduces alignment error, improves the accuracy of drone navigation, reduces dependence on manual annotation, and improves the alignment effect.
Smart Images

Figure CN115131414B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a drone image alignment method, electronic device and storage medium based on deep learning. Background Art
[0002] In recent years, with the development of drone technology, more and more drones can be operated to obtain a large number of drone images. Drones are easy to operate and have strong data collection capabilities. They have become the main platform for obtaining remote sensing images and have a wide range of application scenarios in civil, commercial and military applications. For drone navigation, GPS (Global Positioning System) plays a very important role, but the GPS system is very easy to fail in some scenarios, such as approaching obstacles or encountering interference. At this time, navigation completely relies on the drone's internal navigation unit to estimate its own state, but the state estimation will be affected by time drift, and the accumulated error makes the positioning estimate very easy to make mistakes and become unavailable.
[0003] Therefore, it is very important to match and align the images collected by the drone sensor with the built-in images. Existing image alignment methods all calculate the perspective matrix by finding the most matching target points on the collected image and the target image. However, this approach often results in errors in finding matching points on some images with poor texture information, resulting in the calculated perspective matrix being unable to align the images. It is also not robust to drone images with different lighting and occlusions, and will fail directly on some noisy images. Summary of the invention
[0004] In order to overcome the shortcomings of the above technologies, the purpose of the present invention is to provide a drone image alignment method, an electronic device and a storage medium based on deep learning, which obtains the position offset difference labels of the horizontal and vertical coordinates after the perspective transformation by forging the perspective transformation, and enhances the deep features by constructing an encoder-decoder network structure, designing a residual module, and adding a channel attention mechanism module to increase the expression ability of channel information and improve the robustness of the perspective transformation matrix prediction.
[0005] In order to achieve the above object, the technical solution adopted by the present invention is:
[0006] The UAV image alignment method based on deep learning includes the following steps:
[0007] Step 1: Build a training data set, perform data enhancement on the collected drone images, and expand the data scale;
[0008] Step 2: By A Randomly select points on the image and perform perturbations, calculate the perspective transformation relationship before and after the perturbation, and obtain the forged input image I according to the transformation relationship.B , and calculate the perspective transformation label PF based on the relationship before and after the disturbance;
[0009] Step 3: Design the encoder and decoder networks based on the residual network, and add a channel attention module to model the feature relationship between channels. Use the loss function to regress and predict the perspective transformation label PF in step 2, so that the trained model has the ability to predict perspective transformation;
[0010] Step 4: Use the trained model to align the image I ori Image of target drone I target to align.
[0011] The methods for expanding the data scale in step 1 include vertical flipping, horizontal flipping, reducing brightness, increasing brightness, unifying the size, and graying.
[0012] The process of forging a supervised data set in step 2 is as follows:
[0013] Step 21, in the original image I A A 128×128 square area is randomly generated, and the coordinate of the upper left corner is denoted as p ltc =(x pltc ,y pltc );
[0014] Step 22, perform random perturbation ρ on each of the four vertices, where ρ∈[-32,32]. The coordinate difference between the corresponding point pairs can be calculated based on the four point pairs before and after the random perturbation. The coordinate difference of the first vertex in the upper left corner is represented by Δu1=u1′-u1, and the following coordinate offset matrix can be obtained;
[0015]
[0016] Step 23: Convert the above matrix into a perspective transformation matrix H AB , and then the original image I A Use H BA =(H AB ) -1 Use the following formula to perform perspective transformation, and denote the transformed image as I B , from the original image I A and the transformed image I B p ltc An image block of size 128×128 is intercepted at the position as the input of the network;
[0017] Step 24, for image I A and I B Then the displacement of point p on the horizontal axis is Δx p =xp -x q , the corresponding displacement on the ordinate is Δy p =y p -y q , so in PF It can be expressed as the following formula:
[0018] and
[0019] Similarly, you need to ltc The vector blocks of the same size of 128×128 are intercepted at the position and concatenated to obtain a 128×128×2 feature vector as the supervision label.
[0020] The encoder network in step 3 is used to learn the feature relationship between the input images. It strictly follows the design pattern of the first four layers of the ResNet-50 network, and removes the fifth layer and the last fully connected layer. The output of each layer m is in the form of W m ×H m ×C m The feature map of m = {3, 4}, the corresponding feature map size is C m =2 6+m .
[0021] The decoder network in step 3 is used to restore the size of the feature map so that it has the same size as the input, so that the position offset of each pixel can be estimated, and the image features extracted by the encoder are restored. According to the spatial position information learned from the encoder, the PF of the position offset information is generated. Its structure is symmetrical with the encoder structure. The downsampling module in the encoder is replaced with deconvolution to achieve the purpose of restoring the size of the feature map. Similarly, for the n-th layer network, the output size of each layer feature is W n ×H n ×C n , where n = {1,....,4}, C n =2 10-n .
[0022] The feature attention module in step 3 includes two parts: a compression module and an excitation module. The compression module encodes the entire spatial feature on each channel into a global feature, which is implemented by a global average pooling operation. The excitation module obtains the global description feature from the compression operation, and uses the Sigmoid gating mechanism to learn the nonlinear relationship between each channel. The module is inserted into the last layer of each stage of the encoder and decoder, thereby improving the expressive power of the model.
[0023] The loss function used for regression in step 3 is smooth-l1, which is calculated as follows:
[0024]
[0025] In the above formula, * represents the horizontal coordinate or the vertical coordinate. represents the predicted PF, F AB* Indicates the PF of the label.
[0026] The alignment process in step 4 is as follows:
[0027] Step 41: The two drone images to be aligned are denoted as I and ori and I target , resize it using the method in step S1 and convert it into a grayscale image, then randomly generate a point p ltc And take the point as the upper left corner to crop a 128×128 image block as input;
[0028] Step 42, after inputting the image into the network, the predicted PF is obtained, that is, the offset of the horizontal and vertical coordinates Then use the offset of the horizontal and vertical coordinates to restore the offset coordinates;
[0029] Step 43, use RANSAC to filter out outliers, use the DLT algorithm to calculate the perspective transformation matrix H, and use the inverse matrix of H to transform I ori By performing a perspective transformation, the aligned image can be obtained.
[0030] An electronic device comprises a processor, a memory and a communication bus, wherein the processor and the memory communicate with each other via the communication bus;
[0031] Memory, used to store computer programs;
[0032] The processor is used to implement the above method steps when executing the program stored in the memory.
[0033] A computer-readable storage medium stores a computer program, and the computer program implements the above method steps when executed by a processor.
[0034] Beneficial effects of the present invention:
[0035] 1. The present invention adopts random point selection to arbitrarily generate sub-graphs on the image, and obtains the perspective transformation matrix of the label by randomly perturbing the four vertices of the sub-graph. When conducting supervised learning, theoretically, there can be an infinite number of image pairs with labeled pairs for learning, so that there is no need for tedious manual annotation and there can be a large number of training data sets to enable the network to have the ability to learn perspective transformation;
[0036] 2. When estimating the perspective matrix, the present invention uses PF (Perspective Random Field) for estimation and uses RANSAC for post-processing, which enhances the robustness against outliers and makes image alignment more accurate;
[0037] 3. The present invention adds a channel attention module to the network, automatically acquires the importance of each feature channel through the network self-learning method, and then enhances the useful features according to the importance and suppresses the features that are not very useful for the current task, thereby greatly improving the alignment effect with a smaller increase in parameters. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 is a schematic diagram of faking perspective transformation labels.
[0039] Figure 2 It is a schematic diagram of the deep learning network architecture and modules.
[0040] Figure 3 Figure 1 is a schematic diagram of using the trained model to align drone images. DETAILED DESCRIPTION
[0041] The present invention is further described in detail below in conjunction with embodiments.
[0042] like Figure 1 As shown, the UAV image alignment method based on deep learning includes the following steps:
[0043] Step S1, constructing a training data set. When expanding and enhancing the data set, we mainly used vertical flipping, horizontal flipping, reducing brightness, increasing brightness, unifying the size, and graying;
[0044] The vertical flipping and horizontal flipping are also called vertical mirroring and horizontal mirroring. Horizontal mirroring is to swap the left and right halves of the image with the vertical center axis of the image as the center axis, while vertical mirroring is to swap the upper and lower halves of the image with the horizontal center axis of the image as the center axis.
[0045] The brightness increase is to add the weight of the original image and an image with all pixel values 0, which can be described by the following formula: dst = src1·α+src2·β+γ, where src1 represents the original image, src2 represents the image with all pixel values 0, α represents the weight of the original image, β represents the weight of the image with pixel weight 0, and γ is a scalar added to the sum, which is equivalent to brightness adjustment. The α value must be greater than 1 to achieve the brightness increase effect.
[0046] The brightness reduction is to multiply the pixel values of the three channels RGB of the original image by the weight ω, wherein the value of ω needs to be less than 1. The smaller the value, the darker the image and the lower the brightness.
[0047] The unified size is to adjust all images to a unified size of 240×320 to facilitate subsequent network training and perspective transformation forgery.
[0048] The grayscale conversion is to convert the original input RGB three-channel image into a single-channel image, and the following psychological formula is required to complete the grayscale conversion of the image: Gray = 0.299*R+0.587*G+0.114*B, where R, G, and B represent the values of the three channels respectively.
[0049] Step S2, generating a forged transformed image and a forged perspective transformed field label.
[0050] Since directly labeling two images that need to be aligned requires a lot of material and manpower, we choose to forge the transformed image to enable the network to learn perspective transformation, thereby completing the alignment task; the steps of constructing the forged transformed image are as follows: Figure 3 As shown;
[0051] First, in the original image I A A 128×128 square sub-image is randomly generated on the image, and the coordinates of the upper left corner are Correspondingly, the coordinates of the other three vertices are
[0052] Subsequently, each of these four vertices is randomly perturbed by ρ, where ρ∈[-32,32]. It is worth noting here that the horizontal coordinate x needs to ensure that 32<x<224, and the vertical coordinate y needs to ensure that 32<y<80, so that these four randomly perturbed points will not fall outside the image.
[0053] Then, the coordinate difference between the corresponding point pairs can be calculated based on the four point pairs before and after the random perturbation. The coordinate difference of the first vertex in the upper left corner is represented by Δu1=u1′-u1, and the following coordinate offset matrix can be obtained:
[0054]
[0055] This matrix can be easily converted to a perspective transformation matrix H AB , then the original image I A Use H BA =(H AB ) -1 Use the following formula to perform perspective transformation:
[0056]
[0057] Where [u,v] represents the coordinates before transformation, [u′,v′] represents the coordinates after transformation, and the transformed image is denoted as I B .
[0058] After obtaining the perspective transformed image, we need to A and the transformed image I B Medium ltc Image blocks of the same size of 128×128 are intercepted at the position, and they are spliced to obtain a 128×128×2 feature vector as the input of the network.
[0059] Similarly, in order to obtain the perspective transformation supervision information PF (perspective transformation field), it is necessary to calculate the pixel offset in the horizontal and vertical directions, using W and H to represent the image I respectively. A and I B width and height, then for point p, the offset on the horizontal axis is Δx p =x p -x q , the corresponding displacement on the ordinate is Δy p =y p -y q , so in PF It can be expressed as the following formula:
[0060] and
[0061] Similarly, you need to ltc Vector blocks of the same size of 128×128 are intercepted at the position, and they are concatenated to obtain a 128×128×2 feature vector as a supervision label.
[0062] Step S3, construct a perspective matrix learning network. Figure 2 As shown, the deep learning network module includes an encoder, a decoder, and an attention mechanism channel fusion module.
[0063] The encoder uses ResNet-50 as the backbone network. The first four layers of the network strictly follow the design pattern of ResNet-50. The fifth layer and the last fully connected layer are removed. The fifth layer is removed because too much downsampling will make the feature map too small and the information loss is serious. The fully connected layer is removed to be able to connect to the decoder. The encoder takes the above H×W×2 grayscale image as input and outputs it in the form of W in stage m. m ×H m ×C m The feature map of m = {3, 4}, the corresponding feature map size is Cm =2 6+m The encoder module uniformly samples the local receptive field at the pixel position through a series of convolutional residual modules to capture the spatial correlation of the image to form the final feature map as the input of the decoder.
[0064] The decoder needs to restore the image features extracted by the encoder and generate a PF of position offset information based on the spatial position information learned from the encoder. Therefore, it is necessary to gradually restore the size of the input image from the downsampled feature map. To achieve this goal, deconvolution is used in the upsampling process, and bilinear interpolation is not used. Deconvolution has better information learning capabilities. Drawing on the design of ResNet-50, the decoder has a symmetrical structure similar to that of the encoder. It also has a four-layer network. Each layer has the same residual module as the encoder, except that the convolution module used for downsampling is replaced by a deconvolution module. For a deconvolution module, the resolution of the input vector needs to be increased. Similar to the residual network, for the input feature vector X, its shape is W×H×C, and there are two information streams to decode it. Two 2×2 deconvolution kernels are used, and a step size of 2 is used to achieve upsampling. For the first information stream, the number of convolution kernels is the same as the channel of the input vector to obtain a feature map x1 of size W×H×C. For the second information stream, the number of convolution kernels is halved to obtain The final output result of the feature map of size x2 is out = F(x1) + x2. Similarly, for the n-th layer network, the output size of each layer feature is W n ×H n ×C n , where n = {1,....,4}, C n =2 10-n .
[0065] After the decoder and encoder are built, it is necessary to recover the predicted PF with the same shape as the label PF from the last layer of features of the decoder, that is, the shape of the vector of the final prediction result should also be H×W×2. Two consecutive convolution kernels of the shape of 1×1×512 and 1×1×2 are used in the last layer to achieve this goal. Unlike the currently popular encoder-decoder model, no redundant jump connection structure is added between the encoder and decoder. This is because the image distortion before and after the perspective transformation changes greatly, and the supplementary information is meaningless, and even reduces the alignment performance of the network.
[0066] As mentioned above, the attention mechanism fusion module is embedded in the encoder and decoder modules. By modeling the correlation between feature channels, important features are expressed more significantly, thereby improving the expressiveness of the model. The attention module includes two parts: compression and excitation.
[0067] (1) Compression mechanism. Compression is to encode the entire spatial features on each channel into a global feature, which is achieved by using a global average pooling operation. It can be expressed by the following formula:
[0068]
[0069] In the above formula, F sq Indicates the compression module, u c represents a channel feature of the input feature, H, W is the size of the feature map, (i, j) represents the feature value of the i, j position in the feature map. c Represents a real eigenvalue obtained for this channel.
[0070] (2) Excitation mechanism. To obtain global descriptive features from compression operations, a mechanism is needed to learn the nonlinear relationship between channels. Here, a Sigmoid gating mechanism is used, which can be expressed as the following formula:
[0071] s=F ex (z,W)=σ(g(z,W))=σ(W2ReLU(W1z))
[0072] Where σ represents the Sigmoid function, In order to reduce the complexity of the model and improve the generalization ability, two consecutive fully connected layers are used. The first fully connected layer reduces the number of channels, and the second fully connected layer is used to restore the original number of channels. r is the dimension reduction coefficient, and ReLU is the activation function. Finally, the activation value of each channel learned is multiplied by the original feature u
[0073]
[0074] The whole operation learns the weight coefficients of each channel, which can be regarded as a channel attention mechanism, denoted as Atten(·). Finally, this module is integrated into the last convolution layer of each residual module of the encoder and decoder, which can be expressed as the following operation Where x represents the output of the last layer of the residual module. This enhances the learning of feature correlation.
[0075] Establish the loss function of deep learning to calculate PF features;
[0076] For this type of intensive prediction problem, the more commonly used loss function is the l2 loss function, which is used to calculate the Euclidean distance between the prediction result and the true label. However, for this problem, since outliers have a large impact on the loss, the l2 loss is not suitable for use on this problem. Therefore, smooth-l1 is used as the loss function for this problem, that is,
[0077]
[0078] In the above formula, * represents the horizontal coordinate or the vertical coordinate. represents the predicted PF, F AB* Indicates the PF of the label.
[0079] PyTorch is used to build this network, and Adam is used as the optimizer. The parameters in the Adam optimizer are the default values, that is, β1=0.9, β2=0.99, momentum=0.9, and the initial learning rate is 1×10 -4 As the initial learning rate, the learning rate is reduced to 1 / 10 after every 40 epochs. The size of each batch is 32, and a total of 120 epochs are trained.
[0080] Step S4, aligning the two drone images according to the trained network;
[0081] S4a, the two drone images to be aligned are denoted as I ori and I target , resize it using the method in step S1 and convert it into a grayscale image, then randomly generate a point p ltc , and cut a 128×128 image block with this point as the upper left corner. The same operation needs to be performed on the two images, and then they are spliced into a 128×128×2 vector as input.
[0082] S4b, after inputting the image into the network, the predicted PF is obtained, that is, the offset of the horizontal and vertical coordinates Δpi = (Δxp i ,Δyp i ), then we use the offset of the horizontal and vertical coordinates to restore the offset coordinates, that is, where pi and qi represent
[0083]
[0084] S4c, after obtaining the original coordinates and offset coordinates, first use the RANSAC (random sampling consensus) algorithm to filter out some non-conforming outliers, and then use the DLT (direct linear transformation) algorithm to find the most suitable perspective transformation matrix from these point pairs, and solve h in the equation group shown in the following formula:
[0085]
[0086] Where h is the obtained perspective transformation matrix, whose size is 3×3.
[0087] S4d, invert the perspective transformation matrix h obtained in S3c and record it as h -1 , use h -1 to I ori Use the following formula to perform perspective transformation, and I ori Aligned to I target .
[0088]
[0089] The present invention consults a large number of drone images and analyzes that in drone images, the contextual relationship of the global image content is crucial for reasoning and obtaining the perspective transformation matrix. However, traditional image alignment techniques, such as SIFT, SURF, ORB, etc., first find the most matching feature points based on the features, and then obtain the perspective transformation matrix, which is easy to fail in images with fewer texture features and noise. Some key point matching methods based on deep learning must mark the corresponding points and require a large data scale, which undoubtedly brings huge cost overhead. Some perspective matrix estimation methods directly regress the coordinate difference of four points in the image and estimate the perspective matrix through the four points sampled. This approach will be greatly affected by the outliers in the prediction results, and the correlation between channels is ignored in these methods, so that the best prediction results cannot be achieved. Therefore, the present invention proposes to enhance the data set and enable the network to have the ability to learn perspective transformation by forging data labels. According to this method, an infinite amount of data can be generated, which reduces the threshold and difficulty of training the network.
[0090] The present invention adopts PF (perspective transform field) to estimate the perspective transformation matrix. Compared with the estimation using only four points, all points on the image are sampled, which increases the number of selectable points. The RANSAC method is used for post-processing to reduce the impact of the model on outliers, thereby improving the accuracy of the estimated perspective matrix and making the alignment effect of the model more accurate.
[0091] Based on the residual network, the present invention adds a channel attention mechanism, because it is necessary to learn the relationship between the images before and after the perspective transformation, because the image has been deformed and distorted, the information on the spatial transformation is not generalizable, therefore, the connection and relationship between channels are very important, so a compression-excitation module is introduced to model the correlation relationship between channels, strengthen the features of important channels, weaken the features of unimportant channels, and explicitly describe the interdependence between feature channels, so that the alignment effect is greatly improved within the range of a small increase in the number of parameters.
[0092] The smooth-l1 loss function used in the present invention overcomes the problem that the l1 loss function is not differentiable at 0, which may affect convergence, and overcomes the disadvantage that the l2 loss function is too sensitive to outlier eigenvalues. With a considerable convergence speed, an excellent alignment effect can still be achieved.
[0093] The present invention also includes an electronic device, including a memory and a processor, wherein the memory is used to store various computer program instructions, and the processor is used to execute the computer program instructions to complete all or part of the above steps; the electronic device can communicate with one or more external devices, and can also communicate with one or more devices that enable users to interact with the electronic device, and / or communicate with any device that enables the electronic device to communicate with one or more other computing devices. The electronic device can also communicate with one or more networks (such as local area networks, wide area networks and / or public networks) through a network adapter.
[0094] The present invention also includes a computer-readable storage medium storing a computer program that can be executed by a processor. The computer-readable storage medium may include but is not limited to magnetic storage devices, optical disks, digital versatile disks, smart cards and flash memory devices. In addition, the readable storage medium described in the present invention can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" includes but is not limited to wireless channels and various other media (and / or storage media) that can store, contain and / or carry codes and / or instructions and / or data.
[0095] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. A UAV image alignment method based on deep learning, characterized in that: The steps include: Step 1: Build a training data set, perform data enhancement on the collected drone images, and expand the data scale; Step 2: By A Randomly select points on the image and perform perturbations, calculate the perspective transformation relationship before and after the perturbation, and obtain the input image I according to the transformation relationship B , and calculate the perspective transformation label PF based on the relationship before and after the disturbance; Step 3: Design the encoder and decoder networks based on the residual network, and add a channel attention module to model the feature relationship between channels. Use the loss function to regress and predict the perspective transformation label PF in step 2, so that the trained model has the ability to predict perspective transformation; Step 4: Use the trained model to align the image I ori Image of target drone I target Perform alignment; The alignment process in step 4 is as follows: Step 41: The two drone images to be aligned are denoted as I and ori and I target , resize it using the method in step S1 and convert it into a grayscale image, then randomly generate a point p ltc And take the point as the upper left corner to crop a 128×128 image block as input; Step 42, after inputting the image into the network, the predicted PF is obtained, that is, the offset of the horizontal and vertical coordinates Then use the offset of the horizontal and vertical coordinates to restore the offset coordinates; Step 43, use RANSAC to filter out outliers, use the DLT algorithm to calculate the perspective transformation matrix H, and use the inverse matrix of H to transform I ori By performing a perspective transformation, the aligned image can be obtained.
2. The deep learning-based drone image alignment method according to claim 1, characterized in that: The methods for expanding the data scale in step 1 include vertical flipping, horizontal flipping, reducing brightness, increasing brightness, unifying the size, and graying.
3. The deep learning-based drone image alignment method according to claim 1, characterized in that: The process of supervised dataset in step 2 is as follows: Step 21, in the original image I A A 128×128 square area is randomly generated, and the coordinates of the upper left corner are Step 22, perform random perturbation ρ on each of the four vertices, where ρ∈[-32,32]. The coordinate difference between the corresponding point pairs can be calculated based on the four point pairs before and after the random perturbation. The coordinate difference of the first vertex in the upper left corner is represented by Δu1=u1′-u1, and the following coordinate offset matrix can be obtained; Step 23: Convert the above matrix into a perspective transformation matrix H AB , and then the original image I A Use H BA =(H AB ) -1 Use the following formula to perform perspective transformation, and denote the transformed image as I B , from the original image I A and the transformed image I B p ltc An image block of size 128×128 is intercepted at the position as the input of the network; Step 24, for image I A and I B Then the displacement of point p on the horizontal axis is Δx p =x p -x q , the corresponding displacement on the ordinate is Δy p =y p -y q , so in PF It is expressed as the following formula: and Similarly, you need to ltc The vector blocks of the same size of 128×128 are intercepted at the position and concatenated to obtain a 128×128×2 feature vector as the supervision label.
4. The deep learning-based drone image alignment method according to claim 1, characterized in that: The encoder network in step 3 is used to learn the feature relationship between the input images. It strictly follows the design pattern of the first four layers of the ResNet-50 network, and removes the fifth layer and the last fully connected layer. The output of each layer m is in the form of W m ×H m ×C m The feature map of m = {3, 4}, the corresponding feature map size is C m =2 6+m 。 5. The deep learning-based drone image alignment method according to claim 1, characterized in that: The decoder network in step 3 is used to restore the size of the feature map so that it has the same size as the input, so that the position offset of each pixel can be estimated, and the image features extracted by the encoder are restored. According to the spatial position information learned from the encoder, the PF of the position offset information is generated. Its structure is symmetrical with the encoder structure. The downsampling module in the encoder is replaced with deconvolution to achieve the purpose of restoring the size of the feature map. Similarly, for the n-th layer network, the output size of each layer feature is W n ×H n ×C n , where n = {1,....,4}, C n =2 10-n .
6. The deep learning-based drone image alignment method according to claim 1, characterized in that: The feature attention module in step 3 includes two parts: a compression module and an excitation module. The compression module encodes the entire spatial feature on each channel into a global feature, which is implemented by a global average pooling operation. The excitation module obtains the global description feature from the compression operation, and uses the Sigmoid gating mechanism to learn the nonlinear relationship between each channel. The module is inserted into the last layer of each stage of the encoder and decoder, thereby improving the expressive power of the model.
7. The deep learning-based drone image alignment method according to claim 1, characterized in that: The loss function used for regression in step 3 is smooth-l1, which is calculated as follows: In the above formula, * represents the horizontal coordinate or the vertical coordinate. represents the predicted PF, F AB* Indicates the PF of the label.
8. An electronic device, characterized in that: It includes a processor, a memory and a communication bus, wherein the processor and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement any one of the method steps of claims 1-7 when executing the program stored in the memory.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1 to 7 are implemented.
