Glass curtain wall image splicing method for unmanned aerial vehicle shooting

Abstract

The present application relates to a glass curtain wall image splicing method for unmanned aerial vehicle shooting, comprising: shooting a certain number of glass curtain wall images through an unmanned aerial vehicle, forming a data set, and dividing it into a training set, a validation set and a test set; constructing an image splicing network and training the image splicing network using the training set to obtain the image splicing network after training; the image splicing network comprises a deformation prediction network and an image synthesis network; the network model performance of the image splicing network after training is verified using the validation samples in the validation set, and the optimal image splicing network is screened out; in the test set, two images of the glass curtain wall position at least partially overlapping are selected and input into the optimal image splicing network, and the splicing result is obtained. The advantage lies in: high-precision alignment is realized through the deformation prediction network, and high-quality synthesis is realized through the image synthesis network, so that the glass curtain wall image splicing can be ensured to be the most natural to a certain extent.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for stitching together images of glass curtain walls taken by drones. Background Technology

[0002] To meet the dual demands of aesthetics and functionality in the modern construction industry, glass curtain walls have emerged as a prominent type of curtain wall due to their unique visual effects, convenient construction methods, and high plasticity, and have been widely used in the facades of high-rise buildings.

[0003] Drones, with their flexibility and cost-effectiveness, have become the preferred tool for acquiring images of building facades. However, accurately assessing the condition of glass curtain walls places high demands on image quality. Although drone-mounted cameras offer high resolution, their field of view is relatively limited. When the drone is far from the building, image texture is not clear enough, affecting detection accuracy; conversely, when the distance is close, the field of view is still limited. Utilizing image stitching technology to fuse multiple images into a single high-resolution, wide-field-of-view image is a reasonable choice, effectively improving the accuracy and reliability of the detection.

[0004] However, compared to other natural images, glass curtain walls present challenges in image stitching due to their indistinct feature differences and strong texture repetition. Current deep learning algorithm research largely focuses on stitching natural images with prominent features, with relatively little research specifically targeting image stitching for architectural glass curtain walls. Exploring technologies suitable for glass curtain wall image stitching is of crucial academic significance and also shows great potential in practical applications. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method for stitching glass curtain wall images taken by drones that can achieve high-precision alignment and high-quality synthesis, so as to make the stitching more natural.

[0006] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a method for stitching together glass curtain wall images captured by drones, characterized by the following steps:

[0007] Step 1: Take a certain number of images of glass curtain walls using a drone to form a dataset, and divide the dataset into a training set, a validation set, and a test set;

[0008] Step 2: Construct an image stitching network and train it multiple times using the training set to obtain the trained image stitching network; the image stitching network includes a deformation prediction network and an image synthesis network;

[0009] Two glass curtain wall images are selected from the training set, and one of them is denoted as target image I.t Another image of the glass curtain wall is designated as reference image I. r I t and I r The images are the same size, I t and I r From the same building, and I t and I r The locations of the glass curtain walls shown in the image overlap at least partially, using I t and I r The specific process of training an image stitching network once is as follows:

[0010] Step 2-1: Transfer the target image I t and reference image I r The image is input into the deformation prediction network to obtain the first image I. wt Second image I wr First image I wt The corresponding mask image M t And the second image I wr The corresponding mask image M r M t and M r The pixel value in the valid area is 1, and the pixel value in the invalid area is 0;

[0011] For I respectively t and I r Perform edge extraction, and then extract the I values ​​from the edge-extracted edges. t and I r The input is fed into the deformation prediction network to obtain the first edge map E. wt Second edge map E wr ;

[0012] According to the first image I wt Second image I wr Calculate the first loss function and based on the first edge map E. wt and the second edge graph E wr Calculate the second loss function, combine the first loss function and the second loss function to obtain the first total loss function, and use the first total loss function to update the network parameters of the deformation prediction network, thus completing one training of the deformation prediction network;

[0013] Step 2-2: Transfer the first image I wt Second image I wr The image is input into an image synthesis network to obtain the final stitched image S;

[0014] The second total loss function is calculated based on the final stitched image S, and the network parameters of the image synthesis network are updated using the second total loss function. This completes one training of the image synthesis network and yields a trained image stitching network.

[0015] The image synthesis network consists of two encoders with shared weights and one decoder. The specific steps for obtaining the stitched image S are as follows:

[0016] Step 2-2-1: Transfer the first image I wt Second image I wr Each input is fed into an encoder to obtain I. wt The corresponding feature map and I wr The corresponding feature map, calculate I wt The corresponding feature map and I wr The difference between the corresponding feature maps is used to make a skip connection between the difference and the input of the decoder to obtain a single-channel mask γ;

[0017] Step 2-2-2: Calculate I according to the following formula. wr The weight M in the final stitched image wr and I wt The weight M in the final stitched image wt ;

[0018] M wr =(M r -P)+P×γ

[0019] M wt =(M t -P)+P×(1-γ)

[0020] Where P represents I wt The corresponding mask image M t with I wr The corresponding mask image M r The mask for the overlapping area, where the pixel value of the overlapping area is 1 and the pixel value of the non-overlapping area is 0;

[0021] Step 2-2-3: Calculate the final stitched image S according to the following formula;

[0022] S = I wr ×M wr +I wt ×M wt

[0023] Step 3: Use the validation samples in the validation set to validate the performance of the trained image stitching network model and select the optimal image stitching network.

[0024] Step 4: Select two images from the test set that have at least partial overlap at the glass curtain wall locations. The two images selected in the test set are from the same building. The buildings corresponding to the test samples, the validation samples, and the training samples in the training set are all different. Input them into the optimal image stitching network obtained in Step 3 to obtain the stitching result.

[0025] As an improvement, step 2-1 is specifically as follows:

[0026] The deformation prediction network includes an optical flow calculation module and a homography estimation module. The homography estimation module includes a regression network. The specific processing procedure of the deformation prediction network is as follows:

[0027] Step 2-1-1: Transfer the target image I t and reference image I r The initial optical flow is obtained by inputting the data into the optical flow calculation module.

[0028] Step 2-1-2: Scale the initial optical flow to 1 / N of the initial optical flow, where N is an integer greater than 1, to obtain the final optical flow;

[0029] Step 2-1-3: Transfer the target image I t The four corner points are denoted as p1, p2, p3, and p4. The final optical flow is input into the regression network of the homography estimation module to predict the pixel offsets of the four corner points p1, p2, p3, and p4 relative to their original positions. The target image I is then calculated based on the predicted pixel offsets. t The four corner points in reference image I r The coordinates of the target corresponding points in the coordinate system are used, and then the direct linear transformation (DLT) algorithm is used to solve for the implicit homography matrix H between the four corner points p1, p2, p3 and p4 and their corresponding target corresponding points.

[0030] Step 2-1-4: Transfer the target image I t Multiplying it by the homography matrix H yields the target image I. t The deformed image will be the target image I. t The deformed image is placed into a designated image area to obtain the first image I. wt And generate the first image I wt The corresponding mask image M t ;

[0031] Step 2-1-5: Place reference image I r Place it in a designated image area of ​​the same size, referencing image I. r Placement with target image I t The positions of the two images at least partially overlap to obtain the second image I. wrAnd generate a second image I wr The corresponding mask image M r .

[0032] Preferably, the optical flow calculation module in step 2-1-1 adopts the GMFlow network.

[0033] Preferably, the first total loss function L total1 The calculation formula is:

[0034] L total1 =ω1L1+ω2L2

[0035] Where L1 is the first loss function, L2 is the second loss function, ω1 is the weight of the first loss function, ω2 is the weight of the second loss function, ω1∈(0,1), ω2∈(0,1), and ω1+ω2=1;

[0036]

[0037] Where (i,j) are the coordinates of the pixel, I is the total number of x-coordinates of the pixels, and J is the total number of y-coordinates of the pixels. Indicate I wr The pixel value of the pixel with upper coordinate (i,j). For deformation operation, That is, I wt , For I wt The pixel value of the pixel with coordinates (i,j) is P. (i,j) Let H be the pixel value of the pixel at coordinate (i,j) on P. -1 The inverse transformation of the homography matrix H, For I r With H -1 The resulting product is the pixel value of the pixel at coordinate (i,j) in the image. P′ is the inverse transform of P. ′(i,j) Let be the pixel value of the pixel at coordinate (i,j) on P′;

[0038]

[0039] in, For E wr The pixel value of the pixel with upper coordinate (i,j). For E wt The pixel value of the pixel with coordinates (i,j) is given by ∈, where ∈ is a very small constant to prevent the denominator in the L2 calculation formula from being 0.

[0040] Preferably, in step 2-2, the second total loss function L total2 The calculation formula is:

[0041] L total2 =ω3L3+ω4L4

[0042] Where L3 is the third loss function, L4 is the fourth loss function, ω3 is the weight of the third loss function, ω4 is the weight of the fourth loss function, ω3∈(0,1), ω4∈(0,1), and ω3+ω4=1;

[0043] L3=||S*M br -I wr *M br ||1+||S*M bt -I wt *M bt ||1

[0044] ||.||1 is for calculating the first norm, M br To make M t The result of dilating the inverted region is the same as M. r The boundary mask obtained by multiplication; M bt To make M r The result of dilating the inverted region is the same as M. t The boundary mask obtained by multiplication;

[0045] L4 = L D +L s

[0046]

[0047] For M wr The pixel value of the pixel with upper coordinate (i,j). For M wr The pixel value of the pixel with coordinates (i+1,j) is given by D, where D is the luminance difference, and D = (I... wr -I wt ) 2 D (i,i) Let D be the pixel value of the pixel with coordinates (i,j) on D. (i +1,j) Let be the pixel value of the pixel at coordinate (i+1,j) on D. For M wr The pixel value of the pixel with coordinates (i,j+1), D (i,j+1) Let be the pixel value of the pixel at coordinate (i,j+1) on D;

[0048]

[0049] S (i,j) Let S be the pixel value of the pixel with coordinates (i,j) on S. (i+1,j)Let S be the pixel value of the pixel with coordinates (i+1,j) on S. (i,j+1) Let be the pixel value of the pixel at coordinate (i,j+1) on S.

[0050] Compared with existing technologies, the advantages of this invention are as follows: By setting up two independent networks—a deformation prediction network and an image synthesis network—the deformation prediction network uses extracted dense optical flow as prior information, inputting it into a regression network to predict corner displacement. Then, the DLT algorithm is used to solve the homography matrix, while simultaneously introducing pixel and edge consistency loss, significantly improving global alignment accuracy to achieve high-precision alignment. Furthermore, the image synthesis network learns the feature differences between aligned images to obtain optimal fusion weights, and combines boundary and smoothing losses for optimization, effectively eliminating stitching seams and achieving high-quality synthesis. Therefore, this method can, to a certain extent, ensure the most natural image stitching for glass curtain walls. Attached Figure Description

[0051] Figure 1 This is a block diagram of the deformation prediction network in an embodiment of the present invention;

[0052] Figure 2 This is a block diagram of the image synthesis network in an embodiment of the present invention;

[0053] Figure 3 This is a preliminary alignment diagram of the first image and the second image in an embodiment of the present invention;

[0054] Figure 4 The boundary mask M in this embodiment of the invention br and boundary mask M bt A schematic diagram. Detailed Implementation

[0055] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0056] The image stitching method for glass curtain walls captured by drones in this embodiment includes the following steps:

[0057] Step 1: Take a certain number of images of glass curtain walls using a drone to form a dataset, and divide the dataset into a training set, a validation set, and a test set;

[0058] In this embodiment, the dataset is divided into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0059] Step 2: Construct an image stitching network and train it multiple times using the training set to obtain the trained image stitching network; the image stitching network includes a deformation prediction network and an image synthesis network;

[0060] Two glass curtain wall images are selected from the training set, and one of them is denoted as target image I. t Another image of the glass curtain wall is designated as reference image I. r I t and I r The images are the same size, I t and I r From the same building, and I t and I r The locations of the glass curtain walls shown in the image overlap at least partially, using I t and I r The specific process of training an image stitching network once is as follows:

[0061] Step 2-1: Transfer the target image I t and reference image I r The image is input into the deformation prediction network to obtain the first image I. wt Second image I wr First image I wt The corresponding mask image M t And the second image I wr The corresponding mask image M r M t and M r The pixel value in the valid area is 1, and the pixel value in the invalid area is 0;

[0062] like Figure 1 As shown, for I respectively t and I r Edge extraction is performed to obtain E t (I after edge extraction) t ) and E r (I after edge extraction) r ), and respectively E t and E r The input is fed into the deformation prediction network to obtain the first edge map E. wt Second edge map E wr E t After being input into the deformation prediction network, i.e., using and obtaining I wt Using the same processing method, the first edge map E is obtained. wt Similarly, E r After being input into the deformation prediction network, i.e., using and obtaining I wr Using the same processing method, the second edge map E is obtained. wr ;

[0063] In this embodiment, edge extraction is performed based on the MuGE algorithm. The MuGE algorithm is an existing technology and will not be described in detail here. The core mechanism of MuGE is to integrate this control signal into the deep network to adjust the model's sensitivity to image features. When the threshold α is biased towards 0, the model outputs a relatively coarse outline edge. When the threshold α is biased towards 1, it outputs an edge map containing more fine texture. This controllability allows MuGE to flexibly generate edges that meet different needs and is easy to adjust.

[0064] According to the first image I wt Second image O wr Calculate the first loss function and based on the first edge map E. wt and the second edge graph E wr Calculate the second loss function, combine the first loss function and the second loss function to obtain the first total loss function, and use the first total loss function to update the network parameters of the deformation prediction network, thus completing one training of the deformation prediction network;

[0065] Step 2-2: Transfer the first image I wt Second image I wr The image is input into an image synthesis network to obtain the final stitched image S;

[0066] The second total loss function is calculated based on the final stitched image S, and the network parameters of the image synthesis network are updated using the second total loss function. This completes one training of the image synthesis network and yields a trained image stitching network.

[0067] Step 3: Use the validation samples in the validation set to validate the performance of the trained image stitching network model and select the optimal image stitching network.

[0068] In the validation set, the validation samples used for each validation are from the same building; in order to improve the image stitching effect, the buildings corresponding to the validation samples are different from the buildings corresponding to the training samples in the training set.

[0069] Step 4: Select two images from the test set that have at least partial overlap at the glass curtain wall locations, and input them into the optimal image stitching network obtained in Step 3 to obtain the stitching result.

[0070] The two images selected in the test set are from the same building. Similarly, the buildings corresponding to the test samples, the validation samples, and the training samples in the training set are all different.

[0071] like Figure 1As shown, the deformation prediction network in this embodiment includes an optical flow calculation module and a homography estimation module. The homography estimation module includes a regression network. The specific processing procedure of the deformation prediction network is as follows:

[0072] Step 2-1-1: Transfer the target image I t and reference image I r The initial optical flow is obtained by inputting the data into the optical flow calculation module.

[0073] The optical flow calculation module in this embodiment uses the GMFlow network, which is an existing technology and will not be elaborated further here. The core process of the GMFlow network is to first use a convolutional network to extract image features, then use a Transformer structure to enhance the discriminative and context-aware capabilities of the features, effectively capturing long-distance dependencies. Subsequently, by calculating global feature correlations and combining them with a Softmax layer, matching is performed directly across the entire feature map to obtain a pixel-level probability distribution of correspondences. The initial optical flow is obtained using weighted average coordinates, and a self-attention layer is introduced to propagate high-confidence matching flow information to occluded or unmatched areas. This global matching strategy allows GMFlow to handle large displacements more efficiently, making it suitable for scenarios with large displacements, such as glass curtain walls.

[0074] Step 2-1-2: Scale the initial optical flow to 1 / N of the initial optical flow, where N is an integer greater than 1, to obtain the final optical flow;

[0075] The value of N can be determined based on experiments or experience. In this embodiment, N = 16. The purpose of this step is to reduce the amount of calculation and to better predict global motion.

[0076] Step 2-1-3: Transfer the target image I t The four corner points are denoted as p1, p2, p3, and p4. The final optical flow is input into the regression network of the homography estimation module to predict the pixel offsets of the four corner points p1, p2, p3, and p4 relative to their original positions. The target image I is then calculated based on the predicted pixel offsets. t The four corner points in reference image I r The coordinates of the target corresponding points in the coordinate system are used, and then the direct linear transformation (DLT) algorithm is used to solve for the implicit homography matrix H between the four corner points p1, p2, p3 and p4 and their corresponding target corresponding points.

[0077] Assume the coordinates of p1 are (x1, y1), p2 are (x2, y2), p3 are (x3, y3), and p4 are (x4, y4). p1 is located in reference image I. rThe coordinates of the target point p1′ in the coordinate system are (x1+△x1, y1+△y1), where (△x1, △y1) is the pixel offset of p1 relative to its original position. Similarly, four sets of corresponding point pairs (p1, p1′), (p2, p2′), (p3, p3′), and (p4, p4′) are obtained in the same way as above. Then, the homography matrix H is calculated according to the existing homography transformation. The above calculation method is the existing technology and will not be elaborated here.

[0078] Step 2-1-4: Transfer the target image I t Multiplying it by the homography matrix H yields the target image I. t The deformed image will be the target image I. t The deformed image is placed into a designated image area to obtain the first image I. wt And generate the first image I wt The corresponding mask image M t ;like Figure 1 As shown, in this embodiment, the set image area is an image with a completely black background. Of course, the set image area can also be an image with a completely white background. The background color of the set image area is not limited, as long as the first image I... wt Second image I wr They should be the same size and have the same background color;

[0079] Step 2-1-5: Place reference image I r Place it in a designated image area of ​​the same size, referencing image I. r Placement with target image I t The positions of the two images at least partially overlap to obtain the second image I. wr And generate a second image I wr The corresponding mask image M r .

[0080] The first total loss function L mentioned above total1 The calculation formula is:

[0081] L total1 =ω1L1+ω2L2

[0082] Where L1 is the first loss function, L2 is the second loss function, ω1 is the weight of the first loss function, ω2 is the weight of the second loss function, ω1∈(0,1), ω2∈(0,1), and ω1+ω2=1;

[0083]

[0084] Where (i,j) are the coordinates of the pixel, I is the total number of x-coordinates of the pixels, and J is the total number of y-coordinates of the pixels. Indicate Iwr The pixel value of the pixel with upper coordinate (i,j). For deformation operation, That is, I wt , For I wt The pixel value of the pixel with coordinates (i,j) is P. (i,j) Let H be the pixel value of the pixel at coordinate (i,j) on P. -1 The inverse transformation of the homography matrix H, For I r With H -1 The resulting product is the pixel value of the pixel at coordinate (i,j) in the image. P′ is the inverse transform of P. (i,j) Let be the pixel value of the pixel at coordinate (i,j) on P′;

[0085]

[0086] in, For E wr The pixel value of the pixel with upper coordinate (i,j). For E wt The pixel value of the pixel with coordinates (i,j) is given by ∈, where ∈ is a very small constant to prevent the denominator in the L2 calculation formula from being 0.

[0087] The first loss function mentioned above is to ensure that the pixels in the overlapping area remain consistent. The first summation in the first loss function represents the pixel difference in the overlapping area between the two images, and the second summation represents the pixel difference in the overlapping area between the two images during inverse transformation. The second summation formula is added to make the alignment constraint more robust. The second loss function is to encourage the edge maps of the overlapping area to remain consistent after deformation, so as to achieve accurate alignment of pixels and edges.

[0088] like Figure 2 As shown, the image synthesis network in this embodiment includes two encoders with shared weights and one decoder. The specific steps for obtaining the stitched image S are as follows:

[0089] Step 2-2-1: Transfer the first image I wr Second image I wr Each input is fed into an encoder to obtain I. wt The corresponding feature map and I wr The corresponding feature map, calculate I wt The corresponding feature map and I wr The difference between the corresponding feature maps is used to make a skip connection between the difference and the input of the decoder to obtain a single-channel mask γ;

[0090] like Figure 2 As shown, each encoder includes multiple sequentially connected upsampling layers, and the decoder includes multiple sequentially connected downsampling layers. The difference between the feature maps output by the first upsampling layer of the two encoders is skipped to the input of the fifth downsampling layer; the difference between the feature maps output by the second upsampling layer of the two encoders is skipped to the input of the fourth downsampling layer; the difference between the feature maps output by the third upsampling layer of the two encoders is skipped to the input of the third downsampling layer; the difference between the feature maps output by the fourth upsampling layer of the two encoders is skipped to the input of the second downsampling layer; and the difference between the feature maps output by the fifth upsampling layer of the two encoders is skipped to the input of the first downsampling layer.

[0091] Step 2-2-2: Calculate I according to the following formula. wr The weight M in the final stitched image wr and I wt The weight M in the final stitched image wt ;

[0092] M wr =(M r -P)+P×γ

[0093] M wt =(M t -P)+P×(1-γ)

[0094] Where P represents I wt The corresponding mask image M t with I wr The corresponding mask image M r The mask for the overlapping area, where the pixel value of the overlapping area is 1 and the pixel value of the non-overlapping area is 0;

[0095] Step 2-2-3: Calculate the final stitched image S according to the following formula;

[0096] S = I wr ×M wr +I wt ×M wt .

[0097] The image synthesis network in this embodiment is based on the classic U-Net network architecture with the addition of "skip connections." A key feature of the U-Net architecture is its symmetrical "U"-shaped structure, with an encoder path and a decoder path. The encoder path, similar to traditional convolutional networks, extracts contextual information and high-level semantic features of the image step-by-step through a series of convolutions and downsampling operations, while simultaneously reducing the spatial resolution of the feature maps. The decoder path relies on upsampling and convolution operations to restore the image's details and spatial resolution step-by-step, ultimately generating an output the same size as the input image. In this embodiment, by adding "skip connections," the feature maps of corresponding layers in the encoder path are directly passed and concatenated to the output of the corresponding upsampling layer in the decoder path. These skip connections allow the decoder to simultaneously utilize both low-level, detail-rich features from the encoder and high-level, semantically-rich features from the decoder itself when reconstructing the image.

[0098] The second total loss function L in step 2-2 above total2 The calculation formula is:

[0099] L total2 =ω3L3+ω4L4

[0100] Where L3 is the third loss function, L4 is the fourth loss function, ω3 is the weight of the third loss function, ω4 is the weight of the fourth loss function, ω3∈(0,1), ω4∈(0,1), and ω3+ω4=1;

[0101] L3=||S*M br -I wr *M br ||1+||S*M bt -I wt *M bt ||1

[0102] ||.||1 is for calculating the first norm, M br To make M t The result of dilating the inverted region is the same as M. r The boundary mask obtained by multiplication (Mt) is denoted as 1 for the valid region (white) and 0 for the invalid region (black), M t The inverted region represents the valid region (white) as 0 and the invalid region (black) as 1. Expansion refers to dividing M... t After inversion, the invalid region is expanded into the valid region, that is, the edge of the 0 region is marked as 1, and then M is added. r Multiplying them together gives us M. br M bt To make M r The result of dilating the inverted region is the same as M. tThe boundary mask obtained by multiplication; such as Figure 4 As shown, Figure 4 (a) in the middle is M br , Figure 4 (b) in the middle is M bt ;

[0103] L4 = L D +L s

[0104]

[0105] For M wr The pixel value of the pixel with upper coordinate (i,j). For M wr The pixel value of the pixel with coordinates (i+1,j) is given by D, where D is the luminance difference, and D = (I... wr -I wt ) 2 D (i,i) Let D be the pixel value of the pixel with coordinates (i,j) on D. (i +1,j) Let be the pixel value of the pixel at coordinate (i+1,j) on D. For M wr The pixel value of the pixel with coordinates (j, j+1), D (i,j+1) Let be the pixel value of the pixel at coordinate (i,j+1) on D;

[0106]

[0107] S (i,j) Let S be the pixel value of the pixel with coordinates (i,j) on S. (i+1,j) Let S be the pixel value of the pixel with coordinates (i+1,j) on S. (i,j+1) Let be the pixel value of the pixel at coordinate (i,j+1) on S.

[0108] In this embodiment, when calculating the third loss function, ||S*M br -I wr *M br ||1 will allow the network to learn S and I during the training process. wr Pixels on one side of the boundary are consistent; similarly, ||S*M bt -I wt *M bt ||1 will allow the network to learn S and I during the training process. wt The pixels on the other boundary are consistent; the edge pixels of S originate from I. wr or I wt However, due to M br and Mbt The existence of intersection points introduces uncertainty regarding their attribution, but it is precisely this uncertainty that fixes the endpoints of the seam to these intersection points, such as... Figure 3 As shown; the fourth loss function is to ensure the optimal splice seam.

[0109] This embodiment also relates to a computer-readable storage medium storing a computer program that can be read and executed by a processor. When the computer program is executed by the processor, it implements the above-described method for stitching together glass curtain wall images taken by a drone.

[0110] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for stitching together images of glass curtain walls captured by drones, characterized in that... Includes the following steps: Step 1: Take a certain number of images of glass curtain walls using a drone to form a dataset, and divide the dataset into a training set, a validation set, and a test set; Step 2: Construct an image stitching network and train it multiple times using the training set to obtain the trained image stitching network; the image stitching network includes a deformation prediction network and an image synthesis network; Two glass curtain wall images are selected from the training set, and one of them is denoted as target image I. t Another image of the glass curtain wall is designated as reference image I. r I t and I r The images are the same size, I t and I r From the same building, and I t and I r The locations of the glass curtain walls shown in the image overlap at least partially, using I t and I r The specific process of training an image stitching network once is as follows: Step 2-1: Transfer the target image I t and reference image I r The image is input into the deformation prediction network to obtain the first image I. wt Second image I wr First image I wt The corresponding mask image M t And the second image I wr The corresponding mask image M r M t and M r The pixel value in the valid area is 1, and the pixel value in the invalid area is 0; For I respectively t and I r Perform edge extraction, and then extract the I values ​​from the edge-extracted edges. t and I r The input is fed into the deformation prediction network to obtain the first edge map E. wt Second edge map E wr ; According to the first image I wt Second image I wr Calculate the first loss function and based on the first edge map E. wt and the second edge graph E wr Calculate the second loss function, combine the first loss function and the second loss function to obtain the first total loss function, and use the first total loss function to update the network parameters of the deformation prediction network, thus completing one training of the deformation prediction network; Step 2-2: Transfer the first image I wt Second image I wr The image is input into an image synthesis network to obtain the final stitched image S; The second total loss function is calculated based on the final stitched image S, and the network parameters of the image synthesis network are updated using the second total loss function. This completes one training of the image synthesis network and yields a trained image stitching network. The image synthesis network consists of two encoders with shared weights and one decoder. The specific steps for obtaining the stitched image S are as follows: Step 2-2-1: Transfer the first image I wt Second image I wr Each input is fed into an encoder to obtain I. wt The corresponding feature map and I wr The corresponding feature map, calculate I wt The corresponding feature map and I wr The difference between the corresponding feature maps is used to make a skip connection between the difference and the input of the decoder to obtain a single-channel mask γ; Step 2-2-2: Calculate I according to the following formula. wr The weight M in the final stitched image wr and I wt The weight M in the final stitched image wt ; M wr =(M r -P)+P×γ M wt =(M t -P)+P×(1-γ) Where P represents I wt The corresponding mask image M t with I wr The corresponding mask image M r The mask for the overlapping area, where the pixel value of the overlapping area is 1 and the pixel value of the non-overlapping area is 0; Step 2-2-3: Calculate the final stitched image S according to the following formula; S=I wr ×M wr +I wt ×M wt Step 3: Use the validation samples in the validation set to validate the performance of the trained image stitching network model and select the optimal image stitching network. Step 4: Select two images in the test set that have at least partial overlap at the glass curtain wall locations. The two images selected in the test set are from the same building. The buildings corresponding to the test samples, the validation samples, and the training samples in the training set are all different. Then input it into the optimal image stitching network obtained in step 3 to obtain the stitching result.

2. The glass curtain wall image stitching method according to claim 1, characterized in that: Step 2-1 specifically involves: The deformation prediction network includes an optical flow calculation module and a homography estimation module. The homography estimation module includes a regression network. The specific processing procedure of the deformation prediction network is as follows: Step 2-1-1: Transfer the target image I t and reference image I r The initial optical flow is obtained by inputting the data into the optical flow calculation module. Step 2-1-2: Scale the initial optical flow to 1 / N of the initial optical flow, where N is an integer greater than 1, to obtain the final optical flow; Step 2-1-3: Transfer the target image I t The four corner points are denoted as p1, p2, p3, and p4. The final optical flow is input into the regression network of the homography estimation module to predict the pixel offsets of the four corner points p1, p2, p3, and p4 relative to their original positions. The target image I is then calculated based on the predicted pixel offsets. t The four corner points in reference image I r The coordinates of the target corresponding points in the coordinate system are used, and then the direct linear transformation (DLT) algorithm is used to solve for the implicit homography matrix H between the four corner points p1, p2, p3 and p4 and their corresponding target corresponding points. Step 2-1-4: Transfer the target image I t Multiplying it by the homography matrix H yields the target image I. t The deformed image will be the target image I. t The deformed image is placed into a designated image area to obtain the first image I. wt And generate the first image I wt The corresponding mask image M t ; Step 2-1-5: Place reference image I r Place it in a designated image area of ​​the same size, referencing image I. r Placement with target image I t The positions of the two images at least partially overlap to obtain the second image I. wr And generate a second image I wr The corresponding mask image M r .

3. The glass curtain wall image stitching method according to claim 2, characterized in that: The optical flow calculation module in step 2-1-1 uses the GMFlow network.

4. The glass curtain wall image stitching method according to any one of claims 1 to 3, characterized in that: The first total loss function L total1 The calculation formula is: L total1 =ω1L1+ω2L2 Where L1 is the first loss function, L2 is the second loss function, ω1 is the weight of the first loss function, ω2 is the weight of the second loss function, ω1∈(0,1), ω2∈(0,1), and ω1+ω2=1; Where (i,j) are the coordinates of the pixel, I is the total number of x-coordinates of the pixels, and j is the total number of y-coordinates of the pixels. Indicate I wr The pixel value of the pixel with upper coordinate (i,j). For deformation operation, That is, I wt , For I wt The pixel value of the pixel with coordinates (i,j) is P. (i,j) Let H be the pixel value of the pixel at coordinate (i,j) on P. -1 The inverse transformation of the homography matrix H, For I r With H -1 The resulting product is the pixel value of the pixel at coordinate (i,j) in the image. P′ is the inverse transform of P. (i,j) Let be the pixel value of the pixel at coordinate (i,j) on P′; in, For E wr The pixel value of the pixel with upper coordinate (i,j). For E wt The pixel value of the pixel with coordinates (i,j) is given by ∈, where ∈ is a very small constant to prevent the denominator in the L2 calculation formula from being 0.

5. The glass curtain wall image stitching method according to claim 1, characterized in that: The second total loss function L in step 2-2 total2 The calculation formula is: L total2 =ω3L3+ω4L4 Where L3 is the third loss function, L4 is the fourth loss function, ω3 is the weight of the third loss function, ω4 is the weight of the fourth loss function, ω3∈(0,1), ω4∈(0,1), and ω3+ω4=1; L3=‖S*M br -I wr *M br ‖1+‖S*M bt -I wt *M bt ‖1 ‖.‖1 is used to calculate the first norm, M br To make M t The result of dilating the inverted region is the same as M. r The boundary mask obtained by multiplication; M bt To make M r The result of dilating the inverted region is the same as M. t The boundary mask obtained by multiplication; L4=L D +L s For M wr The pixel value of the pixel with upper coordinate (i,j). For M wr The pixel value of the pixel with coordinates (i+1,j) is given by D, where D is the luminance difference, and D = (I... wr -I wt ) 2 D (i,j) Let D be the pixel value of the pixel with coordinates (i,j) on D. (i+1,j) Let be the pixel value of the pixel at coordinate (i+1,j) on D. For M wr The pixel value of the pixel with coordinates (i,j+1), D (i,j+1) Let be the pixel value of the pixel at coordinate (i,j+1) on D; S (i,j) Let S be the pixel value of the pixel with coordinates (i,j) on S. (i+1,j) Let S be the pixel value of the pixel with coordinates (i+1,j) on S. (i,j+1) Let be the pixel value of the pixel at coordinate (i,j+1) on S.

Images