An image stitching method based on unsupervised learning and adversarial generative network
By optimizing the image stitching algorithm through unsupervised learning and adversarial generative networks, the problems of poor image alignment and low quality of fused images are solved, achieving faster training and inference speeds and more natural stitching results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUZHOU LIANSHITAI ELECTRONIC INFORMATION TECH CO LTD
- Filing Date
- 2022-12-26
- Publication Date
- 2026-08-04
AI Technical Summary
Existing deep learning-based image stitching algorithms suffer from problems such as large number of parameters, long training and inference time, poor image alignment, and artifacts and stitching marks in the merged image during the alignment and fusion stages.
Unsupervised learning is used to train the alignment model, and an adversarial generative network is used to build the stitching model. The image alignment and fusion process is optimized by using grid vertex offset and adversarial generative loss function, which reduces the number of parameters and improves the training and inference speed. The generator automatically learns real-world image features.
It improves image alignment, reduces computational resource consumption, enhances the naturalness and accuracy of image stitching, and reduces stitching artifacts and artifacts.
Smart Images
Figure CN115965529B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to an image stitching method based on unsupervised learning and adversarial generative networks. Background Technology
[0002] Image stitching technology is a technique that can stitch together two images that have parallax but contain overlapping areas to obtain a seamless high-definition panoramic image. It has wide applications in autonomous driving, video security, and virtual reality.
[0003] Traditional image stitching involves manually extracting corresponding feature points from two images to be stitched, calculating a 3×3 homography matrix capable of handling image translation, rotation, scaling, and ideal point transformations, using the homography matrix to project one image to align it with the other, and then fusing the two aligned images to obtain the final panoramic image. However, traditional methods have limited capabilities in feature learning and image fusion, resulting in poor image alignment and often producing misalignment and ghosting artifacts in the final image obtained through the fusion stage.
[0004] Thanks to the powerful automatic feature learning capabilities of deep learning technology, image stitching methods based on neural networks have become mainstream. Deep learning-based image stitching algorithms mainly consist of two stages. The first stage is image alignment, which uses a convolutional neural network to extract corresponding feature points from the two images to be stitched, and then aligns the images through projection transformation. The second stage is image fusion, which uses the aligned two images as input to the neural network, outputting a panoramic stitched image with a smooth transition in the overlapping areas.
[0005] Most current deep learning-based image stitching algorithms use relatively simple network structures in the first stage, resulting in a large number of parameters and long training and inference times. Furthermore, training is typically based on supervised learning methods, using training images manually generated through homography transformations. This deviates from the multi-depth-of-field and multi-plane alignment tasks of real-world images, and the alignment algorithms often only use a single homography matrix to project the target image. All these factors contribute to suboptimal image alignment, leaving significant room for improvement. In the second stage of current mainstream stitching algorithms, multiple manually designed loss functions are often added to the output image to constrain the quality of the stitched image. This approach makes it difficult for the stitched image to truly achieve the texture effect of the real image, resulting in artifacts and noticeable stitching artifacts in the fused image. Summary of the Invention
[0006] In view of the above, the purpose of this invention is to provide an image stitching method based on unsupervised learning and adversarial generative networks, comprising the following steps:
[0007] (1) The two images to be stitched are used as the reference image and the target image and sent into the alignment model. The mesh vertex offset is calculated.
[0008] (2) Perform projection transformation on the target image based on the grid vertex offset to obtain the aligned target image;
[0009] (3) Input the aligned target image and reference image into the stitching model to stitch them together to obtain the stitched image.
[0010] Preferably, the process of constructing the alignment model includes:
[0011] (a) Constructing an alignment model;
[0012] (b) Select a public image dataset, crop and transform the images to obtain image pairs, and form dataset A1. Obtain image pairs with different proportions of overlapping areas and parallax collected in the real world, and form dataset A2. The image pairs include a reference image and a target image.
[0013] (c) Using dataset A1 as the sample set, perform steps (d) and (e);
[0014] (d) Input the image pairs in the sample set into the alignment model to extract features, and calculate the offset of (n+1)×(m+1)×2 grid vertices based on the features;
[0015] (e) Construct n×m transformation matrices based on grid vertex offsets, divide the target image into n×m image blocks, use the n×m transformation matrices to perform projection transformations on the corresponding image blocks respectively, and then stitch them together to obtain the aligned target image. Adjust the network parameters of the alignment model by comparing the similarity of the overlapping areas of the aligned target image and the reference image.
[0016] (e) Using dataset A2 as the sample set, repeat steps (d) and (e) based on step (e) to fine-tune the network parameters of the alignment model and obtain the trained alignment model.
[0017] Preferably, the alignment model includes two branches with the same structure, which are used to extract feature maps of the reference image and the target image in the image, respectively. Each branch includes a convolutional layer and N CSP modules. Each CSP module outputs a feature map and uses the feature map as input to the next CSP module.
[0018] After the feature maps output by the CSP module of the same layer of the two branches are stitched together in the channel direction, the stitched result is subjected to feature extraction and optimization by several convolutional layers. Then, a regression network composed of average pooling layer and fully connected layer is used to regress and calculate a grid vertex offset based on the optimization result.
[0019] Preferably, each CSP module includes two sub-branches. Sub-branch one is composed of a CBS module including a convolutional layer, a batch normalization layer, a SiLU activation layer, a ResNet residual unit, and a convolutional layer connected in sequence, used to extract feature maps. Sub-branch two has only one independent convolutional layer, used to extract feature maps. Then, the feature maps of the two sub-branches are concatenated and input into the batch normalization layer, the Leaky ReLU activation layer, and the CBS module, and the feature map is output after calculation.
[0020] Preferably, the step of constructing N transformation matrices based on mesh vertex offsets includes:
[0021] When there are N network offsets, denoted as S i When i = 1, 2, ..., N, the N transformation matrices constructed are represented as S1, S1+S2, S1+S2+S3, ..., S1+S2+...+S N The N transformation matrices are expressed by the formula:
[0022] Preferably, adjusting the network parameters of the alignment model by comparing the similarity of overlapping regions between the aligned target image and the reference image includes:
[0023] Construct the following loss function L align The network parameters of the alignment model are adjusted and optimized based on minimizing the loss function.
[0024]
[0025] Among them, I A Represents the target image, I B Indicates a reference image. Let E represent the transformation matrix, and let λ represent a matrix of the same size as the image, consisting entirely of 1s. i represents the loss weight corresponding to each transformation, ⊙ represents pixel-level multiplication, and ‖·‖1 represents the L1 norm.
[0026] Preferably, the construction process of the splicing model includes:
[0027] (i) The splicing model adopts an adversarial generative network to establish the generator and discriminator of the adversarial generative network;
[0028] (ii) Obtain image pairs with different proportions of overlapping regions and parallax collected in the real world to form dataset A2. Input dataset A2 into the trained alignment model to obtain the alignment target image. Set the original image pairs in dataset A2 as the real label and the stitched image generated by the generator as the synthetic label. Directly stack the alignment target image and the reference image. Replace the pixel value of the overlapping region with the average value of the pixels of the two images to obtain the superimposed image. Set the superimposed image as the synthetic label to obtain the training dataset.
[0029] (iii) Feed the training dataset into the adversarial generative network, train the adversarial generative network using the adversarial generative loss function, update the network parameters, and use the parameter-optimized generator as the splicing model.
[0030] Preferably, the generator adopts an encoder-decoder structure, specifically comprising: the generator being formed by sequentially connecting multiple convolutional layers and deconvolutional layers of the same number, and each convolutional layer and deconvolutional layer being followed by a batch normalization layer.
[0031] Preferably, the classifier network consists of convolutional layers, average pooling layers, and connection layers.
[0032] Preferably, the adversarial generation loss function includes a generator loss and a discriminator loss;
[0033] The discriminator loss is expressed as:
[0034]
[0035] The generator loss is expressed as:
[0036]
[0037] Where a represents a synthetic tag, b represents a real tag, and p x p represents the distribution of the real image x, where the real image includes pairs of original images. z Let z represent the distribution of the target image and the reference image z to be aligned, D(·) represent the discriminator, G(·) represent the generator, and c represent the predicted value that the generator wants the discriminator to output for the generated data.
[0038] Compared with the prior art, the beneficial effects of the present invention include at least the following:
[0039] This invention optimizes the structure of the alignment model in the alignment stage of the image stitching algorithm, reducing the number of parameters and making the alignment model structure more suitable for image alignment tasks. This reduces computational resource consumption and improves the training, convergence, and inference speeds of the network. Furthermore, it introduces a grid-based image transformation mechanism, which better reflects real-world image projection transformation mechanisms, resulting in superior image alignment performance.
[0040] This invention uses a stitching model constructed with an adversarial generative network to realize the fusion step in image stitching. Compared with the traditional manual design of loss function to constrain the fusion effect, this invention enables the generator to automatically learn the features of real-world images through the game between the generator and the discriminator, making the overlapping and transition areas of the final stitched image more natural and greatly reducing stitching marks and image artifacts. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 This is a flowchart illustrating the image stitching method based on unsupervised learning and adversarial generative networks in this invention.
[0043] Figure 2 This is a flowchart illustrating the construction process of the alignment model in this invention;
[0044] Figure 3 This is a schematic diagram of the alignment model in this invention;
[0045] Figure 4 This is a schematic diagram of the CSP module in the alignment mesh model of the present invention;
[0046] Figure 5 This is a flowchart illustrating the construction process of the splicing model in this invention;
[0047] Figure 6 This is a schematic diagram of the generator structure in the splicing model of this invention;
[0048] Figure 7 This is a schematic diagram of the discriminator in the splicing model of the present invention. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.
[0050] like Figure 1 As shown, the embodiment provides an image stitching method based on unsupervised learning and adversarial generative networks, including the following steps:
[0051] Step 1: Input the two images to be stitched into the alignment model as the reference image and the target image, and calculate the mesh vertex offset.
[0052] In this embodiment, the alignment model is trained in an unsupervised manner, receiving two images that need to be stitched together and have parallax as input, and outputting the transformation result in the form of grid vertex offset.
[0053] In the embodiments, the alignment model is constructed based on a convolutional neural network, such as Figure 2 As shown, the specific construction process includes:
[0054] (a) Constructing an alignment model.
[0055] In the embodiments, such as Figure 3 As shown, the alignment model includes two branches with identical structures and consistent network parameters. The two branches are used to extract feature maps of the reference image and the target image, respectively. Each branch includes a convolutional layer and N CSP modules. Each CSP module outputs a feature map and uses the feature map as input to the next CSP module. The feature maps output by the same layer of the two branches are concatenated in the channel direction. After feature extraction and optimization of the concatenated result using several convolutional layers, a regression network composed of average pooling layers and fully connected layers is used to regress and calculate one grid vertex offset based on the optimization result. Figure 3 The example shows 3 CSP modules, which generate a total of 3 mesh vertex offsets.
[0056] like Figure 4 As shown, each CSP module includes two sub-branches. Sub-branch one consists of a CBS module including a convolutional layer, a batch normalization layer, a SiLU activation layer, a ResNet residual unit, and a convolutional layer connected in sequence, used to extract feature maps. Sub-branch two has only one independent convolutional layer, used to extract feature maps. Then, the feature maps of the two sub-branches are concatenated and input into the batch normalization layer, the Leaky ReLU activation layer, and the CBS module, and the output feature map is calculated.
[0057] (b) Construct datasets A1 and A2 for training the alignment model.
[0058] In this embodiment, a publicly available image dataset, such as the MS COCO dataset, is selected. Image pairs are cropped and transformed from the images to form dataset A1. Specifically, any image from the MS COCO dataset is selected and cropped to obtain image pairs of 64×64 size with different overlapping areas. A random projection transformation is applied to one of the images to obtain a set of image pairs to be stitched together.
[0059] In this embodiment, image pairs with different proportions of overlapping areas and parallax collected in the real world are also acquired to form dataset A2, wherein the image pairs include a reference image and a target image.
[0060] (c) Use dataset A1 as the sample set.
[0061] (d) Input the image pairs in the sample set into the alignment model to extract features, and calculate the offset of N grid vertices based on the features.
[0062] In the embodiments, such as Figure 3 As shown, after each input image passes through the CSP module, the feature maps of the two branches are concatenated and fed into a regression network consisting of an average pooling layer and a fully connected layer, outputting a grid offset value of size (n+1)×(m+1)×2. This invention sets the image transformation to be represented by an n×m grid transformation; therefore, the coordinate offset of the grid vertex can be represented by a tensor of size (n+1)×(m+1)×2.
[0063] (e) Construct n×m transformation matrices based on the grid vertex offsets, and update the network parameters of the alignment model after transforming the target image according to the transformation matrices.
[0064] In this embodiment, n×m transformation matrices are constructed based on the grid vertex offsets. The target image is evenly divided into n×m image blocks. The corresponding image blocks are projected and transformed using the n×m transformation matrices and then stitched together to obtain the aligned target image. The network parameters of the alignment model are adjusted by comparing the similarity of the overlapping areas between the aligned target image and the reference image.
[0065] This involves constructing n×m transformation matrices based on the mesh vertex offsets, including: when there are N sets of network offsets, denoted as S i When i = 1, 2, ..., N, the N transformation matrices constructed are represented as S1, S1+S2, S1+S2+S3, ..., S1+S2+...+S N The N transformation matrices are expressed by the formula: i=1,2,…,N. like Figure 3 As shown, there are 3 mesh vertex offsets, so 3 transformation matrices are constructed, which are represented as S1, S1+S2, and S1+S2+S3 respectively.
[0066] Construct the following loss function L align The network parameters of the alignment model are adjusted and optimized based on minimizing the loss function.
[0067]
[0068] Among them, I A Represents the target image, I B Indicates a reference image. Let E represent the transformation matrix, and let λ represent a matrix of the same size as the image, consisting entirely of 1s. i represents the loss weight corresponding to each transformation, ⊙ represents pixel-level multiplication, and ‖·‖1 represents the L1 norm.
[0069] (e) Use dataset A2 as the sample set and proceed to step (d).
[0070] In the embodiment, based on step (e), the dataset A2 is used as the sample set, and steps (d) and (e) are repeated to fine-tune the network parameters of the alignment model and obtain the trained alignment model.
[0071] Step 2: Perform a projection transformation on the target image based on the grid vertex offset to obtain the aligned target image.
[0072] In this embodiment, based on the obtained grid vertex offset, the target image is projected and transformed according to the grid vertex offset to obtain the aligned target image.
[0073] Step 3: Input the aligned target image and reference image into the stitching model for stitching to obtain the stitched image.
[0074] In this embodiment, the stitching model employs an adversarial generative network structure, using least-squares loss to train the generator to automatically learn how to fuse aligned images to obtain the stitched image. Figure 5 As shown, the construction process of the splicing model includes:
[0075] (i) The splicing model adopts an adversarial generative network to establish the generator and discriminator of the adversarial generative network.
[0076] In the embodiments, such as Figure 6 As shown, the generator adopts an encoder-decoder structure, specifically: the generator is obtained by connecting multiple convolutional layers and deconvolutional layers of the same number in sequence, and each convolutional layer and deconvolutional layer is followed by a batch normalization layer.
[0077] like Figure 7 As shown, the classifier network consists of four convolutional layers, one average pooling layer, and one fully connected layer. The output probability results are used to determine whether the input image is a real image or a synthetic image.
[0078] (ii) Construct the training dataset for the splicing model.
[0079] In this embodiment, image pairs with varying overlap ratios and parallax, collected in the real world, are acquired and formed into dataset A2. Dataset A2 is fed into a trained alignment model to obtain the alignment target image. The original image pairs in dataset A2 are set as the real labels, and the stitched image generated by the generator is set as the synthetic label. The alignment target image and the reference image are directly stacked, and the pixel values of the overlapping regions are replaced by the average of the pixels in the two images to obtain the superimposed image. The superimposed image is set as the synthetic label. This results in a set of data containing two real images (original image pairs) and two synthetic images (stitched image and superimposed image), which in turn yields the training dataset.
[0080] (iii) Feed the training dataset into the adversarial generative network, train the adversarial generative network using the adversarial generative loss function, update the network parameters, and use the parameter-optimized generator as the splicing model.
[0081] In this embodiment, referring to the loss function of the LSGAN network, a represents the synthetic label, b represents the real label, and p x p represents the distribution of the real image x, where the real image includes pairs of original images. z Let z represent the distribution of the target image and the reference image z to be aligned, D(·) represent the discriminator, G(·) represent the generator, and c represent the predicted value that the generator wants the discriminator to output based on the generated data. Then the discriminator loss is expressed as:
[0082]
[0083] The generator loss is expressed as:
[0084]
[0085] Optionally, the label parameters can be set to a = -1, b = 1, c = 0. Then, the four images of each group are input into the generative adversarial network, and the parameters of the generator and discriminator are updated simultaneously through the two losses mentioned above to complete the training.
[0086] After training, the trained generator is used as a stitching model. When applied, the target image and reference image to be aligned are input into the stitching model, and the stitched image is output after calculation.
[0087] The method provided in the above embodiments trains the alignment model based on unsupervised learning and the stitching model based on the adversarial generative network structure. It can quickly train a high-precision model without labeled data, thereby improving the accuracy of image stitching.
[0088] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. An image stitching method based on unsupervised learning and adversarial generative networks, characterized in that, Includes the following steps: (1) The two images to be stitched are used as the reference image and the target image and fed into the alignment model. The mesh vertex offset is calculated. The alignment model includes two branches with the same structure, which are used to extract feature maps of the reference image and the target image respectively. Each branch includes a convolutional layer and N CSP modules. Each CSP module outputs a feature map and uses the feature map as input to the next CSP module. After the feature maps output by the CSP module of the same layer of the two branches are stitched together in the channel direction, the stitched result is subjected to feature extraction and optimization by several convolutional layers. Then, a regression network composed of average pooling layer and fully connected layer is used to regress and calculate one grid vertex offset based on the optimization result. The process of constructing the alignment model includes: (a) Constructing an alignment model; (b) Select a public image dataset, crop and transform the images to obtain image pairs, and form dataset A1. Obtain image pairs with different proportions of overlapping areas and parallax collected in the real world, and form dataset A2. The image pairs include a reference image and a target image. (c) Using dataset A1 as the sample set, perform steps (d) and (e); (d) Input the image pairs in the sample set into the alignment model to extract features, and calculate the offset of (n+1)×(m+1)×2 grid vertices based on the features; (e) Based on the grid vertex offset, construct n×m transformation matrices, divide the target image into n×m image blocks, use the n×m transformation matrices to perform projection transformation on the corresponding image blocks respectively, and then stitch them together to obtain the aligned target image. Adjust the network parameters of the alignment model by comparing the similarity of the overlapping areas between the aligned target image and the reference image. (f) Using dataset A2 as the sample set, repeat steps (d) and (e) to fine-tune the network parameters of the alignment model and obtain the trained alignment model; (2) The target image is obtained by projecting the target image according to the grid vertex offset; (3) Input the aligned target image and reference image into the stitching model to stitch them together to obtain the stitched image.
2. The image stitching method based on unsupervised learning and adversarial generative networks according to claim 1, characterized in that, Each CSP module includes two sub-branches. Sub-branch one consists of a CBS module containing convolutional layers, batch normalization layers, SiLU activation layers, ResNet residual units, and convolutional layers connected in sequence, used to extract feature maps. Sub-branch two has only one independent convolutional layer, used to extract feature maps. The feature maps from the two sub-branches are then concatenated and input into the batch normalization layer, LeakyReLU activation layer, and CBS module, and the output feature map is calculated.
3. The image stitching method based on unsupervised learning and adversarial generative networks according to claim 1, characterized in that, The construction of n×m transformation matrices based on the mesh vertex offsets includes: When there are N sets of network offsets, denoted as S i When i = 1, 2, ..., N, the constructed N transformation matrices are represented as S1, S1 + S2, S1 + S2 + S3, ..., S1 + S2 + ... + S N The N transformation matrices are expressed by the formula: , i=1,2,…,N.
4. The image stitching method based on unsupervised learning and adversarial generative networks according to claim 3, characterized in that, The method of adjusting the network parameters of the alignment model by comparing the similarity of overlapping regions between the aligned target image and the reference image includes: Construct the following loss function The network parameters of the alignment model are adjusted and optimized based on minimizing the loss function. in, Represents the target image. Indicates a reference image. Represents the transformation matrix. E This represents a matrix of the same size as the image, consisting entirely of 1s. This represents the loss weights corresponding to each transformation, and ⨀ represents pixel-level multiplication. It represents the first norm.
5. The image stitching method based on unsupervised learning and adversarial generative networks according to claim 1, characterized in that, The construction process of the splicing model includes: (i) The splicing model adopts an adversarial generative network to establish the generator and discriminator of the adversarial generative network; (ii) Obtain image pairs with different proportions of overlapping regions and parallax collected in the real world to form dataset A2. Input dataset A2 into the trained alignment model to obtain the alignment target image. Set the original image pairs in dataset A2 as the real label and the stitched image generated by the generator as the synthetic label. Directly stack the alignment target image and the reference image. Replace the pixel value of the overlapping region with the average value of the pixels of the two images to obtain the superimposed image. Set the superimposed image as the synthetic label to obtain the training dataset. (iii) Feed the training dataset into the adversarial generative network, train the adversarial generative network using the adversarial generative loss function, update the network parameters, and use the parameter-optimized generator as the splicing model.
6. The image stitching method based on unsupervised learning and adversarial generative networks according to claim 5, characterized in that, The generator adopts an encoder-decoder structure, specifically comprising: the generator is formed by sequentially connecting multiple convolutional layers and deconvolutional layers of the same number, and each convolutional layer and deconvolutional layer is followed by a batch normalization layer.
7. The image stitching method based on unsupervised learning and adversarial generative networks according to claim 5, characterized in that, The classifier network consists of convolutional layers, average pooling layers, and connection layers.
8. The image stitching method based on unsupervised learning and adversarial generative networks according to claim 5, characterized in that, The adversarial generation loss function includes generator loss and discriminator loss; The discriminator loss is expressed as: The generator loss is expressed as: in, Indicates a composite tag. Indicates the true label, Represents a real image x The distribution of real images includes pairs of original images. This represents the distribution of z in the aligned target and reference images. Indicates the discriminator, Represents a generator. This indicates the generator's expectation that the discriminator will output a predicted value based on the generated data.