Non-supervision image splicing method based on recursive thin plate spline transformation
Through the unsupervised image stitching method based on recursive thin plate spline transformation, the problems of misalignment artifacts and high computational cost of image stitching in real scenes with depth variation are solved, and efficient and accurate image alignment and fusion are achieved.
Patent Information
- Application Number
- CN202510768032.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-10
- Publication Date
- 2025-10-17
AI Technical Summary
Existing image stitching methods suffer from misalignment artifacts and high computational costs in real-world scenes with varying depths. Traditional methods rely on feature point detection, while deep learning methods perform poorly in scenes with varying depths. Furthermore, existing thin-plate spline transformation methods increase computational costs but offer limited performance improvements.
An unsupervised image stitching method based on recursive thin-plate spline transformation is adopted. Pre-alignment is performed by estimating the global homography matrix of the reference image and the target image. Then, recursive thin-plate spline transformation is performed, and multiple basic transformations are aggregated by using the transformation flow as a bridge to generate the final transformation flow, thereby achieving accurate alignment and fusion.
It improves the flexibility and accuracy of image stitching, reduces computational costs, is suitable for real-world scenarios with varying depths, and enhances the accuracy and robustness of image alignment.
Smart Images
Figure CN120807272A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of image stitching, and in particular, to an unsupervised image stitching method based on recursive thin plate spline transformation. BACKGROUND
[0002] Image stitching is a practical computer vision application that aims to construct images with wider field of view from narrow field of view images. It generates images with wider field of view and ensures visual coherence through image registration and fusion. In biology and medicine, it is used to stitch microscope images to help comprehensive observation; in autonomous driving, it can effectively generate panoramic views with wider field of view; in video surveillance, it can effectively integrate multi-camera images to improve efficiency; in virtual reality, it can create immersive panoramic experiences. With the continuous advancement of technology, image stitching technology plays an important role in many fields.
[0003] Existing image stitching solutions are divided into traditional methods and deep learning-based methods. Traditional image stitching methods extract features from input images and identify feature points that can be used for matching. By calculating the relationship between these feature points, the geometric transformation parameters of the image are determined to achieve accurate alignment between images. However, these methods are heavily dependent on the quality of feature point detection and often perform poorly in scenes with insufficient light, low texture, etc. Deep learning-based methods use convolutional neural networks or fully convolutional networks to estimate the transformation parameters between images. These methods have stronger feature extraction and expression capabilities compared to traditional methods, and can improve the accuracy and robustness of image stitching to some extent. However, existing deep image stitching methods mainly estimate global homography for image alignment, and in real scenes with depth changes, these methods produce severe misalignment artifacts.
[0004] To alleviate misalignment artifacts, some methods learn a main plane mask to reject outlier regions and estimate homography in the main plane region to achieve alignment in the main plane region. However, these methods are still essentially homography transformations and cannot effectively solve the image stitching problem in real scenes with depth changes. Compared to homography transformation, thin plate spline (Thin Plate Spline) transformation is nonlinear and has more flexible transformation performance. The latest method combines parameterized homography transformation and thin plate spline transformation to solve the image stitching problem in real scenes with depth changes. However, existing methods estimate a large number of control points at once, resulting in a significant increase in computational cost. However, the performance of thin plate spline transformation increases first and then decreases with the increase in the number of control points, resulting in an increase in computational cost without effective performance improvement. At the same time, the use of a single thin plate spline transformation shows limited flexibility and cannot effectively solve the image stitching problem in real scenes with depth changes. SUMMARY
[0005] To overcome at least one of the deficiencies in the prior art, the present application provides an unsupervised image stitching method based on recursive thin plate spline transformation.
[0006] In a first aspect, an unsupervised image stitching method based on recursive thin plate spline transformation is provided, comprising:
[0007] estimating a global homography matrix between a reference image and a target image;
[0008] transforming the target image based on the global homography matrix to obtain a pre-aligned target image;
[0009] performing recursive thin plate spline transformation on the reference image and the pre-aligned target image to obtain a final transformation flow;
[0010] transforming the pre-aligned target image based on the final transformation flow to obtain an accurately aligned target image;
[0011] performing linear fusion on the accurately aligned target image and the reference image to generate a stitched image.
[0012] In one embodiment, performing recursive thin plate spline transformation on the reference image and the pre-aligned target image to obtain a final transformation flow comprises:
[0013] Step S31, inputting the reference image and the pre-aligned target image into a feature extraction network to obtain a reference image feature map and an initial target image feature map;
[0014] Step S32, for the current transformation, inputting the reference image feature map and the target image feature map of the previous transformation into a context-related layer to calculate the correlation between the two feature maps; if it is the first time, the target image feature map of the previous transformation is the initial target image feature map;
[0015] Step S33, inputting the correlation into a TPS estimator to obtain a control point motion result;
[0016] Step S34, using a quadratic interpolation conversion method on the control point motion result to obtain a TPS transformation flow;
[0017] Step S35, converting the transformation flow of the previous transformation based on the TPS transformation flow to obtain a converted transformation flow; adding the converted transformation flow and the TPS transformation flow to obtain the current transformation flow; if it is the first time, the transformation flow of the previous transformation is initialized as an all-zero transformation flow;
[0018] Step S36, transforming the target image feature map of the previous transformation using the current transformation flow to obtain the target image feature map of the current transformation;
[0019] Step S37, return to step S32, and the next transformation is performed until the set number of transformations is reached, and the final transformation stream is obtained.
[0020] In one embodiment, the reference image and the pre-aligned target image are subjected to recursive thin plate spline transformation based on a thin plate spline estimation network, the thin plate spline estimation network including a feature extraction network, a context-dependent layer, a TPS estimator, a quadratic interpolation conversion module and a transformation stream generation module.
[0021] The thin plate spline estimation network is a trained network, and the loss function used in the training process is:
[0022] Loss=W identity L identity +W SSIM L SSIM
[0023] Wherein, Loss is the loss function, W identity is the weight of the feature intensity loss, L identity is the feature intensity loss, W SSIM is the structural similarity loss, and W SSIM is the weight of the structural similarity loss.
[0024] In one embodiment, the feature intensity loss L identity is:
[0025]
[0026] Wherein, I b is the reference image, is the pre-aligned target image, TPS i is the transformation stream of the i-th transformation, and warp represents a transformation operation on i using TPS , and N is the number of recursive transformations.
[0027] In one embodiment, the structural similarity loss W SSIM is:
[0028]
[0029] Wherein, I b is the reference image, is the pre-aligned target image, TPS i is the transformation stream of the i-th transformation, and warp represents a transformation operation on i using TPS , and SSIM represents the calculation of the similarity between images, and N is the number of recursive transformations.
[0030] In a second aspect, an unsupervised image stitching device based on recursive thin plate spline transformation is provided, comprising:
[0031] a global homography estimation module configured to estimate a global homography matrix between the reference image and the target image;
[0032] a first transformation module configured to transform the target image based on the global homography matrix to obtain a pre-aligned target image;
[0033] a recursive thin plate spline transformation module configured to perform recursive thin plate spline transformation on the reference image and the pre-aligned target image to obtain a final transformation flow;
[0034] a second transformation module configured to transform the pre-aligned target image based on the final transformation flow to obtain an accurately aligned target image;
[0035] a fusion module configured to perform linear fusion on the accurately aligned target image and the reference image to generate a stitched image.
[0036] In a third aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the above-mentioned unsupervised image stitching method based on recursive thin plate spline transformation is implemented.
[0037] In a fourth aspect, a computer program product is provided, which includes computer programs / instructions. When the computer programs / instructions are executed by a processor, the above-mentioned unsupervised image stitching method based on recursive thin plate spline transformation is implemented.
[0038] Compared with the prior art, the present application has the following beneficial effects:
[0039] 1. The present application aggregates multiple thin plate spline transformations with the same number of control points but different control points into an unsupervised deep image stitching method with more flexible and powerful transformation. This method uses transformation flow as a bridge for aggregation of multiple basic thin plate spline transformations, which not only effectively improves the flexibility of image registration and is suitable for real scenes with depth changes, but also shares the same network weight each time, which means that even if multiple thin plate spline transformations are aggregated, new weight parameters will not be introduced.
[0040] 2. The present application estimates thin plate spline transformation in a deep learning manner to improve the accuracy and robustness of image stitching, breaking the limitations of traditional methods and deep learning methods in image stitching.
[0041] 3. The present application fully utilizes the flexibility and robustness of thin plate spline transformation, focuses on aggregating multiple thin plate spline transformations into a more flexible and powerful transformation, and provides a more efficient solution for image stitching tasks. BRIEF DESCRIPTION OF DRAWINGS
[0042] The present application can be better understood with reference to the following description in conjunction with the accompanying drawings, in which:
[0043] Figure 1 A flow chart of an unsupervised image stitching method based on recursive thin plate spline transformation according to one embodiment of the present application is shown;
[0044] Figure 2 A flow chart of an unsupervised image stitching method based on recursive thin plate spline transformation according to another embodiment of the present application is shown;
[0045] Figure 3 A structure diagram of a TPS estimator is shown;
[0046] Figure 4 A structure diagram of Block 1 and Block 2 is shown;
[0047] Figure 5 A diagram of experimental results under different scenarios is shown. DETAILED DESCRIPTION
[0048] In the following, exemplary embodiments of the present application will be described with reference to the drawings. In the description, not all features of a practical embodiment are described in order to keep the description clear and concise. It should be appreciated, however, that many embodiment-specific decisions can be made in the course of developing any such practical embodiment in order to achieve the specific goals of the developer, and these decisions can vary from embodiment to embodiment.
[0049] It should also be noted that, in order not to obscure the application with details that are not necessary to understand the essence of the application, only the structures of the devices that are closely related to the solution according to the present application are shown in the drawings, and other details that are not closely related to the present application are omitted.
[0050] It should be understood that the present application is not limited to the described embodiments only due to the following description with reference to the drawings. In this context, the embodiments can be combined with each other, features can be replaced or borrowed between different embodiments, and one or more features can be omitted in one embodiment.
[0051] The application provides an unsupervised image splicing method based on recursive thin plate spline transformation, which is suitable for image splicing methods of real scenes with depth changes. The method first estimates a global homography transformation to realize pre-alignment of images and reduce the influence of large view angle changes. On the basis of the pre-aligned images, a plurality of thin plate spline transformations with the same number of control points but different control points are recursively estimated to improve the flexibility of the alignment transformation. The transformation flow is used as a bridge for aggregation of the plurality of basic thin plate spline transformations, the plurality of different thin plate spline transformations are aggregated into a more flexible and powerful transformation to realize accurate alignment of images and reduce the number of interpolations, reduce interpolation errors and improve the accuracy of image alignment, thereby realizing accurate, consistent and robust image splicing.
[0052] The application provides a specific embodiment for the field of splicing of coal mine underground monitoring video images, related mining equipment, mine background and depth changes in the mine in the coal mine underground monitoring video. The existing image splicing technology cannot achieve flexible and accurate splicing. To solve this problem, an unsupervised image splicing method based on recursive thin plate spline transformation is provided. Figure 1 A flowchart of the unsupervised image splicing method based on recursive thin plate spline transformation according to an embodiment of the application is shown in Figure 1 The method mainly includes the following steps:
[0053] Step S1, estimating a global homography matrix between a reference image and a target image.
[0054] Here, two monitoring videos of the same area in the coal mine underground monitoring video are selected, and monitoring images at the same time are extracted as a reference image and a target image, respectively, and then global homography estimation is performed to obtain a global homography matrix.
[0055] Step S2, transforming the target image based on the global homography matrix to obtain a pre-aligned target image.
[0056] Step S3, performing recursive thin plate spline transformation on the reference image and the pre-aligned target image to obtain a final transformation flow.
[0057] Step S4, transforming the pre-aligned target image based on the final transformation flow to obtain an accurately aligned target image.
[0058] Step S5, performing linear fusion on the accurately aligned target image and the reference image to generate a spliced image.
[0059] Specifically, Figure 2 A flowchart of the unsupervised image splicing method based on recursive thin plate spline transformation according to another embodiment of the application is shown in Figure 2 Step S3, performing recursive thin plate spline transformation on the reference image and the pre-aligned target image to obtain a final transformation flow, including:
[0060] Step S31, the reference image and the pre-aligned target image are input into a feature extraction network to obtain a reference image feature map and an initial target image feature map. Here, the feature extraction network takes resnet50 as a backbone network to extract a feature map with a resolution of 1 / 8 of the original image.
[0061] Step S32, for the current transformation, the reference image feature map and the target image feature map of the previous transformation are input into a context-related layer to calculate the correlation between the two feature maps; if it is the first time, the target image feature map of the previous transformation is the initial target image feature map.
[0062] Step S33, the correlation is input into a TPS (Thin Plate Spline) estimator to obtain a 10x10 control point motion result. Here, the TPS estimator includes Block1, Block1, Block2, Block2, Block2 connected in sequence, Figure 3 The structure diagram of the TPS estimator is shown, Figure 4 The structure diagrams of Block1 and Block2 are shown, see Figure 4 Block1 includes convolution layer Conv, normalization layer Group_Norm, ReLU activation layer and pooling layer MaxPool connected in sequence, and Block2 includes convolution layer Conv, normalization layer Group_Norm and ReLU activation layer connected in sequence.
[0063] Step S34, the control point motion result is converted by a quadratic interpolation conversion method to obtain a TPS transformation flow. This step is realized based on a quadratic interpolation conversion module.
[0064] Step S35, the transformation flow of the previous transformation is converted based on the TPS transformation flow to obtain a converted transformation flow; the converted transformation flow is added to the TPS transformation flow to obtain the current transformation flow; if it is the first time, the transformation flow of the previous transformation is initialized as an all-zero transformation flow. This step is realized based on a transformation flow generation module.
[0065] Step S36, the target image feature map of the previous transformation is transformed by the current transformation flow to obtain the target image feature map of the current transformation.
[0066] Step S37, return to step S32 for the next transformation until the set number of transformations is reached to obtain a final transformation flow. The number of control points is the same for each transformation, but the control points are different.
[0067] In this embodiment, the transformation flow under the final condition is obtained by repeating the transformation multiple times, and the multiple basic sheet metal transformation is aggregated into a more flexible and powerful transformation, and the total transformation can be regarded as a complex deviation aggregated by a plurality of basic transformations of a limited number of control points.
[0068] In one embodiment, the reference image and the pre-aligned target image are subjected to recursive sheet metal transformation, and a sheet metal estimation network is used to estimate the sheet metal, which includes a feature extraction network, a context-related layer, a TPS estimator, a quadratic interpolation conversion module and a transformation flow generation module.
[0069] The sheet metal estimation network is a trained network, and the parameter optimizer used during training is Adam, the initial learning rate is 0.0001, the weight decay factor is 0.01, and the data set is a natural image splicing data set.
[0070] The loss function used in the training process is:
[0071] Loss=W identity L identity +W SSIM L SSIM
[0072] Wherein, Loss is the loss function, W identity is the weight of the feature intensity loss, L identity is the feature intensity loss, W SSIM is the structural similarity loss, and W SSIM is the weight of the structural similarity loss. Here, the weights of the feature intensity loss and the transformation consistency loss are both 1 by default.
[0073] Specifically, the feature intensity loss L identity is:
[0074]
[0075] Wherein, I b is the reference image, is the pre-aligned target image, TPS i is the transformation flow of the i-th transformation, war[ represents that TPS i is used to transform, N is the number of recursive transformations, and ‖‖1 is the norm.
[0076] The structural similarity loss W SSIM is:
[0077]
[0078] Wherein, I b is the reference image, TPS for pre-alignment target image i warp denotes the transformation stream using TPS for the i-th transformation i transform operation, SSIM denotes the calculation of similarity between images, and N is the number of recursive transformations. transform operation, SSIM denotes the calculation of similarity between images, and N is the number of recursive transformations.
[0079] In order to further verify the effectiveness of the method of the present application, the following experimental analysis is carried out.
[0080] As shown in Table 1, the scheme of the present application is trained on the UDIS-D data set, and tested on the data set, by using PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) as evaluation indexes, which can better evaluate the effect of image alignment. The higher the PSNR and SSIM, the better the effect of image alignment, and the average value of the indexes on the test is used as the evaluation standard. The method of the present application compared with global homography transformation and global homography transformation plus single thin plate spline transformation respectively improves the PSNR by 2.58 and 0.42, and improves the SSIM by 0.061 and 0.004 respectively, which fully shows that the scheme of the present application firstly estimates the global homography transformation, then performs recursive thin plate spline transformation on the pre-aligned image, and uses the transformation stream as a unified bridge to aggregate multiple basic transformations into a more flexible and powerful transformation, and uses the final transformation stream for image registration, which effectively improves the accuracy of image registration and the accuracy of image stitching.
[0081] Table 1 Average Index Table
[0082]
[0083]
[0084] Figure 5 The experimental results under different scenes are shown. According to the experimental results, it can be seen that the scheme of the present application has better performance than the global homography transformation and the global homography transformation plus single thin plate spline transformation. Figure 5The first row of the table shows that the scheme is trained on the UDIS-D dataset and tested on the dataset. Using global homography transformation cannot achieve effective alignment of real scene images, resulting in serious stitching artifacts in the stitched image. Using global homography and single thin-plate spline transformation with a large number of control points shows limited flexibility and cannot achieve accurate image alignment. In addition, due to misalignment, distortion is introduced in other regions, making the stitching result inaccurate. The scheme first performs global alignment of the image to reduce the influence of the change in viewing angle, and then searches for the control point motion of multiple basic thin-plate spline transformations in a recursive manner. The transformation flow is used as a unified bridge to aggregate multiple basic transformations into a more flexible and powerful transformation. Using the transformation flow to register the image effectively improves the accuracy of image alignment, and effective alignment reduces distortion in other regions, effectively improving the accuracy of image stitching.
[0085] According to Figure 5 The second row of the table shows that the scheme is trained on the UDIS-D dataset and tested on the traditional dataset. Using global homography transformation cannot achieve stitching of real scenes with varying depths, resulting in misalignment artifacts in the stitched image. Using global homography transformation and single thin-plate spline transformation with a large number of control points cannot achieve effective alignment of the image, resulting in misalignment artifacts and misalignment. The scheme effectively improves the flexibility of image alignment by estimating global homography transformation and recursive thin-plate spline transformation, reduces misalignment artifacts and misalignment distortion in the stitched image, and improves the accuracy of the stitched image.
[0086] In addition, since there is no stitched image as a reference in the real scene, the scheme is unsupervised trained on the UDIS-D dataset using an unsupervised loss function, and tested on the dataset and other datasets. The results fully demonstrate that through unsupervised training, the scheme has stronger generalization ability and robust performance in real scenes, effectively improving the accuracy of image stitching in real scenes.
[0087] Based on the same inventive concept as the unsupervised image stitching method based on recursive thin-plate spline transformation, the embodiment also provides an unsupervised image stitching device based on recursive thin-plate spline transformation, which comprises:
[0088] A global homography estimation module is configured to estimate a global homography matrix between a reference image and a target image.
[0089] A first transformation module is configured to transform the target image based on the global homography matrix to obtain a pre-aligned target image.
[0090] a recursive thin plate spline transformation module, configured to perform recursive thin plate spline transformation on the reference image and the pre-aligned target image to obtain a final transformation flow;
[0091] a second transformation module, configured to perform transformation on the pre-aligned target image based on the final transformation flow to obtain an accurately aligned target image;
[0092] a fusion module, configured to perform linear fusion on the accurately aligned target image and the reference image to generate a stitched image.
[0093] The unsupervised image stitching device based on recursive thin plate spline transformation of the embodiment has the same inventive concept as the unsupervised image stitching method based on recursive thin plate spline transformation described above, and thus the specific implementation of the device can be seen from the embodiment part of the unsupervised image stitching method based on recursive thin plate spline transformation described above, and the technical effects thereof correspond to those of the method described above, which will not be repeated here.
[0094] The embodiment of the present application provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the unsupervised image stitching method based on recursive thin plate spline transformation.
[0095] The embodiment of the present application provides a computer program product, which includes computer programs / instructions. The computer programs / instructions are executed by a processor to implement the unsupervised image stitching method based on recursive thin plate spline transformation.
[0096] The above is only various embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
Claims
1. An unsupervised image stitching method based on recursive thin plate spline transformation, characterized in that: include: Estimate the global homography matrix between the reference image and the target image; transforming the target image based on the global homography matrix to obtain a pre-aligned target image; performing a recursive thin plate spline transform on the reference image and the pre-aligned target image to obtain a final transformed stream; transforming the pre-aligned target image based on the final transformation stream to obtain a precisely aligned target image; The precisely aligned target image and the reference image are linearly fused to generate a stitched image.
2. The method according to claim 1, wherein in, Performing a recursive thin plate spline transform on the reference image and the pre-aligned target image to obtain a final transform stream, comprising: Step S31, inputting the reference image and the pre-aligned target image into a feature extraction network to obtain a reference image feature map and an initial target image feature map; Step S32: for the current transformation, input the reference image feature map and the target image feature map of the previous transformation into a context-related layer, and calculate the correlation between the two feature maps; if the current transformation is the first transformation, the target image feature map of the previous transformation is the initial target image feature map; Step S33, inputting the correlation into a TPS estimator to obtain a control point motion result; Step S34, applying a quadratic interpolation conversion method to the control point movement results to obtain a TPS conversion stream; Step S35: converting the previous transformed stream based on the transformed stream converted by the TPS to obtain a converted transformed stream; adding the converted transformed stream to the transformed stream converted by the TPS to obtain a current transformed stream; if the current time is the first time, the previous transformed stream is a transformed stream initialized to all zeros; Step S36, using the current transformation stream to transform the target image feature map of the previous transformation to obtain the target image feature map of the current transformation; Step S37, returning to step S32, performing the next transformation, until the set number of transformations is reached, and obtaining the final transformed stream.
3. The method according to claim 2, wherein in, Performing a recursive thin plate spline transform on the reference image and the pre-aligned target image, based on a thin plate spline estimation network, wherein the thin plate spline estimation network includes a feature extraction network, a context-dependent layer, a TPS estimator, a quadratic interpolation conversion module, and a transform stream generation module; The thin plate spline estimation network is a trained network, and the loss function used in the training process is: Loss=W identity L identity +W SSIM L SSIM Among them, Loss is the loss function, W identity is the weight of feature strength loss, L identity is the feature strength loss, W SSIM is the structural similarity loss, W SSIM is the weight of the structural similarity loss.
4. The method according to claim 3, wherein The characteristic intensity loss L identity for: Among them, I b is the reference image, For pre-aligned target images, TPS i is the transformation flow of the i-th transformation, warp means TPS i right Perform the transformation operation, N is the number of recursive transformations, and ‖‖1 is the norm.
5. The method according to claim 3, wherein The structural similarity loss W SSIM for: Among them, I b is the reference image, For pre-aligned target images, TPS i is the transformation flow of the i-th transformation, warp means TPS i right Perform transformation operations, SSIM represents the similarity between calculated images, and N is the number of recursive transformations.
6. An unsupervised image stitching device based on recursive thin plate spline transformation, characterized in that: include: A global homography estimation module is used to estimate the global homography matrix between the reference image and the target image; a first transformation module, configured to transform the target image based on the global homography matrix to obtain a pre-aligned target image; a recursive thin plate spline transform module, configured to perform a recursive thin plate spline transform on the reference image and the pre-aligned target image to obtain a final transform stream; a second transformation module, configured to transform the pre-aligned target image based on the final transformation stream to obtain a precisely aligned target image; The fusion module is used to linearly fuse the precisely aligned target image and the reference image to generate a spliced image.
7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the unsupervised image stitching method based on recursive thin plate spline transformation according to any one of claims 1 to 5.
8. A computer program product, characterized in that The invention comprises a computer program / instruction, which, when executed by a processor, implements the unsupervised image stitching method based on recursive thin plate spline transformation according to any one of claims 1 to 5.