An image stitching method based on a multi-stage alignment network

Abstract

The application discloses an image splicing method based on a multi-stage alignment network, and comprises the following steps: S1, acquiring a training data set, wherein the training data set comprises a plurality of input image pairs and image splicing results corresponding to each input image pair, and the input image pair comprises a reference image I1 and a target image I2; S2, constructing an image splicing depth model; S3, training the image splicing depth model based on the training data set and an overall loss function to obtain a target image splicing depth model; and S4, splicing to-be-spliced images by using the target image splicing depth model to obtain an image splicing result. The application utilizes multi-stage alignment to perform deep optimization deformation on images, reduces image content distortion while keeping a joint smooth, and realizes accurate splicing of the images.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing and deep learning, and particularly relates to an image stitching method based on a multi-stage alignment network. BACKGROUND

[0002] As a key technology to obtain high-resolution wide-field panoramic images, image stitching aims to obtain multiple images with overlapping regions by rotating the camera, and stitch the images by feature matching and image fusion. However, when the image acquisition device rotates at a large angle or the scene is not coplanar, it may cause visual artifacts and misalignment in the stitched image. Therefore, how to ensure accurate alignment and natural smoothing of wide-field panoramic images is a challenging problem in image stitching.

[0003] In recent years, researchers have proposed a large number of image stitching methods. Traditional image stitching methods are divided into global alignment methods and spatially varying deformation methods. Global alignment methods match images using invariant local features and align images by establishing a mapping relationship through a homography matrix, such as the dual-homography estimation method, the smoothly varying affine method, etc. Spatially varying deformation methods divide images into uniform grids and obtain the best grid coordinates by optimizing content-based grid deformation functions, including the projective consistency method, the naturally adaptive method, etc. In recent years, researchers have proposed deep learning-based image stitching methods to improve stitching performance. For example, Nie et al. proposed an image stitching network based on global homography, and eliminated image artifacts by constructing a structure stitching stage and a content modification stage. Considering the importance of edge preservation, Dai et al. proposed an edge-guided fusion method for image stitching. Jong et al. designed a deep image rectification solution to maintain the linear and nonlinear structures of the image. However, the performance of these image stitching methods still needs to be further improved.

[0004] During the implementation of the present application, the inventors found that the prior art at least has the following disadvantages and deficiencies:

[0005] The method in the prior art usually aligns images by estimating a single depth mapping transformation, which cannot effectively handle large disparity scenes and may distort the global structure of the panoramic image; the existing method ignores the importance of image content and stitching seams in the image stitching process, which easily leads to inconsistent image content and discontinuous stitching seams. SUMMARY

[0006] The application designs an image stitching method based on a multi-stage alignment network. The application pre-aligns an input image pair by using a content-maintained deep homography estimation module and reduces content artifacts, further aligns the image pair by using an edge-assisted mesh deformation module and avoids seam distortion, constructs content consistency loss and seam smoothness loss to maintain the geometric structure of the image pair and reduce seam discontinuity in the overlapping area, and then predicts a high-quality image stitching result. The image stitching method realizes high-quality stitching of images, avoids content artifacts, and reduces seam distortion.

[0007] The image stitching method based on the multi-stage alignment network comprises the following steps:

[0008] In step S1, a training data set is obtained, wherein the training data set comprises a plurality of input image pairs and image stitching results corresponding to each input image pair, and the input image pair comprises a reference image I1 and a target image I2.

[0009] In step S2, an image stitching deep model is constructed.

[0010] In step S3, the image stitching deep model is trained based on the training data set and the overall loss function to obtain a target image stitching deep model.

[0011] In step S4, the target image stitching deep model is used to stitch the to-be-stitched images to obtain an image stitching result.

[0012] Optionally, the image stitching deep model comprises an image pre-alignment sub-model and an image alignment sub-model, wherein the image pre-alignment sub-model is used to pre-align the input image pair by using a content-maintained deep homography estimation module, and the image alignment sub-model is used to further align the pre-aligned input image pair by using an edge-assisted network.

[0013] Optionally, the deep homography estimation module is formed by interleaving a plurality of symmetric convolution layer units and a corresponding number of content-maintained attention modules, wherein the symmetric convolution layer unit comprises two convolution layers and one maximum pooling layer; the content-maintained attention module comprises a spatial attention module and a plurality of cross operation modules, the spatial attention module comprises two maximum pooling layers, two average pooling layers, one shared fully connected layer and one activation function layer.

[0014] Optionally, the edge-assisted network comprises a convolution layer, three multi-scale residual blocks, an up-sampling layer and a bottleneck layer.

[0015] Optionally, when the image pre-alignment sub-model is used to pre-align the input image pair:

[0016] inputting the input image pair into the deep homography estimation module to obtain output feature maps F corresponding to a reference image I1 and a target image I2 in the input image pair i R and F i T ;

[0017] based on the output feature maps F i R and F i T , a homography matrix is obtained by using a direct linear transformation method;

[0018] The reference image I1 and the target image I2 are deformed respectively by using a spatial deformation network, so as to realize pre-alignment of pixel positions in the overlapping area of the reference image I1 and the target image I2, wherein the pre-aligned input image pair is represented as:

[0019]

[0020]

[0021] wherein E represents a unit matrix, H represents a homography matrix, W STN (·, ·) represents an output of the spatial deformation network.

[0022] Optionally, when the image alignment sub-model is used to align the pre-aligned input image pair:

[0023] The basic feature maps of the pre-aligned reference image and the target image are obtained by using the convolution layer in the edge auxiliary network;

[0024] The edge feature maps of the pre-aligned reference image and the target image are extracted by using the edge auxiliary network;

[0025] The obtained edge feature maps of the pre-aligned reference image and the target image are respectively concatenated with the corresponding basic feature maps, so as to obtain fusion feature maps of the pre-aligned reference image and the target image;

[0026] Based on the fusion feature maps of the pre-aligned reference image and the target image, a feature flow of the pre-aligned reference image and the target image is calculated by using a context-related method;

[0027] The pre-aligned reference image and the target image and the corresponding feature flow are input into a deep grid deformation network, so as to obtain an aligned reference image and a target image wherein the aligned reference image and the target image are represented as:

[0028]

[0029]

[0030] wherein

[0031]

[0032] wherein, F 1conv and F 2conv represent the base feature maps of the pre-aligned image pair, F 1edge and F 2edge represent the edge feature maps of the pre-aligned image pair, F 1c and F 2c represent the fused feature maps of the pre-aligned image pair, [·,·] represents the concatenation operation, CCL(·,·) represents the context correlation method, W mesh (·,·) represents the deep grid deformation network, represents the pre-aligned reference image, represents the pre-aligned target image.

[0033] Optionally, the overall loss function includes a content consistency loss and a seam smoothness loss, and the overall loss function L All is represented as:

[0034] L All = aL cont + bL seam

[0035] wherein, L cont represents the content consistency loss, L seam represents the seam smoothness loss, and a and b are the weights of the content consistency loss and the seam smoothness loss, respectively.

[0036] Optionally, the content consistency loss is composed of a photometric loss term and a structural loss term.

[0037] Optionally, the photometric loss term L photo is represented as:

[0038] L photo = ||I F -I G ||1

[0039] wherein, I F and I G represent the final image stitching result and the true value, respectively, and ||·||1 represents the L1 norm;

[0040] The structural loss term L struc is represented as:

[0041]

[0042] wherein, represents a function of the conv1 i in the VGG-16 network, and ||·||2 represents an L2 norm.

[0043] Optionally, the seam smoothness loss L seam is represented as:

[0044] L seam =||E1-E 1G ||1+||E2-E 2G ||1

[0045] wherein,

[0046]

[0047] wherein, E1 and E2 are edge images of the aligned image pair, E 1G and E 2G are true values of the edge images of the aligned image pair obtained by using a curvature formula, E net (·) represents an output of an edge auxiliary network, represents a reference image after alignment, represents a target image after alignment, and m and n represent horizontal and vertical directions, and div(·) respectively represent gradient operation and divergence operation.

[0048] The technical scheme provided by the present application has the beneficial effects that:

[0049] 1. The present application can accurately align images, reduce image content distortion while keeping the seam smooth, and obtain high-quality image stitching results.

[0050] 2. The present application uses deep learning technology to solve the image stitching problem, reduces image alignment artifacts through a multi-stage alignment method, and reduces seam discontinuity by using edge information assistance and seam smoothness loss. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of an image stitching method based on a multi-stage alignment network according to an embodiment of the present application;

[0052] Fig. 2(a) is a schematic diagram of the content retention attention module structure according to an embodiment of the present application, wherein, represents pixel-level multiplication;

[0053] Fig. 2(b) is a schematic diagram of the spatial attention module structure according to an embodiment of the present application, wherein, represents pixel-level addition, represents a sigmoid function;

[0054] Figure 3 is a schematic diagram of an edge auxiliary network structure according to an embodiment of the present application;

[0055] Figure 4 is a schematic diagram of a structure similarity comparison result of different image stitching methods according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application with reference to the specific embodiments and the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. In addition, in the following description, the description of the known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present application.

[0057] Figure 1 is a flowchart of an image stitching method based on a multi-stage alignment network according to an embodiment of the present application, and the following will take Figure 1 as an example to illustrate some specific implementation processes of the present application. The image stitching method based on a multi-stage alignment network proposed by the present application includes the following steps:

[0058] Step S1, obtaining a training data set, wherein the training data set includes a plurality of input image pairs and an image stitching result corresponding to each input image pair, and the input image pair includes a reference image I1 and a target image I2;

[0059] Step S2, constructing an image stitching deep model;

[0060] In an embodiment of the present application, the image stitching deep model includes an image pre-alignment sub-model and an image alignment sub-model.

[0061] The image pre-alignment sub-model is used to utilize a content-preserving based deep homography estimation module to pre-align the input image pair to reduce content artifacts and obtain a pre-aligned input image pair.

[0062] Further, the deep homography estimation module is formed by interleaving a plurality of symmetric convolution layer units and a corresponding number of content-preserving attention modules, i.e., a content-preserving attention module is arranged between every two symmetric convolution layer units to find correct matching features and reduce false matching features, as shown in FIG. 2(a). The symmetric convolution layer unit includes two convolution layers and a Maxpooling layer; the content-preserving attention module includes a spatial attention module and a plurality of cross operation modules, as shown in FIG. 2(a); the spatial attention module includes two Maxpooling layers, two Avgpooling layers, a shared fully connected layer, and a sigmoid layer, as shown in FIG. 2(b).

[0063] Suppose the deep homography estimation module includes i+1 symmetric convolution layer units and i+1 content-preserving attention modules, for the deep homography estimation module, after the reference image I1 and the target image I2 are input into the first-level symmetric convolution layer unit in the deep homography estimation module, the first-level feature maps of the reference image I1 and the target image I2 can be generated and After the content-preserving attention module between the first-level symmetric convolution layer unit and the second-level symmetric convolution layer unit, the first-level weighted feature maps F0 corresponding to the reference image I1 and the target image I2 can be obtained R and F0 T After the second-level symmetric convolution layer unit, the second-level feature maps and After the content-preserving attention module between the second-level symmetric convolution layer unit and the third-level symmetric convolution layer unit, the second-level weighted feature maps F1 corresponding to the reference image I1 and the target image I2 can be obtained R and F1 T After the i+1 content-preserving attention module connected with the last-level, i.e., the i+1 level symmetric convolution layer unit, the output feature maps F corresponding to the reference image I1 and the target image I2 can be obtained i R and F i T As shown in FIG. 2(a), the output feature maps F i R and F i T can be represented as:

[0064]

[0065]

[0066] wherein

[0067]

[0068] wherein, and respectively represent the (i+1)-th level feature map of the reference image and the target image, and respectively represent the spatial level feature map of the reference image and the target image obtained by pixel-wise multiplication of the (i+1)-th level feature map and the corresponding spatial attention mask, i.e. the output of the corresponding spatial attention module, M s (·) represents the spatial attention mask, represents pixel-wise multiplication.

[0069] When pre-aligning the input image pair by using the image pre-alignment submodel, the method comprises the following steps:

[0070] Firstly, the input image pair is input into the deep homography estimation module to obtain output feature maps F i R and F i T ;

[0071] Then, based on the output feature maps F i R and F i T , a homography matrix is obtained by using a direct linear transformation method, wherein obtaining the homography matrix by using the direct linear transformation method belongs to the technology that should be mastered by those skilled in the art, and the present application does not make superfluous repetition;

[0072] Then, the spatial deformation network is used to deform the reference image I1 and the target image I2 respectively, i.e. to realize the pre-alignment of the pixel positions in the overlapping area of the reference image I1 and the target image I2, wherein the pre-aligned input image pair can be represented as:

[0073]

[0074]

[0075] wherein E represents a unit matrix, H represents a homography matrix, W STN (·, ·) represents the output of the spatial deformation network.

[0076] wherein the image alignment submodel is used to further align the pre-aligned input image pair by using an edge auxiliary network to reduce seam distortion.

[0077] Further, the edge auxiliary network is an edge auxiliary based mesh deformation module, which includes a convolution layer, three multi-scale residual blocks, an up-sampling layer and a bottleneck layer, as shown in Figure 3

[0078] In the alignment of the pre-aligned input image pair by using the image alignment sub-model:

[0079] First, the basic feature maps of the pre-aligned reference image and the target image are obtained by using the convolution layer in the edge auxiliary network;

[0080] Then, the edge feature maps of the pre-aligned reference image and the target image are extracted by using the edge auxiliary network;

[0081] Then, the obtained edge feature maps of the pre-aligned reference image and the target image are respectively concatenated with the corresponding basic feature maps to obtain the fusion feature maps of the pre-aligned reference image and the target image;

[0082] Then, based on the fusion feature maps of the pre-aligned reference image and the target image, the feature flow of the pre-aligned reference image and the target image is calculated by using a context correlation method;

[0083] Finally, the pre-aligned reference image and the target image and their corresponding feature flow are input into a deep mesh deformation network to obtain the aligned reference image and the target image Wherein, the aligned reference image and the target image Can be represented as:

[0084]

[0085]

[0086] Wherein

[0087]

[0088] Wherein, F 1conv and F 2conv respectively represent the basic feature maps of the pre-aligned image pair, F 1edge and F 2edge respectively represent the edge feature maps of the pre-aligned image pair, F 1c and F 2c respectively represent the fusion feature maps of the pre-aligned image pair, [·,·] represents the concatenation operation, CCL(·,·) represents the context correlation method, W mesh (·,·) represents the output of the deep mesh deformation network, represents the pre-aligned reference image, represents the pre-aligned target image.​

[0089] Step S3: training the image stitching depth model based on the training data set and the overall loss function to obtain a target image stitching depth model;

[0090] In an embodiment of the present application, the overall loss function comprises a content consistency loss and a seam smoothness loss, so as to maintain the geometric structure of the image pair and reduce the seam discontinuity of the overlapping area by using the content consistency loss and the seam smoothness loss.

[0091] Wherein, the overall loss function L All may be expressed as:

[0092] L All = αL cont + βL seam

[0093] Wherein, L cont represents the content consistency loss, L seam represents the seam smoothness loss, and α and β are the weights of the content consistency loss and the seam smoothness loss, respectively.

[0094] In an embodiment of the present application, the weights α and β can be set to 0.5.

[0095] Wherein, the content consistency loss is composed of a photometric loss term and a structural loss term, the photometric loss term is used to minimize the pixel difference between the image stitching result and the ground truth, and the structural loss term is used to constrain the image stitching result and the ground truth to have similar feature representations.

[0096] Further, the photometric loss term L photo may be expressed as:

[0097] L photo = ||I F -I G ||1

[0098] Wherein, I F and I G represent the final image stitching result and the ground truth, respectively, and ||·||1 represents the L1 norm.

[0099] The structural loss term L struc may be expressed as:

[0100]

[0101] Wherein, represents the conv1 iwhere f is a function available to those skilled in the art, the receptive field of each pixel in conv11 and conv12 is a 5x5 neighborhood, and ||·||2 represents the L2 norm.

[0102] wherein the seam smoothness loss is used to constrain the edge image of the aligned image pair to be close to the edge image ground truth of the aligned image pair.

[0103] Further, the seam smoothness loss L seam may be represented as:

[0104] L seam =||E1-E 1G ||1+||E2-E 2G ||1

[0105] wherein

[0106]

[0107] wherein E1 and E2 are the edge images of the aligned image pair, E 1G and E 2G are the edge image ground truths of the aligned image pair obtained using the curvature formula, E net (·) represents the output of the edge auxiliary network, and m and n represent the horizontal direction and the vertical direction, and div(·) represent gradient operation and divergence operation, respectively.

[0108] In step S4, the target image stitching depth model is used to stitch the to-be-stitched images to obtain a high-quality image stitching result.

[0109] Figure 4 Structural similarity comparison results of image stitching results obtained by different methods are listed, and the comparison algorithms include the method of Zaragoza and the method of Zhao, wherein the method of Zaragoza is a traditional image stitching method, and the method of Zhao is an image stitching method based on deep learning. The greater the structural similarity, the higher the quality of the image stitching result. It can be seen from Figure 4 It can be seen that the structural similarity of the present application is greater than that of the method of Zaragoza, which shows the important role of the deep homography model based on content preservation in image stitching. In addition, the method of Zhao also performs worse than the present application in terms of structural similarity. The main reason is that the method of Zhao only uses a single deep homography for image alignment, which will produce undesirable alignment distortion, and further cause discontinuous seams. In contrast, the present application reduces the content artifacts and seam distortion of the image stitching result by constructing a multi-stage alignment model and combining the content consistency loss and the seam smoothness loss.

[0110] The model of each device is not limited unless otherwise specified, and any device that can complete the above functions can be used.

[0111] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred embodiment, and the above embodiment numbers are only for description, not representing the advantages and disadvantages of the embodiments.

[0112] The above is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image stitching method based on a multi-stage alignment network, characterized in that, The method comprises the following steps: In step S1, a training data set is obtained, wherein the training data set includes a plurality of input image pairs and image stitching results corresponding to each input image pair, the input image pair includes a reference image and a target image ; Step S2, constructing an image stitching depth model; Step S3, training the image stitching depth model based on the training data set and the overall loss function to obtain a target image stitching depth model; Step S4, stitching the to-be-stitched images using the target image stitching depth model to obtain an image stitching result; Wherein, the image stitching depth model comprises an image pre-alignment sub-model and an image alignment sub-model, wherein the image pre-alignment sub-model is used to pre-align the input image pair using a content-preserving depth homography estimation module, and the image alignment sub-model is used to further align the pre-aligned input image pair using an edge auxiliary network; Wherein, the depth homography estimation module is formed by interleaving a plurality of symmetric convolution layer units and a corresponding number of content-preserving attention modules, and a content-preserving attention module is arranged between every two symmetric convolution layer units to find correct matching features and reduce false matching features, wherein the symmetric convolution layer unit comprises two convolution layers and a max-pooling layer; the content-preserving attention module comprises a spatial attention module and a plurality of cross operation modules, the spatial attention module comprises two max-pooling layers, two average-pooling layers, a shared fully connected layer and an activation function layer; Wherein, the edge auxiliary network is a grid deformation module based on edge assistance, which comprises a convolution layer, three multi-scale residual blocks, an up-sampling layer and a bottleneck layer.

2. The method of claim 1, wherein, When pre-aligning the input image pair using the image pre-alignment sub-model: inputting the input image pair into the deep homography estimation module to obtain corresponding output feature maps of a reference image and a target image in the input image pair and the target image corresponding output feature maps and ; based on the output feature map and a homography matrix is obtained using a direct linear transformation method. Using spatial deformation networks to separate reference images and target image Perform deformation to achieve the reference image and target image Pre-alignment of pixel positions in overlapping regions, where the pre-aligned input image pairs are represented as: wherein, E denotes the identity matrix, H denotes the homography matrix, denotes the output of the spatial deformation network.

3. The method of claim 1, wherein, When aligning the pre-aligned input image pair using the image alignment sub-model: The convolution layer in the edge auxiliary network is used to obtain the base feature maps of the pre-aligned reference image and the target image; The edge auxiliary network is used to extract the edge feature maps of the pre-aligned reference image and the target image; The obtained edge feature maps of the pre-aligned reference image and the target image are respectively concatenated with the corresponding base feature maps to obtain the fusion feature maps of the pre-aligned reference image and the target image; Based on the fusion feature maps of the pre-aligned reference image and the target image, the context-related method is used to calculate the feature flow of the pre-aligned reference image and the target image. feeding the pre-aligned reference image and target image and their corresponding feature flows into a deep grid deformation network to obtain an aligned reference image and target image wherein the aligned reference image and target image is represented as: Wherein wherein, and respectively represent base feature maps of a pre-aligned image pair, and respectively represent edge feature maps of a pre-aligned image pair, and respectively represent fused feature maps of a pre-aligned image pair, represents a concatenation operation, represents a context-related method, represents a deep mesh deformation network, represents a pre-aligned reference image, represents a pre-aligned target image.

4. The method of claim 1, wherein, The overall loss function includes a content consistency loss and a seam smoothness loss, the overall loss function L All is represented as: wherein, L cont represents a content consistency loss, L seam represents a seam smoothness loss, and are weights for the content consistency loss and the seam smoothness loss, respectively.

5. The method of claim 4, wherein, The content consistency loss is composed of a photometric loss term and a structure loss term.

6. The method of claim 5, wherein, the photometric loss term is represented as: wherein, and denote the final image stitching result and the ground truth, respectively, denotes the L1 norm; The structural loss term is represented as: wherein, represents a function of the VGG-16 network conv1 i represents the L2 norm.​ 7. The method of claim 4, wherein, The loss of seam smoothness L seam is represented as: Wherein wherein, and are edge images of the aligned image pair, and are ground truths of the edge images of the aligned image pair obtained by using a curvature formula, denotes an output of the edge auxiliary network, denotes a reference image of the alignment, denotes a target image of the alignment, m and n denote a horizontal direction and a vertical direction, and denote a gradient operation and a divergence operation, respectively.

Images