A seam-driven stitching method integrating scene depth information

Abstract

The present invention relates to a seam-driven stitching method that fuses scene depth information. By constructing a difference cost matrix that fuses the color information, texture information, and depth information of the scene, the seam can be selected in the overlapping area. The seam can pass through highly similar continuous areas of the images and avoid passing through protruding objects, thereby effectively solving the problem of structural distortion that is prone to occur in traditional seam driving. It can also solve the problems of artifacts, visible seams, etc. in image stitching, reasonably and effectively improve the image stitching quality, and generate a stitched image that is well perceived by the human eye.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to a seam-driven stitching method integrating scene depth information. Background Art

[0002] Image stitching combines images with a certain overlap ratio to create an image with a wide field of view. It has a wide range of applications in multimedia content generation, computer vision, image processing, and other fields. Examples include creating panoramic images, generating aerial images, generating medical composite images, and virtual reality.

[0003] Image stitching methods mainly include image preprocessing, image registration, image fusion and other steps. In the image registration stage, a transformation model is calculated for the two images to be stitched in order to align the two images. In the image fusion stage, the registered images are fused to obtain the final stitched image. Traditional image stitching methods are mostly based on the basic assumption that the images are taken at the same depth. However, in actual shooting, it is difficult to ensure that the captured scenes are at the same depth. Such images usually have large parallax. For images with relatively large parallax, it is difficult to achieve precise alignment of the overlapping areas in the image registration stage, so artifacts will be introduced in the image fusion stage, which will seriously affect the perceptual quality of the stitched image. To this end, it is necessary to design a reasonable image fusion strategy to eliminate the influence of artifacts on the stitching results. Summary of the Invention

[0004] The problem solved by the present invention is how to solve the problem of artifacts and splicing seams in image splicing.

[0005] To solve the above problems, the present invention provides a seam-driven stitching method that integrates scene depth information, comprising:

[0006] Step 1: Collect two left viewpoint images I to be stitched L and right viewpoint image I R ;

[0007] Step 2: Use feature matching method to stitch the left viewpoint image I L and right viewpoint image I R Twisted to the same coordinate system, denoted as I0 and I1 respectively;

[0008] Step 3: Based on the coordinate relationship between I0 and I1, intercept the overlapping area between I0 and I1;

[0009] Step 4: Define a difference cost matrix S(·) based on the overlapping area, where the difference cost matrix S(·) integrates the color information, texture information, and depth information in the image;

[0010] Step 5: Construct energy function based on overlapping area and difference cost matrix

[0011] Step 6: Use image segmentation to minimize The path with the minimum cost is used as the suture line;

[0012] Step 7: Based on the stitching line, the pixels in the overlapping area are fused to obtain the stitched image.

[0013] The beneficial effects of the present invention are as follows: by constructing a difference cost matrix that integrates the color information, texture information and depth information of the scene, the seam line selected in the overlapping area can avoid passing through protruding objects when passing through continuous areas with highly similar images, thereby effectively solving the problem of structural distortion that is prone to occur in traditional seam line driving; it can also solve the problems of artifacts, visible seams and other problems in image stitching, reasonably and effectively improve the image stitching quality, and generate a stitched image that is well perceived by the human eye.

[0014] As an example, in step 2, the left viewpoint image I to be stitched is L and right viewpoint image I R Distorting to the same coordinate system specifically includes:

[0015] Step 201: Use the Single-perspective warps method to treat the spliced left viewpoint image I L and right viewpoint image I R Perform pre-alignment;

[0016] Step 202: Take the aligned left viewpoint image I L and right viewpoint image I R Warped to the same coordinate system, they are I0 and I1 respectively.

[0017] Preferably, the step 4 specifically includes:

[0018] Step 401: perform monocular depth estimation on I0 and I1 to obtain corresponding depth information D0 and D1 respectively;

[0019] Step 402: Perform texture detection on I0 and I1 to obtain corresponding texture information T0 and T1 respectively;

[0020] Step 403: Define a difference cost matrix that integrates the depth information, texture information, and color information of the scene:

[0021] S(·)=||C color (·)||2+||C texture (·)||2+||C depth (·)||2

[0022] Where C color (·) is the color difference cost; C texture (·) is the texture difference cost; C depth (·) is the texture color difference cost of fusion depth information;

[0023] C color The calculation formula of (·) is:

[0024] C color (·)=I0(·)-I1(·)

[0025] Where I0(·) and I1(·) represent the color values of the corresponding pixels in the overlapping area of I0 and I1, respectively;

[0026] C texture The calculation formula of (·) is:

[0027] C texture (·)=T0(·)-T1(·)

[0028] Where T0(·) and T1(·) represent the texture values of the corresponding pixels in the overlapping area of T0 and T1 respectively;

[0029] C depth The calculation formula of (·) is:

[0030]

[0031] Wherein, D0(·) and D1(·) represent the depth values of the corresponding pixels in the overlapping area of D0 and D1 respectively.

[0032] As an example, the step 5 specifically constructs an energy function for:

[0033]

[0034] Where N is the number of pixels in the overlapped area; the data item E is d (p,l p ) represents assigning a label l to pixel p∈P p The cost of smoothing It is represented as a pair of pixels (p,q)∈N assigned label (l p ,l q )

[0035] Data item E d (p,l p ) is calculated as:

[0036]

[0037] Among them, λ m For penalty items;

[0038] Smoothness term The calculation formula is:

[0039] BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of the present invention;

[0041] Figure 2 This is the effect diagram of the distortion to the same coordinate system in step 2 of the present invention;

[0042] Figure 3 This is a graph showing the detection result of image depth information in step 4 of the present invention;

[0043] Figure 4 This is a detection result diagram of image texture information in step 4 of the present invention;

[0044] Figure 5 The fusion result of step 7 of the present invention and the position of the suture line found;

[0045] Figure 6a Take images for the left and right viewpoints to be stitched in the experiment;

[0046] Figure 6b This is the heat map of the difference cost of overlapping areas solved in the experiment;

[0047] Figure 6c This is the effect diagram of the APAP method used for experimental comparison;

[0048] Figure 6d This is the effect diagram of the AANAP method used for experimental comparison;

[0049] Figure 6e The effect diagram of the REW method is used for experimental comparison;

[0050] Figure 6f The effect diagram of the SPW method is used for experimental comparison;

[0051] Figure 6g The effect diagram of the proposed method is used for experimental comparison;

[0052] Figure 6h For experimental comparison, the suture line position of the proposed method was adopted;

[0053] Figure 7a This is the effect diagram of the parallax tolerance method for robust elastic deformation proposed by Li et al. in the prior art;

[0054] Figure 7b The effect diagram of the present invention is shown. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0056] like Figures 1 to 5 As shown, a seam-driven stitching method integrating scene depth information includes:

[0057] Step 1: Collect two left viewpoint images I to be stitched L and right viewpoint image I R ;

[0058] Step 2: Use feature matching method to stitch the left viewpoint image I L and right viewpoint image I R Distorted to the same coordinate system, denoted as I0 and I1 respectively; specifically including:

[0059] Step 201: Use the Single-perspective warps method to treat the spliced left viewpoint image I L and right viewpoint image I R Perform pre-alignment;

[0060] Step 202: Take the aligned left viewpoint image I L and right viewpoint image I R Distorted to the same coordinate system, they are I0 and I1 respectively;

[0061] In this specific embodiment, the Single-perspective warps method is used to align the left viewpoint image I L and right viewpoint image I R The specific implementation steps of warping it to the same coordinate system are prior art and will not be described in detail here.

[0062] Step 3: Based on the coordinate relationship between I0 and I1, intercept the overlapping area between I0 and I1;

[0063] Step 4: Define a difference cost matrix S(·) based on the overlapping area. The difference cost matrix S(·) integrates the color information, texture information, and depth information in the image. Specifically, it includes:

[0064] Step 401: perform monocular depth estimation on I0 and I1 to obtain corresponding depth information D0 and D1 respectively;

[0065] Step 402: Perform texture detection on I0 and I1 to obtain corresponding texture information T0 and T1 respectively;

[0066] Step 403: Define a difference cost matrix that integrates the depth information, texture information, and color information of the scene:

[0067] S(·)=||C color (·)||2+||C texture (·)||2+||C depth (·)||2

[0068] Where C color (·) is the color difference cost; C texture (·) is the texture difference cost; C depth (·) is the texture color difference cost of fusion depth information;

[0069] C color The calculation formula of (·) is:

[0070] C color (·)=I0(·)-I1(·)

[0071] Where I0(·) and I1(·) represent the color values of the corresponding pixels in the overlapping area of I0 and I1, respectively;

[0072] C texture The calculation formula of (·) is:

[0073] C texture (·)=T0(·)-T1(·)

[0074] Where T0(·) and T1(·) represent the texture values of the corresponding pixels in the overlapping area of T0 and T1 respectively;

[0075] C depth The calculation formula of (·) is:

[0076]

[0077] Where D0(·) and D1(·) represent the depth values of the corresponding pixels in the overlapping area of D0 and D1 respectively;

[0078] Step 5: Construct energy function based on overlapping area and difference cost matrix for:

[0079] Constructing the energy function for:

[0080]

[0081] Where N is the number of pixels in the overlapped area; the data item E is d (p,l p ) represents assigning a label l to pixel p∈P p The cost of smoothing It is represented as a pair of pixels (p,q)∈N assigned label (l p ,l q )

[0082] Data item E d (p,l p ) is calculated as:

[0083]

[0084] Among them, λ m For penalty items;

[0085] Smoothness term The calculation formula is:

[0086]

[0087] Step 6: Use image segmentation to minimize The path with the minimum cost is used as the suture line;

[0088] Step 7: Based on the stitching line, the pixels in the overlapping area are fused to obtain the stitched image.

[0089] experiment

[0090] In order to further illustrate the feasibility and effectiveness of the method of the present invention, experiments were conducted on the method of the present invention.

[0091] The method of the present invention is compared with the existing image stitching methods. Existing image stitching methods are divided into two technical routes: one is the spatially varying warping method (Spatially-varying warping), and the other is the seam-driven image stitching method (Seam-driven image stitching). For the sake of fairness, the method of the present invention is compared with these two methods respectively. Among them, the spatially varying warping methods used for comparison are: the As-Projective-As-Possible Image Stitching (APAP) image stitching method proposed by Zaragoza et al., the Adaptive As-Natural-As-Possible Image Stitching (AANAP) image stitching method proposed by Lin et al., and the Parallax-Tolerant Image Stiching Basedon Robust Elastic Warping (REW) image stitching method proposed by Li et al. The single-perspective warps in natural image stiching (SPW) stitching method proposed by Liao et al. The seam-driven method used for comparison is the perception-based seam cutting for image stitching proposed by Li et al. The image data used for comparison comes from the public databases Parallax and SEAGULL.

[0092] FIG6 shows the comparison results between the method of the present invention and the spatial deformation rendering type image stitching method:

[0093] Figure 6a is the input image; Figure 6b It is the overlapping area difference cost map solved by the method of the present invention. In order to show the effect, it is displayed in the form of a heat map; Figure 6c It is the result of APAP method; Figure 6d is the AANAP method result; Figure 6e is the result of REW method; Figure 6f is the result of SPW method; Figure 6g The results of the proposed method; Figure 6h The suture line positions of the proposed method.

[0094] Respectively Figure 6g to Figure 6c and Figure 6fBy comparison, it can be clearly seen that the existing spatial deformation rendering type image stitching method will have serious artifact distortion; in contrast, the stitching result generated by the method of the present invention has a clear texture structure and can generate a stitching image that is well perceived by the human eye.

[0095] To verify the rationality of the proposed method for seam position, the present invention compared the perception-based seam cutting for image stitching. To ensure the rationality of the experiment, all methods used the registration module of the SPW method during the image registration stage. Because the area through which the seam passes may be geometrically distorted, the present invention focuses on comparing the final stitching results in the area through which the seam passes.

[0096] Will Figure 7a and Figure 7b For comparison, Figure 7a The results of Li et al.'s method are shown. Figure 7b The results of our invention are shown in Figure 2. The red magnified area shows geometric distortion and broken steps in Li et al.'s results. Furthermore, the white clouds above the tallest building are cut off in the final stitching result. In contrast, our method does not exhibit this phenomenon, generating a geometrically complete stitched image. The generated stitching lines also reflect changes in scene depth.

[0097] In order to quantitatively evaluate the rationality of the seams found by the proposed method, this paper uses the current evaluation method of seam drive type for evaluation. Specifically, for each pixel p on the seam i , extract a p i Then, let S ZNCC Represents the zero-normalized cross-correlation score between the local image blocks of the left image and the right image. The specific calculation method is as follows:

[0098]

[0099] Where L(x,y) and R(x,y) represent the pre-aligned left and right images, i.e., I0 and I1, respectively, n is the number of pixels in the local patch, μ L and μ R is the average value of local patches in the pre-aligned left and right images, σ L and σ R are the variances of local patches in the pre-aligned left and right images, respectively;

[0100] Finally, along the seam line, the seam quality fraction can be expressed as:

[0101]

[0102] Among them, m is the total number of pixels on the seam. The smaller the quality score Q is, the more reasonable the seam position is and the better the final stitching effect is.

[0103] The performance of the method proposed by Li et al. and the method of the present invention was quantitatively measured using the quality score of the seam. A total of 24 sets of images were tested, all from the public datasets Parallax and SEAGULL. The objective quality scores Q were obtained as shown in Table 1 below:

[0104]

[0105] Table 1 Objective quality scores Q of joints obtained using different methods

[0106] Table 1 shows the quality scores of the proposed method compared with the method proposed by Li et al., where the best results are indicated in bold. From the data listed in Table 1, it can be seen that the proposed method outperforms the method proposed by Li et al. in most test images.

[0107] Although the present disclosure is disclosed as above, the protection scope of the present disclosure is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present disclosure, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A seam-driven stitching method integrating scene depth information, characterized in that: include: Step 1: Collect two left-viewpoint images to be stitched and right viewpoint image ; Step 2: Use feature matching method to stitch the left viewpoint image and right viewpoint image Distorted to the same coordinate system, respectively and ; Step 3: Based on and The coordinate relationship of and The overlapping area between Step 4: Define the difference cost matrix based on the overlapping area , the difference cost matrix It integrates the color information, texture information and depth information in the image; specifically includes: Step 401: and Perform monocular depth estimation and obtain corresponding depth information and ; Step 402: and Perform texture detection and obtain corresponding texture information and ; Step 403: Define a difference cost matrix that integrates the depth information, texture information, and color information of the scene: ； Where, is the cost of color difference; is the texture difference cost; The texture color difference cost for fusing depth information; The calculation formula is: ； in, and Respectively and The color values of the corresponding pixels in the overlapping area; The calculation formula is: ； in, and Respectively and The texture values of the corresponding pixels in the overlapping area; The calculation formula is: ； in, and Respectively and Depth values of corresponding pixels in the overlapping area; Step 5: Construct energy function based on overlapping area and difference cost matrix ; Step 6: Use image segmentation to minimize , taking the path with the minimum cost as the suture line; Step 7: Based on the stitching line, the pixels in the overlapping area are fused to obtain the stitched image.

2. The stitching line driven stitching method for integrating scene depth information according to claim 1, characterized in that: In step 2, the left viewpoint image to be spliced and right viewpoint image Distorting to the same coordinate system specifically includes: Step 201: Use the Single-perspective warps method to treat the stitched left viewpoint image and right viewpoint image Perform pre-alignment; Step 202: Take the aligned left viewpoint image and right viewpoint image Distorted to the same coordinate system, they are and .

3. The stitching line driven stitching method for integrating scene depth information according to claim 1, characterized in that: The step 5 specifically constructs the energy function for: ； Where, are the pixels in the overlapped area; the data items Represented as pixels Assign a label The cost of smoothing Represented as a pair of pixels Assigning Tags the price; Represents pixels Tags, Represents pixels 's label; Data Item The calculation formula is: ； in, For penalty items; Smoothness term The calculation formula is: ； Where, Represents pixels The difference cost matrix, Represents pixels The difference cost matrix of .

Images