Image stitching method, apparatus and device, and computer program product

Abstract

The invention discloses an image splicing method, apparatus and device, and a computer program product. The method comprises the steps of obtaining a plurality of images to be spliced and calculating an initial homography matrix of each image; performing image segmentation and depth estimation on each image to obtain an image segmentation result and a depth estimation result of each image; according to the image segmentation result, the depth estimation result and the initial homography matrix, calculating a corrected homography matrix of each image by using a weight-based homography matrix correction strategy; and performing post-processing on the plurality of images to be spliced according to the corrected homography matrix to obtain a spliced image. According to the method, image segmentation and depth estimation are performed on the images, and the homography matrix is corrected by using the homography matrix correction strategy based on the weight, so that error propagation among the plurality of spliced images is prevented, image splicing is completed with more accurate global alignment and higher visual consistency quality, and the image splicing efficiency is improved. And the accuracy and the visual effect of image splicing are improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image stitching method, apparatus and device, and computer program product. Background Technology

[0002] In image processing and related applications, accurate image stitching and transformation are crucial. To achieve high-quality image stitching and transformation, it is necessary to calculate the weights of objects in the image and precisely adjust the homography matrix during stitching to avoid problems such as image distortion or deformation. Currently, several related techniques exist to address these issues.

[0003] On the one hand, camera- and camera parameter-based technologies are a common solution, such as using stereo cameras. Stereo cameras acquire images through two cameras with different perspectives and use camera parameters to calculate depth information and the spatial relationships of objects. However, this technology has significant limitations. First, it requires additional equipment, namely the stereo camera, which increases hardware costs and system complexity. Second, it requires additional calibration of the camera parameters, a process that is not only cumbersome but also less correlated with the content of the image itself, making it difficult to flexibly adjust according to the actual image content. This limits its use in applications with specific requirements for image content.

[0004] On the other hand, techniques based on extensive computation and good initial values ​​are also widely used in image processing, such as bundle adjustment and global homography methods. Bundle adjustment is an iterative algorithm that optimizes camera parameters and 3D point coordinates by minimizing reprojection errors, while global homography methods attempt to find a global homography matrix to describe the transformation relationships between images. Although these methods can improve the accuracy of image processing to some extent, they also have significant drawbacks. These methods have relatively high computational costs and require substantial computational resources. Moreover, when applying the original homography matrix for image transformation, the transformation must be correct; otherwise, local optima are easily generated. Once local optima appear, they can lead to unnatural visual effects in the image, affecting image quality and subsequent application results. Summary of the Invention

[0005] This application provides an image stitching method, apparatus, device, and computer program product to improve the image stitching effect.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide an image stitching method, the image stitching method comprising:

[0008] Obtain multiple images to be stitched together and calculate the initial homography matrix of each image;

[0009] Image segmentation and depth estimation are performed on each image to obtain the image segmentation results and depth estimation results for each image;

[0010] Based on the image segmentation results and depth estimation results of each image and the initial homography matrix, the corrected homography matrix of each image is calculated using a weighted homography matrix correction strategy.

[0011] The multiple images to be stitched are post-processed based on the homography matrix of each image after correction to obtain the stitched image.

[0012] Optionally, calculating the initial homography matrix of each image includes:

[0013] Feature extraction is performed on the multiple images to be stitched together to obtain feature extraction results for the multiple images;

[0014] Feature matching is performed based on the feature extraction results of multiple images to obtain the feature matching results between pairs of images;

[0015] The initial homography matrix between each image and its neighboring images is calculated based on the feature matching results between each pair of images. There are overlapping regions between each image and its neighboring images.

[0016] Calculate the initial homography matrix between each image and a predefined reference image based on the initial homography matrix between each image and its neighboring images.

[0017] Optionally, the step of calculating the corrected homography matrix of each image using a weighted homography matrix correction strategy based on the image segmentation results and depth estimation results of each image and the initial homography matrix includes:

[0018] Based on the image segmentation results of each image, the object orientation is estimated to obtain the object orientation estimation result;

[0019] The object's angle before transformation is calculated based on the object's orientation estimation result, and the object is transformed using the initial homography matrix to obtain the object's angle after transformation.

[0020] Based on the absolute value of the angle difference between the object's angle before and after the transformation and the preset angle difference threshold, determine whether the object meets the preset correction conditions.

[0021] When the object meets the preset correction conditions, the corrected homography matrix of each image is calculated using a weighted homography matrix correction strategy based on the image segmentation results and depth estimation results of each image, the absolute value of the angle difference between the angle before and after transformation of the object that meets the preset correction conditions, and the initial homography matrix.

[0022] Optionally, the step of calculating the corrected homography matrix of each image using a weighted homography matrix correction strategy based on the image segmentation results and depth estimation results of each image, the absolute value of the angle difference between the angle before and after transformation of the object satisfying the preset correction conditions, and the initial homography matrix includes:

[0023] Based on the image segmentation results and depth estimation results of each image, calculate the object weight and image weight of each image;

[0024] The correction angle of each image is calculated based on the absolute value of the angle difference between the angle before and after transformation of the object that meets the preset correction conditions and the corresponding object weight.

[0025] Based on the correction angle and corresponding image weight of each image, as well as the initial homography matrix, the corrected homography matrix of each image is calculated.

[0026] Optionally, calculating the object weights and image weights of each image based on the image segmentation results and depth estimation results of each image includes:

[0027] The size, distance from the camera, and number of objects in each image are determined based on the image segmentation and depth estimation results of each image.

[0028] The object weights for each image are calculated based on the size of the objects in each image and their distance from the camera.

[0029] The image weights of each image are calculated based on the size of the objects in each image, their distance from the camera, and the number of objects.

[0030] Optionally, calculating the correction angle of each image based on the absolute value of the angle difference between the angle before and after transformation of the object satisfying the preset correction conditions and the corresponding object weight includes:

[0031] By utilizing the object weights of each image, the absolute values ​​of the angle differences between the angles before and after transformation of objects in each image that meet the preset correction conditions are summed in a weighted manner to obtain the correction angle of each image.

[0032] The correction angles of each image are converted into a correction angle matrix.

[0033] Optionally, the step of post-processing the multiple images to be stitched together based on the corrected homography matrix of each image to obtain the stitched image includes:

[0034] The images are transformed based on the homography matrix after correction to obtain the transformed images.

[0035] The transformed images are stitched together and rendered to obtain the stitched image.

[0036] Secondly, embodiments of this application also provide an image stitching device, the image stitching device comprising:

[0037] The acquisition unit is used to acquire multiple images to be stitched together and to calculate the initial homography matrix of each image;

[0038] The image segmentation and depth estimation unit is used to perform image segmentation and depth estimation on each image, and obtain the image segmentation results and depth estimation results for each image.

[0039] The correction unit is used to calculate the corrected homography matrix of each image based on the image segmentation results and depth estimation results of each image and the initial homography matrix, using a weighted homography matrix correction strategy.

[0040] The post-processing unit is used to perform post-processing on the multiple images to be stitched according to the homography matrix of each image after correction, so as to obtain the stitched image.

[0041] Thirdly, embodiments of this application also provide an apparatus, comprising:

[0042] A processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the aforementioned image stitching methods.

[0043] Fourthly, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement any of the aforementioned image stitching methods.

[0044] The image stitching method of this application embodiment achieves the following beneficial effects: First, it acquires multiple images to be stitched and calculates the initial homography matrix of each image; then, it performs image segmentation and depth estimation on each image to obtain the image segmentation result and depth estimation result of each image; subsequently, based on the image segmentation result and depth estimation result of each image and the initial homography matrix, it calculates the corrected homography matrix of each image using a weighted homography matrix correction strategy; finally, it performs post-processing on the multiple images to be stitched based on the corrected homography matrix of each image to obtain the stitched image. The image stitching method of this application embodiment effectively solves the image distortion and deformation problems caused by inaccurate homography matrices in the prior art by performing image segmentation and depth estimation on the images to be stitched and calculating the corrected homography matrix using a weighted homography matrix correction strategy. This method reduces the correction influence of the foreground dominant image, prevents error propagation between multiple stitched images, and can complete image stitching with more accurate global alignment and higher visual consistency quality, improving the accuracy of image information and visual effect. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a flowchart illustrating an image stitching method according to an embodiment of this application;

[0047] Figure 2 This is a schematic diagram of a feature matching result in an embodiment of this application;

[0048] Figure 3 This is a schematic diagram showing the calculation result of a correction angle in one embodiment of this application;

[0049] Figure 4 This is a schematic diagram comparing the image stitching effect before and after correction in an embodiment of this application;

[0050] Figure 5 This is a schematic diagram of the structure of an image stitching device according to an embodiment of this application;

[0051] Figure 6 This is a schematic diagram of the structure of a device according to an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0054] This application provides an image stitching method, such as... Figure 1 The diagram shows a flowchart of an image stitching method according to an embodiment of this application. The image stitching method includes the following steps S110 to S140:

[0055] Step S110: Obtain multiple images to be stitched together and calculate the initial homography matrix of each image.

[0056] When performing image stitching, it is necessary to first acquire multiple images to be stitched together. For example, in a logistics and warehousing scenario, multiple images to be stitched together are obtained by taking pictures of goods on a conveyor belt using a camera mounted on a robot.

[0057] The homography matrix characterizes the geometric relationship between multiple overlapping image frames. In the embodiments of this application, the initial homography matrix of each image can be understood as the initial homography matrix of each image relative to a predefined reference plane or reference image, which can be calculated based on the homography matrix between multiple overlapping image frames. The reference plane or reference image here is flexibly defined according to the actual application scenario. For example, the middle frame of multiple frames of images taken by a robot can be used as the reference image.

[0058] When calculating the homography matrix, traditional feature matching algorithms, such as Scale Invariant Feature Transform (SIFT) and Speed-Up Robust Feature Transform (SURF), can be used to extract feature points from the image. Then, by matching feature points from different images, robust estimation methods such as Random Sample Consensus (RANSAC) can be used to calculate the homography matrix.

[0059] Step S120: Perform image segmentation and depth estimation on each image to obtain the image segmentation results and depth estimation results for each image.

[0060] After acquiring multiple images, image segmentation and depth estimation are required for each image. Image segmentation can employ various image segmentation algorithms, such as semantic segmentation algorithms based on deep learning, including U-Net and Mask R-CNN. These algorithms can divide the image into different object regions based on the features of pixels in the image and generate corresponding masks, thereby obtaining the contour and position information of each object in the image.

[0061] Depth estimation can be achieved through methods such as binocular vision depth estimation, which utilizes the disparity between the left and right views obtained by a binocular camera to calculate depth information; alternatively, deep learning-based monocular depth estimation methods can be used, which directly predict the depth map from a single image using a trained neural network model. By performing image segmentation and depth estimation, depth maps and masks for all objects in the image can be obtained, providing crucial information for subsequent homography matrix correction.

[0062] Step S130: Based on the image segmentation results and depth estimation results of each image and the initial homography matrix, calculate the corrected homography matrix of each image using a weighted homography matrix correction strategy.

[0063] Based on the image segmentation results, depth estimation results, and initial homography matrix of each image, a weighted homography matrix correction strategy is used to calculate the corrected homography matrix. This strategy comprehensively considers factors such as the presence and importance of objects in the image, as well as the stitching requirements of the actual application scenario. The weighted homography matrix correction strategy can reduce the correction influence of the foreground dominant image and prevent error propagation between multiple stitched images, thus improving the accuracy of the homography matrix. Specifically, a weight function is constructed based on the object's depth information and the segmentation results to adjust the initial homography matrix, thereby obtaining the corrected homography matrix.

[0064] Step S140: Post-process the multiple images to be stitched according to the homography matrix of each image after correction to obtain the stitched image.

[0065] Based on the corrected homography matrix of each image, post-processing operations are performed on the multiple images to be stitched, mainly including image transformation, stitching, and rendering. The transformation operation transforms each image to a unified reference coordinate system according to the corrected homography matrix; the stitching operation stitches the transformed images together; the rendering operation adjusts the color, brightness, and other aspects of the stitched image to reduce distortion, ultimately presenting the final stitched panorama with improved global alignment and visual consistency.

[0066] The image stitching method of this application effectively solves the image distortion and deformation problems caused by inaccurate homography matrices in existing technologies by performing image segmentation and depth estimation on the images to be stitched, and then calculating the corrected homography matrix using a weighted homography matrix correction strategy. This method reduces the correction influence of the dominant foreground image, prevents error propagation between multiple stitched images, and enables image stitching with more accurate global alignment and higher visual consistency quality, thus improving the accuracy of image information and visual effects.

[0067] In some embodiments of this application, calculating the initial homography matrix of each image includes: performing feature extraction on the multiple images to be stitched to obtain feature extraction results for multiple images; performing feature matching based on the feature extraction results of multiple images to obtain feature matching results between pairs of images; calculating the initial homography matrix between each image and its adjacent images based on the feature matching results between pairs of images, wherein there are overlapping regions between each image and its adjacent images; and calculating the initial homography matrix between each image and a predefined reference image based on the initial homography matrix between each image and its adjacent images.

[0068] To perform feature extraction on multiple stitched images, various mature feature extraction algorithms can be employed. For example, the Scale Invariant Feature Transform (SIFT) algorithm can detect keypoints in different scale spaces and extract feature descriptors that are rotation-invariant and scale-invariant. The Speed-Up Robust Feature Transform (SURF) algorithm is also a commonly used choice, as it improves the speed of feature extraction through approximation while maintaining good robustness. By processing each image using these algorithms, the feature extraction results for each image are obtained, namely, the keypoints in each image and their corresponding feature descriptors.

[0069] Feature matching is performed based on feature extraction results from multiple images. For pairwise image matching, a feature matching algorithm, such as nearest neighbor matching, is used to calculate the distance between feature points in one image and feature points in another image, and the closest feature point pair is selected as the matching pair. Alternatively, the Random Sample Consensus (RANSAC) algorithm can be used to refine the initial matching results, removing mismatched point pairs to obtain accurate feature matching results. Figure 2 As shown, a schematic diagram of a feature matching result in an embodiment of this application is provided. The green line in the diagram indicates the matching features in the two images.

[0070] The initial homography matrix between each image and its neighboring images is calculated based on the feature matching results between pairs of images. It should be noted that in practical applications, image stitching requires that adjacent frames have sufficient overlap. If the multiple images to be stitched are obtained sequentially, feature points between adjacent frames can be directly matched. Using the successfully matched feature point pairs, robust estimation methods, such as least squares-based optimization algorithms, can be employed to calculate the homography matrix between adjacent frames. If the multiple images to be stitched are not in any order, the adjacent order of the images needs to be determined first. Then, following the pairwise matching method described above, the feature matching score between each frame and all other images is calculated. The image with the highest feature matching score is designated as its neighbor, and the homography matrix is ​​calculated based on the adjacent images.

[0071] The initial homography matrix calculated above refers to the homography matrix between each image and its neighboring images. It needs to be further transformed using a predefined reference image, that is, the homography matrix of each image relative to its neighboring images is transformed into its homography matrix relative to the reference image. This provides a globally consistent basis for subsequent correction and stitching processes. For example, assuming there are 5 images (1-5) to be stitched, and the 3rd image is set as the reference image, for the 1st image, multiplying the homography matrix of the 1st image relative to the 2nd image by the homography matrix of the 2nd image relative to the 3rd image yields the homography matrix of the 1st image relative to the 3rd image (the reference image). The same process is applied to other images, thus transforming the homography matrix of each image relative to its neighboring images into its homography matrix relative to the reference image.

[0072] The method for calculating the initial homography matrix of each image in this embodiment ensures the accuracy of homography matrix calculation between adjacent images through reasonable feature extraction and matching strategies. For unordered images, it accurately determines the adjacent order and calculates the homography matrix, then converts the homography matrices of adjacent images into homography matrices relative to the reference image. This provides a globally unified geometric basis for subsequent image stitching, effectively avoiding stitching errors caused by inaccurate homography matrices, improving the accuracy and stability of image stitching, and contributing to achieving high-quality image stitching results.

[0073] In some embodiments of this application, the step of calculating the corrected homography matrix of each image using a weighted homography matrix correction strategy based on the image segmentation results and depth estimation results of each image and the initial homography matrix includes: estimating the object orientation based on the image segmentation results of each image to obtain the object orientation estimation result; calculating the object's pre-transformation angle based on the object orientation estimation result, and transforming the object using the initial homography matrix to obtain the object's post-transformation angle; determining whether the object meets a preset correction condition based on the absolute value of the angle difference between the object's pre-transformation angle and post-transformation angle and a preset angle difference threshold; and, if the object meets the preset correction condition, calculating the corrected homography matrix of each image using a weighted homography matrix correction strategy based on the image segmentation results and depth estimation results of each image, the absolute value of the angle difference between the pre-transformation angle and post-transformation angle of the object that meets the preset correction condition, and the initial homography matrix.

[0074] After obtaining the segmented regions of objects in an image, orientation estimation can be performed using image feature analysis methods. For example, for objects with obvious texture or shape features, their edge information can be used to calculate the object's principal orientation to determine the orientation estimation result. Specifically, the Hough transform can be used to detect the linear features of the object's edges, and the principal orientation of the object can be determined based on the orientation distribution of these lines; alternatively, a machine learning-based method can be used to train an orientation estimation model, which takes the segmented image of the object as input and outputs the object's orientation estimation result.

[0075] Based on the object's orientation estimation, the angle between the object's orientation and the horizontal direction of the image is calculated, i.e., the object's untransformed angle θ. This step is mainly based on geometric calculations, calculated by determining the formula for the angle between the object's orientation vector and the horizontal direction vector.

[0076] The object is transformed using the initial homography matrix. The initial homography matrix describes the geometric transformation relationship between images. After transforming the object's coordinates using this matrix, the angle between the transformed object's orientation and the horizontal direction of the corresponding target image is recalculated to obtain the transformed angle θ' of the object.

[0077] Based on the absolute value of the angle difference between the object's angle before and after the transformation, Δθ = |θ - θ'|, and the preset angle difference threshold θ thd Determine whether the object meets the preset correction conditions. If △θ>θ thd If the result is positive, it indicates that the object's orientation has changed significantly after homography matrix transformation, satisfying the homography matrix correction condition; otherwise, it will not participate in subsequent correction calculations. The preset angle difference threshold can be reasonably set according to the actual application scenario and image characteristics.

[0078] Given that objects meet preset correction conditions, a weighted homography matrix correction strategy is used to calculate the corrected homography matrix for each image, based on the image segmentation and depth estimation results, the absolute value of the angle difference between the pre-transformed and post-transformed angles of objects meeting the preset correction conditions, and the initial homography matrix. This strategy comprehensively considers factors such as object segmentation information (e.g., object size and position), depth information (reflecting the object's distance in space), and the absolute value of the angle difference, assigning corresponding weights to each object meeting the correction conditions. Then, the initial homography matrix is ​​adjusted and optimized based on these weights to obtain the corrected homography matrix, thus more accurately describing the geometric transformation relationships between images.

[0079] This application's embodiments effectively filter out objects requiring correction by accurately estimating object orientation and calculating angles. By combining the object segmentation results, depth estimation results, and angle differences, weight allocation and homography matrix correction are performed. This enables more precise adjustment of the geometric transformation relationships between images, reducing image distortion and deformation caused by inaccurate homography matrices, improving the global alignment accuracy and visual consistency of image stitching, and providing a strong guarantee for high-quality image stitching.

[0080] In some embodiments of this application, the step of calculating the corrected homography matrix of each image using a weighted homography matrix correction strategy based on the image segmentation results and depth estimation results of each image, the absolute value of the angle difference between the pre-transformation angle and the post-transformation angle of the object satisfying the preset correction conditions, and the initial homography matrix includes: calculating the object weight and image weight of each image based on the image segmentation results and depth estimation results of each image; calculating the correction angle of each image based on the absolute value of the angle difference between the pre-transformation angle and the post-transformation angle of the object satisfying the preset correction conditions and the corresponding object weight; and calculating the corrected homography matrix of each image based on the correction angle, the corresponding image weight, and the initial homography matrix.

[0081] Object weights represent the weight of each object within its respective image. The object weights w are calculated based on the image segmentation results and depth estimation results of each image. obj Depth information reflects the distance of objects from the camera, and is combined with factors such as the size of the object's area in the image to determine the object's weight w. objThe strategy for calculating object weights can be adjusted according to application requirements. For example, when prioritizing foreground texture stitching, objects closer to the camera or with larger areas are given higher weights. This allows subsequent calculations to focus on these objects that have a greater impact on the geometric relationships of the image. In specific calculations, a weight calculation function can be defined, taking the object's depth and area values ​​as input parameters, and using the function to obtain the weight value for each object.

[0082] Image weights represent the weight of each image among all images, and are determined based on the presence of objects in the image. img If an image contains many large objects, it is assigned a smaller weight. The purpose of this is to reduce the impact of the image's deformation on other images in subsequent calculations.

[0083] The object's Δθ and corresponding object weight w that satisfy the preset correction conditions are obtained from the foregoing embodiments. obj The step of calculating the correction angle for each image takes into account the contribution of different objects to the image's orientation change, and obtains a more accurate image correction angle through weighted calculation. This image correction angle represents the angle adjustment required for the image as a whole to correct the orientation deviation caused by the inaccuracy of the initial homography matrix. It is used to subsequently adjust the initial homography matrix to obtain a more accurate corrected homography matrix, thereby optimizing the geometric alignment effect of image stitching. Figure 3 As shown, a schematic diagram illustrating the calculation results of a correction angle in an embodiment of this application is provided.

[0084] Based on the correction angle, corresponding image weights, and initial homography matrix of each image, the corrected homography matrix of each image is calculated. By incorporating the correction angle and image weights into the calculation process and adjusting the initial homography matrix, a more accurate corrected homography matrix describing the geometric transformation relationship between images can be obtained.

[0085] This application embodiment, by reasonably calculating object weights and image weights, fully considers the characteristics of objects and the importance of images in the overall stitching, which can effectively reduce deformation and distortion during the image stitching process, improve the global alignment accuracy of image stitching, and make the stitched image more natural and accurate in terms of geometric relationships and visual effects, providing key technical support for high-quality image stitching.

[0086] In some embodiments of this application, the step of calculating the object weight and image weight of each image based on the image segmentation results and depth estimation results of each image includes: determining the size, distance from the camera, and number of objects in each image based on the image segmentation results and depth estimation results of each image; calculating the object weight of each image based on the size and distance from the camera of each object; and calculating the image weight of each image based on the size, distance from the camera, and number of objects in each image.

[0087] Based on the image segmentation and depth estimation results for each image, the size, distance from the camera, and number of objects in each image are determined. The size of objects can be determined through pixel statistics in the image segmentation results; the distance from the camera can be determined using depth estimation results; and the number of objects is determined by counting the segmented objects.

[0088] The weights of objects in each image are calculated based on their size and distance from the camera. The weighting strategy can be flexibly adjusted according to actual application needs. For example, when prioritizing foreground texture stitching, objects closer to the camera or with larger areas are assigned higher weights. Specifically, a weight calculation function can be defined, taking the object's size and distance from the camera as input parameters, and the function calculates the weight value w for each object. obj .

[0089] Image weights are calculated for each image based on the size, distance from the camera, and number of objects in each image. The purpose is to determine the degree of distortion an image can withstand across all images based on the presence of objects, thus minimizing the impact of distortion on other images. During the calculation, the size and number of objects in the image are comprehensively considered, and depth information can also be further incorporated. For example, a comprehensive evaluation function can be defined, taking parameters such as object size, distance, and number as input, and the image weight w is obtained through function calculation. img For example, if there are more and closer objects in image A, and fewer and farther objects in image B, the weight of image A is calculated to be 0.2 and the weight of image B is 0.8 by the evaluation function. In this way, the image with the lower weight can be corrected for greater deformation in subsequent processing, while preventing the image with the higher weight from having a greater impact on other images after deformation.

[0090] This application embodiment, by comprehensively analyzing the feature information of objects and flexibly adjusting the weight calculation strategy according to actual application needs, can reasonably allocate the importance of different objects and images in the image stitching process. This helps to more accurately consider the characteristics of objects and images when calculating the homography matrix after correction, reduce deformation and error propagation during the image stitching process, improve the global alignment accuracy and visual consistency of image stitching, and thus improve the quality and effect of image stitching.

[0091] In some embodiments of this application, the step of calculating the correction angle of each image based on the absolute value of the angle difference between the angle before and after transformation of the object that satisfies the preset correction condition and the corresponding object weight includes: using the object weight of each image, performing a weighted summation of the absolute values ​​of the angle difference between the angle before and after transformation of the object that satisfies the preset correction condition in each image to obtain the correction angle of each image; and converting the correction angle of each image into a correction angle matrix.

[0092] For each image, iterate through all objects that meet the preset correction conditions, and associate the absolute value of the angle difference Δθ of each object with its corresponding object weight w. obj Multiply the values ​​to obtain the weighted angle difference for each object. Then, sum the weighted angle differences of all objects in the image that meet the correction criteria. The summed result is the correction angle of the image. This step fully considers the importance of different objects in the image, and by using a weighted summation method, the calculated correction angle better reflects the overall orientation correction requirements of the image.

[0093] Since the homography matrix to be corrected needs to be corrected subsequently through matrix multiplication, the correction angles need to be represented in matrix form. Based on the numerical values ​​of the correction angles and following specific matrix construction rules, they can be filled into the corresponding positions in the matrix to construct the correction angle matrix H. pre-correction Finally, based on the correction angle matrix H of each image... pre-correction and the corresponding image weights w img and the initial homography matrix H img Calculate the homography matrix H of each image after correction. img-final Specifically, it can be represented as H pre-correction * w img *H img =H img-final .

[0094] It should be noted that in matrix multiplication, H pre-correction Need to be left-multiplied by H imgLeft multiplication is not allowed, but right multiplication is not. This is because matrix multiplication does not satisfy the commutative law; left multiplication and right multiplication will yield different results. In the context of image geometric transformation, left multiplication correctly applies the transformation represented by the correction angle matrix to the geometric relationship described by the initial homography matrix, thus obtaining an accurate corrected homography matrix. Through this operation, the orientation correction requirements of the image, the image's deformation tolerance, and the initial geometric transformation relationship are comprehensively considered, enabling the calculated corrected homography matrix to more accurately describe the geometric transformations between images.

[0095] This application's embodiments comprehensively consider the angular changes and object weights of objects that meet the correction conditions through weighted summation, enabling more accurate calculation of the image correction angle and reflecting the overall orientation correction requirements of the image. Converting this into a correction angle matrix facilitates efficient and accurate correction operations on the homography matrix via matrix multiplication, effectively reducing orientation deviations and geometric distortions during image stitching, improving image stitching accuracy and visual effects, and providing strong support for high-quality image stitching.

[0096] In some embodiments of this application, the step of post-processing the multiple images to be stitched according to the corrected homography matrix of each image to obtain the stitched image includes: transforming each image according to the corrected homography matrix of each image to obtain transformed images; and stitching and rendering the transformed images to obtain the stitched image.

[0097] Post-processing the images to be stitched based on the corrected homography matrix is ​​the final and crucial step in the image stitching process. This step transforms the images using the corrected homography matrix, then stitches and renders the transformed images to obtain a high-quality stitched image, thus completing the entire image stitching task.

[0098] Specifically, each image is first transformed based on its corrected homography matrix. The homography matrix describes the geometric transformation relationship between images, and the corrected homography matrix more accurately reflects this relationship. For each image, its image coordinates are subjected to linear and perspective transformations using the corrected homography matrix. Specifically, if the homogeneous coordinates of a point in the image are (x, y, 1), and the corrected homography matrix is ​​H, then the transformed coordinates (x′, y′, w′) can be calculated using matrix multiplication. The homogeneous coordinates are then converted to non-homogeneous coordinates to obtain the point's position in the transformed image. This transformation operation is performed on all points in the image to obtain the transformed images.

[0099] The transformed images are then stitched together and rendered. During stitching, the transformed images need to be combined according to their correct positional relationships. A seamless stitching algorithm can be used to smooth the transition of pixels in overlapping areas, eliminating stitching artifacts. Rendering involves adjusting and optimizing the color, brightness, and other aspects of the stitched image to make it visually more natural and harmonious, ultimately resulting in the stitched image. Figure 4 As shown, a schematic diagram comparing the image stitching effect before and after correction is provided in an embodiment of this application.

[0100] This application embodiment uses accurate homography matrix transformation to precisely transform each image into a unified geometric space, ensuring the geometric alignment accuracy between images. Further seamless stitching and rendering effectively eliminates stitching artifacts, improves the visual consistency of the stitched images, and enables the final stitched image to more realistically and accurately present the original scene, meeting the requirements for high-quality image stitching.

[0101] This application embodiment also provides an image stitching device 500, such as... Figure 5 As shown, a schematic diagram of an image stitching device according to an embodiment of this application is provided. The image stitching device 500 includes:

[0102] The acquisition unit 510 is used to acquire multiple images to be stitched together and to calculate the initial homography matrix of each image;

[0103] The image segmentation and depth estimation unit 520 is used to perform image segmentation and depth estimation on each image to obtain the image segmentation results and depth estimation results for each image.

[0104] The correction unit 530 is used to calculate the corrected homography matrix of each image based on the image segmentation results and depth estimation results of each image and the initial homography matrix using a weighted homography matrix correction strategy.

[0105] The post-processing unit 540 is used to perform post-processing on the multiple images to be stitched according to the homography matrix of each image after correction, so as to obtain the stitched image.

[0106] In some embodiments of this application, the acquisition unit 510 is specifically used for: performing feature extraction on the multiple images to be stitched to obtain feature extraction results of multiple images; performing feature matching based on the feature extraction results of multiple images to obtain feature matching results between pairs of images; calculating the initial homography matrix between each image and its adjacent images based on the feature matching results between pairs of images, wherein there are overlapping regions between each image and its adjacent images; and calculating the initial homography matrix between each image and a predefined reference image based on the initial homography matrix between each image and its adjacent images.

[0107] In some embodiments of this application, the correction unit 530 is specifically used for: estimating the object orientation based on the image segmentation results of each image to obtain the object orientation estimation result; calculating the object's pre-transformation angle based on the object orientation estimation result, and transforming the object using the initial homography matrix to obtain the object's post-transformation angle; determining whether the object meets a preset correction condition based on the absolute value of the angle difference between the object's pre-transformation angle and post-transformation angle and a preset angle difference threshold; and, if the object meets the preset correction condition, calculating the corrected homography matrix of each image using a weighted homography matrix correction strategy based on the image segmentation results and depth estimation results of each image, the absolute value of the angle difference between the pre-transformation angle and post-transformation angle of the object that meets the preset correction condition, and the initial homography matrix.

[0108] In some embodiments of this application, the correction unit 530 is specifically used to: calculate the object weight and image weight of each image based on the image segmentation result and depth estimation result of each image; calculate the correction angle of each image based on the absolute value of the angle difference between the angle before and after transformation of the object that satisfies the preset correction condition and the corresponding object weight; and calculate the corrected homography matrix of each image based on the correction angle and the corresponding image weight of each image and the initial homography matrix.

[0109] In some embodiments of this application, the correction unit 530 is specifically used to: determine the size, distance from the camera, and number of objects in each image based on the image segmentation results and depth estimation results of each image; calculate the object weight of each image based on the size and distance from the camera of each object; and calculate the image weight of each image based on the size, distance from the camera, and number of objects in each image.

[0110] In some embodiments of this application, the correction unit 530 is specifically used to: use the object weights of each image to perform a weighted summation of the absolute values ​​of the angle differences between the angles before and after transformation of the objects in each image that meet the preset correction conditions, so as to obtain the correction angles of each image; and convert the correction angles of each image into a correction angle matrix.

[0111] In some embodiments of this application, the post-processing unit 540 is specifically used to: transform each image according to the homography matrix after correction of each image to obtain transformed images; and stitch and render the transformed images to obtain the stitched image.

[0112] It is understood that the above-described image stitching device can implement all the steps of the image stitching method provided in the foregoing embodiments. The relevant explanations of the image stitching method are applicable to the image stitching device and will not be repeated here.

[0113] Figure 6 This is a schematic diagram of the structure of a device according to an embodiment of this application. For example... Figure 6 As shown, the device includes one or more processors (or processing units), and may also include one or more memories coupled to the processors, and may also include a communication module coupled to the processors.

[0114] A communication module can be used to communicate with other devices or apparatuses, such as sending or receiving data and / or signals. A communication module may have at least one communication module for communication. A communication module may include any interface necessary for communicating with other devices. Exemplarily, a communication module may be a transceiver, circuit, bus, module, or other type of communication module.

[0115] The processor may include, but is not limited to, one or more of the following: a general-purpose computer, a special-purpose computer, a microcontroller, a digital signal processor (DSP), or a controller-based multi-core controller architecture. The device may have multiple processors, such as application-specific integrated circuit (ASIC) chips, which are time-dependent on a clock synchronized with the main processor.

[0116] The memory may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, at least one of the following: read-only memory (ROM), electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disc (CD), digital video disc (DVD), or other magnetic and / or optical storage. Examples of volatile memories include, but are not limited to, at least one of the following: random access memory (RAM), or other volatile memories that do not persist during the duration of a power outage.

[0117] A computer program consists of computer-executable instructions that are executed by an associated processor. Programs can be stored in ROM. A processor can perform any appropriate action and processing by loading the program into RAM.

[0118] Possible implementations of this application can be achieved through a program, enabling the communication device to execute any of the processes discussed in the foregoing embodiments. Possible implementations of this application can also be achieved through hardware or a combination of software and hardware.

[0119] In some implementations, the program may be tangibly contained in a computer-readable storage medium, which may include in a device (such as in memory) or other storage device accessible by the device. The program may be loaded from the computer-readable storage medium into RAM for execution. The computer-readable storage medium may include any type of tangible non-volatile memory, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc.

[0120] This application also provides a computer-readable storage medium storing computer instructions or program code thereon, which, when executed by a processor, causes the processor to perform the methods and functions involved in any of the above embodiments. A computer-readable medium can be any tangible medium that contains or stores a program for or relating to an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. More detailed examples of computer-readable storage media include electrical connections with one or more wires, magnetic media (e.g., disks, floppy disks, hard disks, magnetic tapes, magnetic storage devices), optical media (e.g., optical storage devices, DVDs), semiconductor media (e.g., solid-state drives), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), or any suitable combination thereof.

[0121] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. Embodiments of this application also provide at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. This computer program product includes one or more computer-executable instructions, such as instructions included in a program module, which execute in a device on a target real or virtual processor to perform the processes, methods, and functions involved in any of the above embodiments. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0122] This application also proposes a computer program product, including a computer program or instructions that, when run on a computer, cause the computer to perform the processes, methods, and functions described in the above embodiments. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided as needed. The machine-executable instructions for the program modules can be executed locally or in a distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0123] Generally, the various embodiments of this application can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software, which can be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are shown and described as block diagrams, flowcharts, or represented using some other illustration, it should be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0124] It should be noted that although embodiments of this application have been described above with reference to the accompanying drawings, these embodiments are not independent of each other, and they can be combined to obtain other embodiments. The methods, situations, categories, and classifications of embodiments in this application are only for the convenience of description and should not constitute a special limitation. Various methods, categories, situations, and features in embodiments can be combined with each other if logically consistent. The various embodiments of this application can be arbitrarily combined to achieve different technical effects. The embodiments of this application will not list various combinations.

[0125] Furthermore, although the operation of the methods of this disclosure is described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. It should also be noted that the features and functions of two or more devices according to this disclosure may be embodied in one device. Conversely, the features and functions of one device described above may be further divided and embodied by multiple devices.

[0126] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0127] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. An image stitching method, characterized in that, The image stitching method includes: Obtain multiple images to be stitched together and calculate the initial homography matrix of each image; Image segmentation and depth estimation are performed on each image to obtain the image segmentation results and depth estimation results for each image; Based on the image segmentation results and depth estimation results of each image and the initial homography matrix, the corrected homography matrix of each image is calculated using a weighted homography matrix correction strategy. The multiple images to be stitched are post-processed based on the homography matrix of each image after correction to obtain the stitched image.

2. The image stitching method according to claim 1, characterized in that, The calculation of the initial homography matrix for each image includes: Feature extraction is performed on the multiple images to be stitched together to obtain feature extraction results for the multiple images; Feature matching is performed based on the feature extraction results of multiple images to obtain the feature matching results between pairs of images; The initial homography matrix between each image and its neighboring images is calculated based on the feature matching results between each pair of images. There are overlapping regions between each image and its neighboring images. Calculate the initial homography matrix between each image and a predefined reference image based on the initial homography matrix between each image and its neighboring images.

3. The image stitching method according to claim 1, characterized in that, The step of calculating the corrected homography matrix of each image using a weighted homography matrix correction strategy based on the image segmentation results, depth estimation results, and the initial homography matrix of each image includes: Based on the image segmentation results of each image, the object orientation is estimated to obtain the object orientation estimation result; The object's angle before transformation is calculated based on the object's orientation estimation result, and the object is transformed using the initial homography matrix to obtain the object's angle after transformation. Based on the absolute value of the angle difference between the object's angle before and after the transformation and the preset angle difference threshold, determine whether the object meets the preset correction conditions. When the object meets the preset correction conditions, the corrected homography matrix of each image is calculated using a weighted homography matrix correction strategy based on the image segmentation results and depth estimation results of each image, the absolute value of the angle difference between the angle before and after transformation of the object that meets the preset correction conditions, and the initial homography matrix.

4. The image stitching method according to claim 3, characterized in that, The step of calculating the corrected homography matrix of each image using a weighted homography matrix correction strategy, based on the image segmentation results and depth estimation results of each image, the absolute value of the angle difference between the angle before and after transformation of the object that meets the preset correction conditions, and the initial homography matrix, includes: Based on the image segmentation results and depth estimation results of each image, calculate the object weight and image weight of each image; The correction angle of each image is calculated based on the absolute value of the angle difference between the angle before and after transformation of the object that meets the preset correction conditions and the corresponding object weight. Based on the correction angle and corresponding image weight of each image, as well as the initial homography matrix, the corrected homography matrix of each image is calculated.

5. The image stitching method according to claim 4, characterized in that, The step of calculating the object weights and image weights of each image based on the image segmentation results and depth estimation results of each image includes: The size, distance from the camera, and number of objects in each image are determined based on the image segmentation and depth estimation results of each image. The object weights for each image are calculated based on the size of the objects in each image and their distance from the camera. The image weights of each image are calculated based on the size of the objects in each image, their distance from the camera, and the number of objects.

6. The image stitching method according to claim 4, characterized in that, The step of calculating the correction angle of each image based on the absolute value of the angle difference between the angle before and after transformation of the object that meets the preset correction conditions and the corresponding object weight includes: By utilizing the object weights of each image, the absolute values ​​of the angle differences between the angles before and after transformation of objects in each image that meet the preset correction conditions are summed in a weighted manner to obtain the correction angle of each image. The correction angles of each image are converted into a correction angle matrix.

7. The image stitching method according to any one of claims 1 to 6, characterized in that, The post-processing of the multiple images to be stitched together based on the corrected homography matrix of each image to obtain the stitched image includes: The images are transformed based on the homography matrix after correction to obtain the transformed images. The transformed images are stitched together and rendered to obtain the stitched image.

8. An image stitching device, characterized in that, The image stitching device includes: The acquisition unit is used to acquire multiple images to be stitched together and to calculate the initial homography matrix of each image; The image segmentation and depth estimation unit is used to perform image segmentation and depth estimation on each image, and obtain the image segmentation results and depth estimation results for each image. The correction unit is used to calculate the corrected homography matrix of each image based on the image segmentation results and depth estimation results of each image and the initial homography matrix, using a weighted homography matrix correction strategy. The post-processing unit is used to perform post-processing on the multiple images to be stitched according to the homography matrix of each image after correction, so as to obtain the stitched image.

9. An apparatus comprising: processor; And a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the image stitching methods of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the image stitching method according to any one of claims 1 to 7.

Images