Image stitching method and device based on foreground and background segmentation

Abstract

The present invention discloses an image stitching method and device based on foreground and background segmentation. The method comprises: segmenting the two images to be stitched into foreground and background, estimating the 2D rotation matrix between the two images based on angular velocity data collected by a gyroscope, and estimating the direction of gravity based on acceleration data when the two images were captured; preprocessing the images using the rotation matrix between the two images, dividing the images into grids, and stitching the segmented foreground and background separately; and calculating the final foreground position in the background based on the relative position of the foreground and performing fusion stitching as required, optionally performing gravity correction on the final image. The present invention reduces artifacts and excessive edge distortion in stitching foreground and background images, and can remove dynamic foregrounds as required, thereby improving the quality and accuracy of image stitching.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image splicing method and device based on foreground and background segmentation. Background Art

[0002] Common image registration methods align two images by extracting feature regions from the images and calculating the optimal global transformation of the images to be stitched. These methods typically perform well only in ideal, nearly planar scenes, where the input images contain little or no parallax, allowing the images to be aligned using a global homography. However, when scenes have large parallax, are non-planar, are captured from multiple perspectives, or exhibit lens distortion, a single homography matrix cannot effectively handle these complex transformations and often performs poorly, resulting in artifacts and misalignment.

[0003] Therefore, some methods attempt to adapt to different areas of the image by dividing the image to be stitched into regions and warping them separately. From simply separating a single scene into a far plane and a ground plane, stitching is performed by calculating two homography transformations on the image to be stitched, which reduces artifacts to a certain extent, to gradually dividing the entire image area into a grid and calculating multiple local homography matrices to achieve better local adaptation, but this also introduces severe perspective distortion in non-overlapping areas. Subsequently, some methods strive to reduce the distortion of non-overlapping areas through methods such as shape preservation and setting global homography constraints. These image distortions have produced certain effects in reducing artifacts, but it is difficult to achieve satisfactory results when facing image stitching with larger parallax. For large parallax, many studies use the stitching line method to stitch graphics to reduce or even eliminate artifacts. These methods have more or less produced certain effects, but there are still some problems. Summary of the Invention

[0004] In view of the above-mentioned deficiencies of the prior art, the purpose of the present invention is to provide an image stitching method and device based on foreground and background segmentation to solve the problems of large parallax, artifacts and severe edge distortion in the existing image stitching.

[0005] In order to achieve the above object of the invention, the technical solution of the present invention is as follows:

[0006] In a first aspect, an image stitching method based on foreground and background segmentation comprises the following steps:

[0007] The two images to be stitched are segmented into foreground and background, the 2D rotation matrix between the two images is estimated based on the angular velocity data collected by the gyroscope, and the direction of gravity is estimated based on the acceleration data when the two images were taken;

[0008] The two segmented background images are rotated according to the estimated rotation matrix, then scaled and the initial feature point pair vectors are calculated. The SIFT feature point pair accelerated matching method is used to obtain the final matching feature point pairs. The images are divided into grids, and the homography matrix is ​​calculated using the feature point pairs. By setting anchor points and overlapping area boundaries, both images are smoothed and extrapolated to complete the background image stitching.

[0009] Preprocessing the two segmented foreground images includes applying the same scaling to the foreground image based on the scaling ratio obtained for the background image, rotating the foreground image accordingly, and meshing the foreground image. Local homography matrices between the grids are calculated based on feature point pairs, and the foreground images are stitched together based on the local homography matrices.

[0010] Based on the stitched foreground and background images, the offset is calculated by finding pairs of matching feature points with high credibility during the stitching process. The position of the foreground image in the background image is calculated based on the offset, and the foreground and background are smoothly fused. Finally, gravity correction is performed according to the direction of gravity.

[0011] Furthermore, the two images are segmented into foreground and background, including:

[0012] The U2-Net model is used to separate the foreground target and divide the original image into foreground image and background image.

[0013] Furthermore, the 2D rotation matrix between the two images is estimated based on the angular velocity data collected by the gyroscope, including:

[0014] Based on the gyroscope data collected at a specified frequency, the device's angular velocity data is obtained. The collected angular velocity data is integrated to obtain the device's rotation. Based on the conversion between the device's local coordinates and the camera's coordinate system, the initial body coordinate system of the inertial sensor is selected as the fixed coordinate system. Based on the device's rotation, the 2D rotation matrix between the two images is derived, denoted as C.

[0015] Estimate the direction of gravity based on the acceleration data when the two images were taken, including:

[0016] The acceleration data of the camera is collected when it is stationary and taking pictures. The angle with the direction of gravity is calculated based on the gravity data of the two images. The gravity correction rotation matrix is ​​calculated based on the two angles and is recorded as R.

[0017] Furthermore, the SIFT feature point pair accelerated matching method is used to obtain the final matching feature point pairs, including:

[0018] The two background images are rotationally aligned to obtain two sets of feature points kp1 and kp2 and their corresponding descriptors ds1 and ds2. The pre-obtained vector relationship V is used to determine the ideal position o of the feature point in background image 2 corresponding to the feature point in background image 1. A search is performed based on the preset tolerance range δ to find candidate feature points within the tolerance range among the feature points in background image 2. Valid matching pairs are selected using the preset distance threshold and used as the final matching feature point pairs.

[0019] Furthermore, the homography matrix is ​​calculated using the feature point pairs, and a smooth extrapolation operation is performed on both images by setting anchor points and overlapping area boundaries, including:

[0020] According to one of the matching point pairs in the two images obtained, it is converted into homogeneous coordinates, and the transformation relationship between the point pairs is established using the homography matrix. By constructing a linear equation system and minimizing the objective function Solve the homography matrix H, where a i It is the coefficient matrix corresponding to each matching point, and the constraint ||h|| 2 = 1 to ensure 8 degrees of freedom of isomorphic transformation;

[0021] Set anchor points at the edge of the image and use homography linearization for smooth extrapolation;

[0022] For any point q in the non-overlapping region, the anchor points in the boundary are linearized and a weighted average of the transformations is calculated; when q is far away from an anchor point, all anchor points are given similar weights.

[0023] Furthermore, in the process of stitching the background images, similarity transformation is used in the non-overlapping areas of the two background images, specifically including:

[0024] Obtain the grid point coordinates of the images to be stitched and calculate the minimum distance from each grid point to the boundary of the overlapping area. Normalize the grid points based on the minimum distance. Then use the Gaussian weighted kernel function to calculate the local affine transformation matrix, and make the matrix maintain numerical stability through deconditioning and denormalization.

[0025] For each grid point, a distance normalization weight is calculated based on its distance to the overlapping area. In the non-overlapping area, the affine transformation matrix is ​​extrapolated using local linearization and smooth interpolation methods, while the affine transformation matrix remains unchanged in the overlapping area.

[0026] The affine transformation matrix and its inverse matrix are applied to the image stitching process to ensure smooth transition of the stitching edges and reduce distortion.

[0027] Furthermore, by finding matching feature point pairs with high credibility during the stitching process and calculating the offset, the position of the foreground image in the background image is calculated based on the offset, and the foreground and background images are smoothly fused, including:

[0028] Based on the feature point pair of the foreground image, the center of the feature point pair is selected as a reference, and the centers of the feature points of foreground image 1 and foreground image 2 are calculated, denoted as g1 and g2. The parallax offset of the two foreground images is obtained by calculating the difference between g1 and g2. The position g1 of foreground image 1 in the image to be stitched is calculated, and the position g1′ after the transformation is calculated. The current new foreground position is obtained by adding the position offset g to the position of g1′.

[0029] A foreground valid area mask is created based on the brightness of the foreground image, and the foreground is divided into a core area and an edge area through edge extension technology. In the edge area, distance transform is used to generate fusion weights, and the foreground pixels in the core area are directly replaced by the background pixels through linear interpolation method, while the edge area is naturally transitioned through weighted interpolation of distance weights, completing the smooth fusion of the foreground image and the background image.

[0030] In a second aspect, an image stitching device based on foreground and background segmentation includes:

[0031] The foreground and background segmentation and attitude estimation module is used to segment the two images to be stitched into foreground and background, estimate the 2D rotation matrix between the two images based on the angular velocity data collected by the gyroscope, and estimate the direction of gravity based on the acceleration data when the two images were taken;

[0032] The background processing module is used to rotate the two segmented background images according to the estimated rotation matrix, scale the images, calculate the initial feature point pair vectors, and use the SIFT feature point pair accelerated matching method to obtain the final matching feature point pairs. The images are divided into grids, the homography matrix is ​​calculated using the feature point pairs, and the two images are smoothed and extrapolated by setting anchor points and overlapping area boundaries to complete the background image stitching.

[0033] The foreground processing module is used to pre-process the two segmented foreground images, including performing the same processing on the foreground image according to the scaling ratio obtained for the background image, rotating the foreground image accordingly, and dividing the foreground image into grids. The local homography matrix between the grids is calculated based on the feature point pairs, and the foreground image is stitched together based on the local homography matrix.

[0034] The image fusion module is used to calculate the offset based on the already stitched foreground and background images by finding matching feature points with high credibility during the stitching process. Based on the offset, the position of the foreground image in the background image is calculated, and the foreground and background are smoothly fused. Finally, gravity correction is performed according to the direction of gravity.

[0035] In a third aspect, the present invention further provides a computer device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the image stitching method based on foreground and background segmentation as described in the first aspect are implemented.

[0036] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image stitching method based on foreground and background segmentation as described in the first aspect.

[0037] Beneficial effects:

[0038] (1) The present invention proposes a foreground segmentation strategy for large parallax images. By stitching the foreground and background separately and then integrating them, the problems of shadows and dislocations caused by traditional stitching technology when facing depth changes or moving foregrounds are significantly reduced. At the same time, the dynamic foreground can be removed as needed, thereby improving the overall quality of the stitched image.

[0039] (2) This invention introduces IMU-based gravity-assisted correction technology, which can automatically perform gravity correction on images, effectively correcting image tilt. This technology avoids the incongruity caused by the tilt of the first image by adjusting the final stitching perspective to the overlapping area of ​​the images, ensuring the visual consistency and coordination of the stitching result.

[0040] (3) To address the time-consuming problem of SIFT (Scale-invariant Feature Ttransform) feature point matching, the present invention uses IMU to calculate the rotation matrix to complete preliminary image rotation alignment, and on this basis effectively estimates the vector relationship of feature points, thereby greatly accelerating the matching speed of feature points and improving the efficiency of image stitching.

[0041] (4) The present invention draws on the idea of ​​the mobile DLT (Direct Linear Transform) algorithm to perform smooth extrapolation on the image, and sets global similarity and edge anchor points. Through weighted combination, the distortion of the two images during the stitching process is minimized, effectively solving the edge distortion problem. At the same time, a fusion smoothing algorithm is designed to further improve the visual effect of the stitched image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Flowchart of the image stitching method based on foreground and background segmentation of the present invention;

[0043] Figure 2 A framework diagram for implementing a method according to an embodiment of the present invention;

[0044] Figure 3 Schematic diagram of two images to be spliced ​​according to the present invention;

[0045] Figure 4 Schematic diagram of foreground and background segmentation of two images to be spliced ​​according to the present invention;

[0046] Figure 5 Schematic diagram of two background images of the present invention;

[0047] Figure 6 Schematic diagram of two foreground images of the present invention;

[0048] Figure 7 A schematic diagram of a rotationally aligned image according to the present invention;

[0049] Figure 8 Schematic diagrams of the splicing results of the present invention, wherein (a) is a schematic diagram of the splicing result after removing the dynamic foreground, and (b) is a schematic diagram of the splicing result after fusing the foreground and background. DETAILED DESCRIPTION

[0050] The present invention uses an inertial measurement unit (IMU) to assist in image stitching, and uses the IMU to compensate for the lack of context during the stitching process. Using IMU data to estimate the displacement and rotation between images can provide more accurate and reliable camera pose estimation, compensate for the image stitching process, effectively eliminate ghosting and image information loss in traditional stitching methods, and obtain high-quality panoramic images. The participation of the IMU improves the matching between the images to be stitched, especially in situations where there are fewer overlapping areas, fewer feature point pairs, or serious lack of context information. It is difficult to obtain a satisfactory stitching effect by relying solely on visual algorithms.

[0051] This paper addresses large parallax, particularly when there is a large difference in depth between the foreground and background. It proposes an image stitching method based on foreground and background segmentation. This method divides the image into two parts, the foreground and background, for processing. Even with small parallax in the background, a grid method can still achieve good results. Finally, the separately processed foreground and background are recombined, thus avoiding ghosting issues under large parallax conditions. This method utilizes inertial sensors to accelerate the matching of feature point pairs and perform angle correction on the stitched image to achieve the angle that is most interesting to the user, resulting in a more satisfactory result.

[0052] Figure 1 The flow chart of the image splicing method based on foreground and background segmentation disclosed in the present invention is shown. Figure 1 The method mainly includes: foreground and background segmentation and attitude estimation, background processing, foreground processing, and foreground and background fusion. Among them: in the foreground and background segmentation and attitude estimation stage, the two images to be stitched are segmented into foreground and background, the 2D rotation matrix between the two images is estimated based on the angular velocity data collected by the gyroscope, and the direction of gravity is estimated based on the acceleration data when the two images are taken. In the background processing stage and the foreground processing stage, the image is pre-processed using the rotation matrix between the two images, the image is divided into grids, and the segmented foreground and background are stitched separately. In the foreground and background fusion stage, according to needs, the final foreground position in the background is calculated based on the relative position of the foreground and the fusion and stitching are performed. The final image can be optionally gravity corrected.

[0053] In one embodiment, a method for image stitching based on foreground and background segmentation of the present invention includes the following steps:

[0054] Step 1: Acquire the images to be stitched and the inertial sensor data.

[0055] like Figure 2 As shown in the figure, when shooting the image to be stitched, the inertial sensor data is collected at a frequency of 100Hz, and the angular velocity data of the device is obtained according to the gyroscope data. The rotation of the device is obtained by integrating the collected angular velocity data ω. Furthermore, if the posture of the inertial sensor at time t is C(t), the posture at time t+δt is obtained using the data ω b =[ω bx ω by ω bz ] T pass To calculate, where is

[0056] Identity matrix, σ=|ω b δt|, Based on the conversion between the local coordinates of the device and the camera coordinate system, the 2D rotation matrix between the two images is derived, which is recorded as C. And the acceleration data collected at the time of the camera shooting is used to calculate accel = [accel x accel y accel z ]Use the inverse tangent function to calculate the angle with the direction of gravity, considering only the roll angle and denoted as roll, to obtain the gravity correction rotation matrix Denoted as R, the captured image is as follows Figure 3 The two images to be stitched are represented by (a) and (b), respectively, and the same applies below.

[0057] Step 2: Preprocess the image to be stitched, including foreground and background segmentation, image rotation and scaling, etc.

[0058] The present invention uses U 2 -Net model performs foreground and background segmentation on the spliced ​​image. The segmentation effect is as follows Figure 4 As shown, Figure 5 To remove the foreground image, the image is rotated and aligned using the pre-obtained 2D rotation matrix C. Both images are rotated to prevent the rotation angle of a single image from being too large. Both images are rotated to the middle overlapping area for alignment. Figure 7 As shown in the figure, since the background is far away from the camera, a simple 2D rotation can achieve good alignment. That is, the vector relationship between the feature points of the two images after a simple 2D rotation will approach an equal vector relationship. However, the image scaling problem must be taken into account to ensure that the scene size of the two images is consistent. SURF feature point extraction and matching are then used on the two images. To reduce errors and eliminate incorrect matching point pairs, the distance between adjacent feature points in the same image and the distance between corresponding adjacent feature points in the other image satisfy the same proportional relationship. Bidirectional matching is performed simultaneously to obtain more accurate feature point pairs. Based on the above proportional relationship, the two images are scaled. At this point, the feature point pairs of the images all approach the same vector relationship, which is denoted as V.

[0059] Step 3: Stitch the processed background images.

[0060] (1) By using IMU, the background image is rotated and aligned. At this time, the feature point pairs between the two images tend to satisfy the same vector relationship. According to this characteristic, the SIFT feature point pair is used to accelerate the matching model. First, two sets of feature points kp1 and kp2 and corresponding descriptors ds1 and ds2 are obtained. The feature points kp1 and kp2 record the information of the feature points detected in image 1 and image 2, including the coordinates of the feature points. The descriptors ds1 and ds2 record the description information of the local area around the feature points in image 1 and image 2, which are used to calculate the similarity of the matching feature point pairs. Then, the feature point position information recorded in kp1 obtained in image 1 and the pre-obtained vector relationship V can be used to determine the approximate position of the feature point in image 2 corresponding to the feature point in image 1, and use this as a priori to determine the possible matching area. For each matching point m in image 1, to obtain the corresponding point m′ in image 2, we first use the vector relationship V to determine the ideal position O of the corresponding feature point in image 2. We then set a tolerance range δ and search within this area. Then, we find candidate feature points within the tolerance range among the feature points in image 2. Here, candidate feature points are those with the greatest similarity calculated from descriptors ds1 and ds2. To further filter matching points, we set a distance threshold to screen valid matching pairs, and then extract more reliable feature point pairs from the successfully matched feature pairs.

[0061] (2) Using the acquired feature point pairs, using the local homography model, referring to the APAP (As-Projective-As-Possible) and AANAP (Adaptive As-Natural-As-Possible) homography models, the global homography matrix is ​​calculated based on the extracted feature point pairs, and then the image is divided into grids with a grid size of 100*100. Next, the local homography matrix of each grid is calculated using the feature point pairs. First, according to one of the matching point pairs m=[xy]T and m′=[x′ y′]T in the two acquired images, it is converted into homogeneous coordinates. and Use the homography matrix H∈R 3×3 Establish the transformation relationship between point pairs, expressed as By constructing a system of linear equations and minimizing the objective function Solve the homography matrix H, where a i is the coefficient matrix corresponding to each matching point. At the same time, the constraint ||h|| 2 = 1 to ensure the 8 degrees of freedom of the isomorphic transformation. On this basis, the offset Gaussian is used to generate the weight ω according to the distance between the pixels. i,j =max(exp(-‖m i-m j || 2 / σ 2 ), γ). The parameter γ∈[0 1] is an offset used to prevent numerical problems. The weight is added to the minimization objective function and weighted minimization is used. To estimate the position m j Since corresponding feature point pairs only exist in the overlapping region, the local homography can only be calculated in the region where the reference image overlaps. For each pixel in the non-overlapping region, its transformation is calculated as a weighted linear combination of the local homographies in the overlapping region.

[0062] (3) Anchor points are set at the edge of the image, and homography linearization is used for smooth extrapolation. The homography linearization of any point q in the neighborhood of anchor point p is expressed as h(q) = h(p) + J through the Taylor series of homography transformation h(q) h (p)(qp)+o(||qp||), where J h (p) is the Jacobian of the homography h at point p. For any point q in the non-overlapping region, the anchor points in the boundary are linearized and the weighted average of the transformations is calculated. The weighted combination of the linearizations is The weight a i The Student's t weighting method is used to reduce perspective distortion by extrapolating the non-overlapping areas of the two images and avoid regional distortion caused by extrapolation of a single image.

[0063] (4) In order to further reduce image distortion and make the panorama look more natural, we first use similarity transformation in the non-overlapping areas of the two images with a large threshold ε g The RANSAC (RANdom SAmple Consensus) algorithm removes outliers from the matching. After removing the outliers, the RANSAC algorithm with a smaller threshold ε1 is further used to find the plane homography matrix with the most inliers, where ε1 < ε g Finally, the rotation angles of each local similarity transformation are compared, and the transformation with the smallest rotation angle is selected as the final global similarity transformation to ensure that the alignment between images has high consistency and naturalness.

[0064] The calculated global similarity transformation is used to adjust the distortion of the image, and the weight distribution is performed by calculating the distance from the grid point to the boundary of the overlapping area. Specifically, for each grid point M i (Its coordinates are (x i ,y i )), calculate its distance to all edges on the boundary of the overlapping region and find the minimum value d among these distances iThen, calculate the maximum value d among the minimum distances from all grid points to the boundary of the overlapping area max To normalize these distances, the distance d of each grid point is i Map to the interval [0, 1] and get the weight coefficient a i and b i , expressed as b i =1-a i In the subsequent image transformation processing, different degrees of transformation interpolation and extrapolation are performed in the non-overlapping areas based on these weights. First, the transformation of the second image is performed, and the non-overlapping areas of the second image are gradually transformed by global similarity, which is expressed as Here Ht i satisfy H i is the i-th local homography matrix, is the updated local transformation, S1 is transformed by the global similarity transformation S, satisfying For the transformation of image 1, it can be expressed as here Is the updated local transformation. For the overlapping area of ​​the two images, the calculated local homography matrix is ​​directly used for transformation, that is, Ht is directly used i Transform the overlapping area of ​​the second image using Hr i Transform the overlapping area of ​​the first image. Figure 8 As shown in (a), the spliced ​​background image is gravity corrected to finally obtain the dynamic foreground image.

[0065] Step 4: Stitch the processed foreground images.

[0066] like Figure 6 As shown, the two foreground images are first preprocessed. The foreground is scaled according to the scaling ratio of the background and rotated accordingly. Then, they are stitched together directly based on feature point pairs. If the two foreground images do not have enough reliable feature point pairs, the foreground image with a larger pixel range is used to retain more foreground information, or one of the foreground images is selected as the final foreground through user interaction. Similarly, the foreground is divided into grids, and a linear transformation is calculated to calculate the local homography matrix between the grids, ultimately completing the stitching of the foreground images.

[0067] Step 5: Fusion of foreground and background and pixel smoothing.

[0068] Because the foreground and background are processed separately, the position of the foreground within the background must be calculated and then fused. First, using the most reliable matching feature point pairs from the two images as a reference, the foreground's position within the background is calculated. Specifically, the foreground feature point pairs are extracted and their centers are selected as references. The feature point centers of foreground 1 and foreground 2 are calculated, denoted as g1 and g2. The difference between g1 and g2 yields the parallax offset between the two foreground images. Strictly speaking, this is the foreground parallax offset without considering the background offset, expressed as g = g1 - g2. The position g1 of foreground 1 in image 1 to be stitched is calculated, and the transformed position g1′ is calculated. The new foreground position is then obtained by adding the position offset g to g1′. A foreground valid area mask is then created based on the foreground image's brightness, and the foreground is divided into a core region and an edge region using edge extension techniques. In the edge region, a distance transform is used to generate fusion weights, and the foreground and background images are smoothly fused using linear interpolation. Finally, the foreground pixels in the core area directly replace the background pixels, while the edge area achieves a natural transition through weighted interpolation of distance weights, thereby completing the smooth fusion processing of the image. Figure 8 As shown in (b), gravity correction is finally performed to obtain the foreground and background fusion result.

[0069] This invention incorporates gravity-assisted correction technology through inertial-vision fusion, utilizing an inertial measurement unit (IMU) to correct for gravity in images. By capturing the camera's pose during image capture, the relative rotation matrix of the two images is estimated, accelerating feature point matching. Leveraging the concept of the Direct Linear Transform (DLT) algorithm, both images are smoothed and extrapolated. Global similarity is applied and anchor points are set to minimize distortion.

[0070] Based on the same technical concept as the method embodiment, the present invention also provides an image stitching device based on foreground and background segmentation, comprising:

[0071] The foreground and background segmentation and attitude estimation module is used to segment the two images to be stitched into foreground and background, estimate the 2D rotation matrix between the two images based on the angular velocity data collected by the gyroscope, and estimate the direction of gravity based on the acceleration data when the two images were taken;

[0072] The background processing module is similar to the two segmented background images. The images are rotated according to the estimated rotation matrix, and then scaled. The initial feature point pair vectors are calculated. The SIFT feature point pair accelerated matching method is used to obtain the final matching feature point pairs. The images are divided into grids, and the homography matrix is ​​calculated using the feature point pairs. By setting anchor points and overlapping area boundaries, both images are smoothed and extrapolated to complete the background image stitching.

[0073] The foreground processing module is used to pre-process the two segmented foreground images, including performing the same processing on the foreground image according to the scaling ratio obtained for the background image, rotating the foreground image accordingly, and dividing the foreground image into grids. The local homography matrix between the grids is calculated based on the feature point pairs, and the foreground image is stitched together based on the local homography matrix.

[0074] The image fusion module is used to calculate the offset based on the already stitched foreground and background images by finding matching feature points with high credibility during the stitching process. Based on the offset, the position of the foreground image in the background image is calculated, and the foreground and background are smoothly fused. Finally, gravity correction is performed according to the direction of gravity.

[0075] The present invention also provides a computer device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the image stitching method based on foreground and background segmentation as described above are implemented.

[0076] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the image stitching method based on foreground and background segmentation as described above are implemented.

[0077] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus (systems), computer devices, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] The present invention is described with reference to flowcharts of methods according to embodiments of the present invention. It should be understood that each process in the flowcharts and combinations of processes in the flowcharts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts. Figure 1 A device that specifies functions in a process or multiple processes.

[0079] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A function specified in a process or multiple processes.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 The steps of a specified function in a process or multiple processes.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. An image stitching method based on foreground and background segmentation, characterized in that: The following steps are involved: The two images to be stitched are segmented into foreground and background, the 2D rotation matrix between the two images is estimated based on the angular velocity data collected by the gyroscope, and the direction of gravity is estimated based on the acceleration data when the two images were taken; The two segmented background images are rotated according to the estimated rotation matrix, then scaled and the initial feature point pair vectors are calculated. The SIFT feature point pair accelerated matching method is used to obtain the final matching feature point pairs. The images are divided into grids, and the local homography matrix is ​​calculated using the feature point pairs. By setting anchor points and overlapping area boundaries, both images are smoothed and extrapolated to complete the background image stitching. Preprocessing the two segmented foreground images includes applying the same scaling to the foreground image based on the scaling ratio obtained for the background image, rotating the foreground image accordingly, and meshing the foreground image. Local homography matrices between the grids are calculated based on feature point pairs, and the foreground images are stitched together based on the local homography matrices. Based on the stitched foreground and background images, the offset is calculated by finding pairs of matching feature points with high credibility during the stitching process. The position of the foreground image in the background image is calculated based on the offset, and the foreground and background images are smoothly fused. Finally, gravity correction is performed according to the direction of gravity.

2. The method according to claim 1, characterized in that Segment the two images into foreground and background, including: Using U 2 -Net model separates the foreground target and divides the original image into foreground image and background image.

3. The method according to claim 1, characterized in that Estimate the 2D rotation matrix between the two images based on the angular velocity data collected by the gyroscope, including: Based on the gyroscope data collected at a specified frequency, the device's angular velocity data is obtained. The collected angular velocity data is integrated to obtain the device's rotation. Based on the conversion between the device's local coordinates and the camera's coordinate system, the initial body coordinate system of the inertial sensor is selected as the fixed coordinate system. Based on the device's rotation, the 2D rotation matrix between the two images is derived, denoted as C. Estimate the direction of gravity based on the acceleration data when the two images were taken, including: The acceleration data of the camera is collected when it is stationary and taking pictures. The angle with the direction of gravity is calculated based on the gravity data of the two images. The gravity correction rotation matrix is ​​calculated based on the two angles and is recorded as R.

4. The method according to claim 1, wherein The SIFT feature point pair accelerated matching method is used to obtain the final matching feature point pairs, including: The two background images are rotationally aligned to obtain two sets of feature points kp1 and kp2 and their corresponding descriptors ds1 and ds2. The pre-obtained vector relationship V is used to determine the ideal position O of the feature point in background image 2 corresponding to the feature point in background image 1. A search is performed based on the preset tolerance range δ to find candidate feature points within the tolerance range among the feature points in background image 2. Valid matching pairs are selected using the preset distance threshold and used as the final matching feature point pairs.

5. The method according to claim 1, wherein The homography matrix is ​​calculated using the feature point pairs, and both images are smoothed and extrapolated by setting the anchor points and the overlap region boundaries, including: According to one of the matching point pairs in the two images obtained, it is converted into homogeneous coordinates, and the transformation relationship between the point pairs is established using the homography matrix. By constructing a linear equation system and minimizing the objective function Solve the homography matrix H, where a i It is the coefficient matrix corresponding to each matching point, adding the constraint ‖h‖ 2 = 1 to ensure 8 degrees of freedom of isomorphic transformation; Set anchor points at the edge of the image and use homography linearization for smooth extrapolation; For any point q in the non-overlapping region, the anchor points in the boundary are linearized and a weighted average of the transformations is calculated; when q is far away from an anchor point, all anchor points are given similar weights.

6. The method according to claim 5, characterized in that In the process of stitching the background images, similarity transformation is applied in the non-overlapping areas of the two background images, including: Obtain the grid point coordinates of the images to be stitched and calculate the minimum distance from each grid point to the boundary of the overlapping area. Normalize the grid points based on the minimum distance. Then use the Gaussian weighted kernel function to calculate the local affine transformation matrix, and make the matrix maintain numerical stability through deconditioning and denormalization. For each grid point, a distance normalization weight is calculated based on its distance to the overlapping area. In the non-overlapping area, the affine transformation matrix is ​​extrapolated using local linearization and smooth interpolation methods, while the affine transformation matrix remains unchanged in the overlapping area. The affine transformation matrix and its inverse matrix are applied to the image stitching process to ensure smooth transition of the stitching edges and reduce distortion.

7. The method according to claim 1, characterized in that By finding highly reliable matching feature point pairs during the stitching process and calculating the offset, the position of the foreground image in the background image is calculated based on the offset, and the foreground and background images are smoothly fused, including: According to the feature point pair of the foreground image, the center of the feature point pair is selected as a reference, and the center of the feature point of foreground image 1 and foreground image 2 is calculated, recorded as g1 and g2. The parallax offset of the two foreground images is obtained by calculating the difference between g1 and g2, and the position g1 of the foreground image 1 in the image to be stitched is calculated, and the position g1 after the transformation is calculated. ′ , then in g1 ′ Add the position offset of g to the position of to get the current foreground new position; A foreground valid area mask is created based on the brightness of the foreground image, and the foreground is divided into a core area and an edge area through edge extension technology. In the edge area, distance transform is used to generate fusion weights, and the foreground pixels in the core area are directly replaced by the background pixels through linear interpolation method, while the edge area is naturally transitioned through weighted interpolation of distance weights, completing the smooth fusion of the foreground image and the background image.

8. An image splicing device based on foreground and background segmentation, characterized in that: include: The foreground and background segmentation and attitude estimation module is used to segment the two images to be stitched into foreground and background, estimate the 2D rotation matrix between the two images based on the angular velocity data collected by the gyroscope, and estimate the direction of gravity based on the acceleration data when the two images were taken; The background processing module is used to rotate the two segmented background images according to the estimated rotation matrix, scale the images, calculate the initial feature point pair vectors, and use the SIFT feature point pair accelerated matching method to obtain the final matching feature point pairs. The images are divided into grids, the homography matrix is ​​calculated using the feature point pairs, and the two images are smoothed and extrapolated by setting anchor points and overlapping area boundaries to complete the background image stitching. The foreground processing module is used to pre-process the two segmented foreground images, including performing the same processing on the foreground image according to the scaling ratio obtained for the background image, rotating the foreground image accordingly, and dividing the foreground image into grids. The local homography matrix between the grids is calculated based on the feature point pairs, and the foreground image is stitched together based on the local homography matrix. The image fusion module is used to calculate the offset based on the already stitched foreground and background images by finding matching feature points with high credibility during the stitching process. Based on the offset, the position of the foreground image in the background image is calculated, and the foreground and background are smoothly fused. Finally, gravity correction is performed according to the direction of gravity.

9. A computer device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the image stitching method based on foreground and background segmentation according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image stitching method based on foreground and background segmentation according to any one of claims 1 to 7 are implemented.

Images