Image splicing method and device based on foreground and background segmentation

Abstract

The invention discloses an image splicing method and device based on foreground and background segmentation. The method comprises the following steps: carrying out foreground and background segmentation on two images to be spliced, estimating a 2D rotation matrix between the two images according to angular velocity data acquired by a gyroscope, and estimating a gravity direction according to acceleration data when the two images are shot; preprocessing the images by using a rotation matrix between the two images, dividing the images into grids, and separately splicing the segmented foreground and background; according to requirements, the position of the final foreground in the background is calculated through the relative position of the foreground, fusion and splicing are carried out, and gravity correction can be selectively carried out on the final image. According to the method, the artifact problem occurring in the foreground and background image splicing is reduced, the excessive distortion of the edge is reduced, the dynamic foreground can be removed according to requirements, and the image splicing quality and accuracy are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to an image splicing method and device based on foreground and background segmentation. Background Art

[0002] Common image registration methods align two images by extracting feature regions of the images to calculate the best global transformation of the images to be stitched. They often perform well only in ideal near-planar scenes, that is, when the input images contain little or no parallax, the images can be aligned through global homography transformation. However, when the scene has complex situations such as large parallax, non-planar, multi-view shooting or lens distortion, a single homography matrix cannot effectively cope with these complex transformations and usually performs poorly, resulting in artifacts, misalignment, etc.

[0003] Therefore, some methods try to adapt to different areas of the image by dividing the image to be stitched and distorting them separately. From simply separating a single scene into a far plane and a ground plane, stitching is performed by calculating two homography transformations on the image to be stitched, which reduces artifacts to a certain extent, to gradually dividing the entire image area into a grid and calculating multiple local homography matrices to achieve better local adaptation, but introducing serious perspective distortion in non-overlapping areas. Subsequently, some methods strive to reduce the distortion of non-overlapping areas by methods such as shape preservation and setting global homography constraints. These image distortions have produced certain effects in reducing artifacts, etc., but it is difficult to achieve satisfactory results when facing image stitching with larger parallax. For large parallax, many studies use the stitching line method to stitch graphics to reduce or even eliminate artifacts. These methods have more or less produced certain effects, but there are still some problems. Summary of the invention

[0004] In view of the above-mentioned deficiencies of the prior art, the purpose of the present invention is to provide an image stitching method and device based on foreground and background segmentation to solve the problems of large parallax, artifacts and severe edge distortion that are difficult to solve in the prior art stitching.

[0005] In order to achieve the above invention object, the technical solution of the present invention is as follows:

[0006] In a first aspect, an image stitching method based on foreground and background segmentation comprises the following steps:

[0007] The two images to be stitched are segmented into foreground and background, the 2D rotation matrix between the two images is estimated based on the angular velocity data collected by the gyroscope, and the direction of gravity is estimated based on the acceleration data when the two images were taken;

[0008] For the two segmented background images, the image rotation angle is adjusted according to the estimated rotation matrix, and then the image is scaled, and the initial feature point pair vector is calculated. The SIFT feature point pair accelerated matching method is used to obtain the final matching feature point pair; the image is divided into grids, the homography matrix is ​​calculated using the feature point pairs, and the two images are smoothed and extrapolated by setting anchor points and overlapping area boundaries to complete the splicing of the background images;

[0009] Preprocessing the two segmented foreground images includes performing the same processing on the foreground image according to the scaling ratio obtained for the background image, rotating the foreground image accordingly to the background image, and dividing the foreground image into grids, calculating the local homography matrix between the grids according to the feature point pairs, and completing the splicing of the foreground image according to the local homography matrix;

[0010] According to the stitched foreground and background images, the offset is calculated by finding the matching feature point pairs with high credibility in the stitching process. The position of the foreground image in the background image is calculated based on the offset, the foreground and background are smoothly fused, and finally gravity correction is performed according to the gravity direction.

[0011] Furthermore, the two images are segmented into foreground and background, including:

[0012] The U2-Net model is used to separate the foreground target and divide the original image into the foreground image and the background image.

[0013] Furthermore, the 2D rotation matrix between the two images is estimated based on the angular velocity data collected by the gyroscope, including:

[0014] According to the gyroscope data collected at a specified frequency, the angular velocity data of the device is obtained, and the rotation of the device is obtained by integrating the collected angular velocity data; based on the conversion between the local coordinates of the device and the camera coordinate system, the initial body coordinate system of the inertial sensor is selected as the fixed coordinate system, and the 2D rotation matrix between the two images is derived according to the rotation of the device, which is recorded as C;

[0015] Estimate the direction of gravity based on the acceleration data when the two images were taken, including:

[0016] The acceleration data of the camera when it is stationary is collected, and the angle with the gravity direction is calculated based on the gravity data of the two images. The gravity correction rotation matrix is ​​calculated based on the two angles and is recorded as R.

[0017] Furthermore, the SIFT feature point pair acceleration matching method is used to obtain the final matching feature point pair, including:

[0018] The two background images are rotated and aligned to obtain two sets of feature points kp1 and kp2 and corresponding descriptors ds1 and ds2. The ideal position o of the feature point in the background image 2 corresponding to the feature point in the background image 1 is determined by using the pre-obtained vector relationship V. The candidate feature points within the tolerance range are found among the feature points of the background image 2 according to the preset tolerance range δ, and the valid matching pairs are selected by the preset distance threshold as the final matching feature point pairs.

[0019] Furthermore, the homography matrix is ​​calculated using the feature point pairs, and the two images are smoothed and extrapolated by setting the anchor points and the overlap region boundaries, including:

[0020] According to one of the matching point pairs in the two images obtained, it is converted into homogeneous coordinates, and the transformation relationship between the point pairs is established using the homography matrix. By constructing a linear equation system and minimizing the objective function Solve the homography matrix H, where a i is the coefficient matrix corresponding to each matching point, adding the constraint ||h|| 2 = 1 to ensure 8 degrees of freedom for isomorphic transformation;

[0021] Set anchor points at the edge of the image and use homography linearization for smooth extrapolation;

[0022] For any point q in the non-overlapping region, the anchor points in the boundary are linearized and a weighted average of the transformations is computed; when q is far away from an anchor point, all anchor points are given similar weights.

[0023] Furthermore, in the process of stitching the background images, similarity transformation is used in the non-overlapping regions of the two background images, specifically including:

[0024] Obtain the grid point coordinates of the image to be stitched and calculate the minimum distance from each grid point to the boundary of the overlapping area, and normalize the grid points based on the minimum distance; then use the Gaussian weighted kernel function to calculate the local affine transformation matrix, and maintain the numerical stability of the matrix through deconditioning and denormalization;

[0025] For each grid point, the distance normalization weight is calculated based on its distance to the overlapping area. In the non-overlapping area, the affine transformation matrix is ​​extrapolated using local linearization and smooth interpolation methods, while the affine transformation matrix is ​​kept unchanged in the overlapping area.

[0026] The affine transformation matrix and its inverse matrix are applied to the image stitching process to ensure smooth transition of the stitching edges and reduce distortion.

[0027] Furthermore, by finding matching feature point pairs with high credibility in the stitching process, the offset is calculated, and the position of the foreground image in the background image is calculated based on the offset, and the foreground and background images are smoothly fused, including:

[0028] According to the feature point pair of the foreground image, the center of the feature point pair is selected as a reference, and the centers of the feature points of the foreground image 1 and the foreground image 2 are calculated, which are recorded as g1 and g2. The parallax offset of the two foreground images is obtained by calculating the difference between g1 and g2. The position g1 of the foreground image 1 in the image to be stitched is calculated, and the position g1′ after the transformation is calculated. Then, the current new foreground position is obtained by adding the position offset of g to the position of g1′.

[0029] A foreground effective area mask is created according to the brightness of the foreground image, and the foreground is divided into a core area and an edge area through edge extension technology. In the edge area, distance transform is used to generate fusion weights, and the foreground pixels in the core area directly replace the background pixels through linear interpolation method, while the edge area achieves a natural transition through weighted interpolation of distance weights, completing the smooth fusion of the foreground image and the background image.

[0030] In a second aspect, an image stitching device based on foreground and background segmentation includes:

[0031] The foreground and background segmentation and attitude estimation module is used to segment the two images to be stitched into foreground and background, estimate the 2D rotation matrix between the two images based on the angular velocity data collected by the gyroscope, and estimate the gravity direction based on the acceleration data when the two images were taken;

[0032] The background processing module is used to adjust the rotation angle of the two segmented background images according to the estimated rotation matrix, then scale the images, calculate the initial feature point pair vectors, and use the SIFT feature point pair accelerated matching method to obtain the final matching feature point pair; divide the images into grids, use the feature point pairs to calculate the homography matrix, and perform smooth extrapolation operations on both images by setting anchor points and overlapping area boundaries to complete the splicing of the background images;

[0033] The foreground processing module is used to pre-process the two segmented foreground images, including performing the same processing on the foreground image according to the scaling ratio obtained for the background image, rotating the foreground image accordingly with the background image, and dividing the foreground image into grids, calculating the local homography matrix between the grids according to the feature point pairs, and completing the splicing of the foreground image according to the local homography matrix;

[0034] The image fusion module is used to calculate the offset based on the already stitched foreground and background images by finding matching feature point pairs with high credibility during the stitching process, calculate the position of the foreground image in the background image based on the offset, smoothly fuse the foreground and background, and finally perform gravity correction according to the gravity direction.

[0035] In a third aspect, the present invention further provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the image stitching method based on foreground and background segmentation as described in the first aspect are implemented.

[0036] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the image stitching method based on foreground and background segmentation as described in the first aspect are implemented.

[0037] Beneficial effects:

[0038] (1) The present invention proposes a foreground segmentation strategy for large parallax images. By splicing the foreground and background separately and then integrating them, the problems of shadows, dislocations, etc. caused by traditional splicing technology when facing depth changes or moving foregrounds are significantly reduced. At the same time, the dynamic foreground can be removed as needed, thereby improving the overall quality of the spliced ​​image.

[0039] (2) The present invention introduces IMU-based gravity-assisted correction technology, which can automatically perform gravity correction on images, thereby effectively correcting the image tilt problem. This technology avoids the incoordination effect caused by the tilt of the first image by adjusting the final stitching perspective to the overlapping area of ​​the image, ensuring the visual consistency and coordination of the stitching result.

[0040] (3) To address the time-consuming problem of SIFT (Scale-invariant Feature Ttransform) feature point matching, the present invention uses IMU to calculate the rotation matrix to complete preliminary image rotation alignment, and on this basis effectively estimates the vector relationship of feature points, thereby greatly accelerating the matching speed of feature points and improving the efficiency of image stitching.

[0041] (4) The present invention draws on the idea of ​​the mobile DLT (Direct Linear Transform) algorithm to smooth the image and set the global similarity and edge anchor points. Through weighted combination, the distortion of the two images in the stitching process is minimized, which effectively solves the edge distortion problem. At the same time, a fusion smoothing algorithm is designed to further improve the visual effect of the stitched image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the image stitching method based on foreground and background segmentation of the present invention;

[0043] Figure 2 A framework diagram for implementing a method in an embodiment of the present invention;

[0044] Figure 3 It is a schematic diagram of two images to be spliced ​​according to the present invention;

[0045] Figure 4 It is a schematic diagram of the foreground and background segmentation of two images to be spliced ​​according to the present invention;

[0046] Figure 5 Schematic diagram of two background images of the present invention;

[0047] Figure 6 Schematic diagram of two foreground images of the present invention;

[0048] Figure 7 It is a schematic diagram of the rotation alignment image of the present invention;

[0049] Figure 8 Schematic diagrams of the splicing results of the present invention, wherein (a) is a schematic diagram of the splicing result after removing the dynamic foreground, and (b) is a schematic diagram of the splicing result after fusing the foreground and background. DETAILED DESCRIPTION

[0050] The present invention uses an inertial measurement unit (IMU) to assist in image stitching, and uses the IMU to compensate for the lack of context during the stitching process. Using IMU data to estimate the displacement and rotation between images can provide a more accurate and reliable camera pose estimation, compensate the image stitching process, effectively eliminate ghosting and image information loss in traditional stitching methods, and obtain high-quality panoramic images. The participation of the IMU improves the matching between the images to be stitched, especially in situations where there are fewer overlapping areas, fewer feature point pairs, or serious lack of context information. It is difficult to obtain a satisfactory stitching effect relying solely on visual algorithms.

[0051] The present invention conducts research on large parallax, especially for the case where the depth difference between the foreground and background is large, and proposes an image stitching method based on foreground and background segmentation, which divides the image into two parts, the foreground and the background, for processing. For the background, even if there is some small parallax, a better effect can be achieved by using the grid method. Finally, the foreground and background that have been processed separately are recombined together, thereby avoiding the ghosting problem under large parallax. The present invention is assisted by inertial sensors. On the one hand, it speeds up the matching speed of feature point pairs, and on the other hand, it corrects the angle of the stitched image in the hope of obtaining an angle that the user is more interested in, thereby obtaining a more satisfactory effect.

[0052] Figure 1 The flow chart of the image stitching method based on foreground and background segmentation disclosed in the present invention is shown. Figure 1 , the method mainly includes: foreground and background segmentation and attitude estimation, background processing, foreground processing, and foreground and background fusion; wherein: in the foreground and background segmentation and attitude estimation stage, the two images to be spliced ​​are segmented into foreground and background, the 2D rotation matrix between the two images is estimated according to the angular velocity data collected by the gyroscope, and the gravity direction is estimated according to the acceleration data when the two images are taken. In the background processing stage and the foreground processing stage, the image is preprocessed using the rotation matrix between the two images, the image is divided into grids, and the segmented foreground and background are spliced ​​separately. In the foreground and background fusion stage, according to the needs, the position of the final foreground in the background is calculated by the relative position of the foreground and the fusion is spliced, and the final image can be optionally gravity corrected.

[0053] In one embodiment, a working method of image stitching based on foreground and background segmentation of the present invention comprises the following steps:

[0054] Step 1: Acquisition of images to be stitched and inertial sensor data.

[0055] like Figure 2 As shown in FIG. 1 , when shooting the image to be stitched, the inertial sensor data is collected at a frequency of 100 Hz, and the angular velocity data of the device is obtained according to the gyroscope data. The rotation of the device is obtained by integrating the collected angular velocity data ω. Furthermore, if the posture of the inertial sensor at time t is C(t), the posture at time t+δt is obtained using the data ω b =[ω bx ω by ω bz ] T pass To calculate, where is

[0056] The identity matrix, σ = |ω b δt|, Based on the conversion between the local coordinates of the device and the camera coordinate system, the 2D rotation matrix between the two images is derived, denoted as C. And the acceleration data collected at the time of the camera shooting is used to calculate accel = [accel x accel y accel z ]Use the inverse tangent function to calculate the angle with the direction of gravity, only consider the roll angle, and record it as roll, and get the gravity correction rotation matrix Denoted as R, the captured image is as follows Figure 3 The two images to be stitched are represented by (a) and (b) respectively, the same below.

[0057] Step 2: Preprocess the image to be stitched, including foreground and background segmentation, image rotation and scaling, etc.

[0058] The present invention uses U 2 -Net model performs foreground and background segmentation on the spliced ​​image. The segmentation effect is as follows Figure 4 As shown, Figure 5 To remove the foreground image, the image is rotated and aligned using the pre-obtained 2D rotation matrix C. Both images are rotated to prevent a single image from rotating too far. Both images are rotated to the middle overlapping area for alignment. Figure 7 As shown in the figure, since the background is far away from the camera, a simple 2D rotation can achieve good alignment, that is, the vector relationship between the feature points of the two images after a simple 2D rotation will approach an equal vector relationship, but the image scaling problem must be considered here to ensure that the scene sizes of the two images are consistent. Then, SURF feature point extraction and matching are used for the two images. In order to reduce errors and eliminate incorrect matching point pairs, the distance between adjacent feature points of the same image and the distance between corresponding adjacent feature points of the other image satisfy the same proportional relationship. At the same time, two-way matching is performed to obtain more accurate feature point pairs, and the two images are scaled according to the above proportional relationship. At this time, the feature point pairs of the images are close to satisfying the same vector relationship, which is denoted as V.

[0059] Step 3: Stitch the processed background images.

[0060] (1) By using IMU, the background image is rotated and aligned. At this time, the feature point pairs between the two images tend to satisfy the same vector relationship. Based on this feature, the SIFT feature point pairs are used to accelerate the matching model. First, two sets of feature points kp1 and kp2 and corresponding descriptors ds1 and ds2 are obtained. The feature points kp1 and kp2 record the information of the feature points detected in image one and image two, including the coordinates of the feature points. The descriptors ds1 and ds2 record the description information of the local area around the feature points in image one and image two, which are used to calculate the similarity of the matching feature point pairs. Then, the feature point position information recorded in kp1 obtained in image one and the pre-obtained vector relationship V can be used to determine the approximate position of the feature point in image two corresponding to the feature point in image one, and use this as a priori to determine the possible matching area. For each matching point m of image one, in order to obtain the corresponding m′ in image two, first use the vector relationship V to obtain the ideal position O of the corresponding feature point in image two, set a tolerance range δ, and search in this area, and then find the candidate feature points within the tolerance range among the feature points of image two, where the candidate feature points are the feature points with the greatest similarity calculated based on the descriptors ds1 and ds2. In order to further screen the matching points, the present invention sets a distance threshold for screening effective matching pairs, and further extracts feature point pairs with higher credibility from the successfully matched feature pairs.

[0061] (2) Using the acquired feature point pairs, using the local homography model, referring to the homography models of APAP (As-Projective-As-Possible) and AANAP (AdaptiveAs-Natural-As-Possible), the global homography matrix is ​​calculated based on the extracted feature point pairs, and then the image is divided into grids with a grid size of 100*100. Next, the local homography matrix of each grid is calculated using the feature point pairs. First, according to one of the matching point pairs m=[xy]T and m′=[x′ y′]T in the two acquired images, it is converted into homogeneous coordinates. and Use the homography matrix H∈R 3×3 Establish the transformation relationship between point pairs, expressed as By constructing a system of linear equations and minimizing the objective function Solve the homography matrix H, where a i is the coefficient matrix corresponding to each matching point. At the same time, the constraint ||h|| 2 = 1 to ensure the 8 degrees of freedom of the isomorphic transformation. On this basis, the weight ω is generated using the offset Gaussian according to the distance between the pixels. i,j =max(exp(-‖m i-m j || 2 / σ 2 ), γ). The parameter γ∈[0 1] is an offset used to prevent numerical problems. The weights are added to the minimization objective function and weighted minimization is used. To estimate the position m j The local homography at . Since corresponding feature point pairs only exist in the overlapping area, the local homography can only be calculated in the area where the reference image overlaps. For each pixel in the non-overlapping area, its transformation is calculated as a weighted linear combination of the local homographies in the overlapping area.

[0062] (3) Anchor points are set at the edge of the image and homography linearization is used for smooth extrapolation. The homography linearization of any point q in the neighborhood of anchor point p is expressed as h(q) = h(p) + J through the Taylor series of the homography transformation h(q). h (p)(qp)+o(||qp||), where J h (p) is the Jacobian of the homography h at point p. For any point q in the non-overlapping region, the anchor points in the boundary are linearized and the weighted average of the transformations is calculated. The weighted combination of the linearizations is The weight a i The Student's t weighted method is used to reduce perspective distortion by extrapolating the non-overlapping areas of the two images and avoid regional distortion caused by extrapolation of a single image.

[0063] (4) In order to further reduce image distortion and make the panorama look more natural, we first use a similarity transformation in the non-overlapping regions of the two images with a large threshold ε. g The RANSAC (RANdom SAmple Consensus) algorithm removes outliers in the match. After removing the outliers, the RANSAC algorithm with a smaller threshold ε1 is further used to find the plane homography matrix with the most inliers, where ε1 < ε g Finally, the rotation angles of each local similarity transformation are compared, and the transformation with the smallest rotation angle is selected as the final global similarity transformation to ensure that the alignment between images has high consistency and naturalness.

[0064] The calculated global similarity transformation is used to adjust the distortion of the image, and the weight is assigned by calculating the distance from the grid point to the boundary of the overlapping area. Specifically, for each grid point M i (Its coordinates are (x i ,y i )), calculate its distance to all edges on the boundary of the overlapping region, and find the minimum value d among these distances iThen, calculate the maximum value d among the minimum distances from all grid points to the boundary of the overlapping area max To normalize these distances, we transform the distance d of each grid point into i Map to the interval [0, 1] and get the weight coefficient a i and b i , expressed as b i =1-a i In the subsequent image transformation processing, different degrees of transformation interpolation and extrapolation are performed in the non-overlapping areas based on these weights. First, the second image is transformed, and the non-overlapping areas of the second image are gradually transformed by global similarity, which is expressed as Here i satisfy H i is the ith local homography matrix, is the updated local transformation, S1 is obtained by the global similarity transformation S, satisfying For the transformation of image 1, it can be expressed as here is the updated local transformation. For the overlapping area of ​​the two images, the calculated local homography matrix is ​​directly used for transformation, that is, Ht is directly used i Transform the overlapping area of ​​the second image using Hr i Transform the overlapping area of ​​the first image. Figure 8 As shown in (a) in the figure, the spliced ​​background image is gravity corrected to finally obtain the dynamic foreground image.

[0065] Step 4: Stitch the processed foreground image.

[0066] like Figure 6 As shown, the two foreground images are first preprocessed, the foreground is scaled according to the scaling ratio of the background, and the foreground image is rotated accordingly, and then directly spliced ​​according to the feature point pairs. When the two foreground images do not have enough available and highly reliable feature point pairs, the foreground image with a larger pixel range is directly used to retain more foreground information, or one of the foregrounds is selected as the final foreground through user interaction. Similarly, the foreground is gridded, and the linear transformation is calculated to calculate the local homography matrix between the grids, and finally the foreground images are spliced.

[0067] Step 5: Fusion of foreground and background and pixel smoothing.

[0068] Since the foreground and background are processed separately, the position of the foreground in the background must be calculated and fused when the foreground and background are fused. First, the position of the foreground in the background is calculated using the matching feature point pairs with higher credibility of the two images as a reference. Specifically, by extracting the feature point pairs of the foreground, the center of the feature point pairs is selected as a reference, and the feature point centers of foreground 1 and foreground 2 are calculated, which are recorded as g1 and g2. The parallax offset of the two foregrounds is obtained by calculating the difference between g1 and g2. Strictly speaking, it is the parallax offset of the foreground without considering the background offset, which can be expressed as g = g1-g2. The position g1 of the image to be spliced ​​where the foreground 1 is located is calculated, and the position g1′ after the transformation is calculated. Then, the position offset of g is added to the position of g1′ to obtain the current new foreground position. Then, a foreground effective area mask is created based on the brightness of the foreground image, and the foreground is divided into a core area and an edge area by edge extension technology. In the edge area, the distance transform is used to generate the fusion weight, and the foreground image is smoothly fused with the background image by the linear interpolation method. Finally, the foreground pixels in the core area directly replace the background pixels, while the edge area achieves a natural transition through weighted interpolation of distance weights, thus completing the smooth fusion processing of the image. Figure 8 As shown in (b), gravity correction is finally performed to obtain the foreground and background fusion result.

[0069] The present invention introduces gravity-assisted correction technology through inertial-vision fusion, and uses the inertial measurement unit (IMU) to perform gravity correction on the image. By collecting the camera posture during the image shooting process, the relative rotation matrix of the two images is estimated, which speeds up the matching of feature point pairs. Using the concept of the moving direct linear transform (DLT) algorithm, both images are smoothly extrapolated, and global similarity is used and anchor points are set to reduce the distortion of the two images.

[0070] Based on the same technical concept as the method embodiment, the present invention also provides an image stitching device based on foreground and background segmentation, comprising:

[0071] The foreground and background segmentation and attitude estimation module is used to segment the two images to be stitched into foreground and background, estimate the 2D rotation matrix between the two images based on the angular velocity data collected by the gyroscope, and estimate the gravity direction based on the acceleration data when the two images were taken;

[0072] The background processing module is the same as the two segmented background images. The image rotation angle is adjusted according to the estimated rotation matrix, and then the image is scaled, and the initial feature point pair vector is calculated. The SIFT feature point pair accelerated matching method is used to obtain the final matching feature point pair; the image is divided into grids, the homography matrix is ​​calculated using the feature point pairs, and the two images are smoothed and extrapolated by setting anchor points and overlapping area boundaries to complete the splicing of the background images;

[0073] The foreground processing module is used to pre-process the two segmented foreground images, including performing the same processing on the foreground image according to the scaling ratio obtained for the background image, rotating the foreground image accordingly with the background image, and dividing the foreground image into grids, calculating the local homography matrix between the grids according to the feature point pairs, and completing the splicing of the foreground image according to the local homography matrix;

[0074] The image fusion module is used to calculate the offset based on the already stitched foreground and background images by finding matching feature point pairs with high credibility during the stitching process, calculate the position of the foreground image in the background image based on the offset, smoothly fuse the foreground and background, and finally perform gravity correction according to the gravity direction.

[0075] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the image stitching method based on foreground and background segmentation as described above are implemented.

[0076] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image stitching method based on foreground and background segmentation as described above are implemented.

[0077] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices (systems), computer equipment or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0078] The present invention is described with reference to a flowchart of a method according to an embodiment of the present invention. It should be understood that each process in the flowchart and a combination of processes in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A device that specifies functions in a process or multiple processes.

[0079] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A function specified in a process or multiple processes.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 The steps of a specified function in a process or multiple processes.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An image stitching method based on foreground and background segmentation, characterized in that: The following steps are involved: The two images to be stitched are segmented into foreground and background, the 2D rotation matrix between the two images is estimated based on the angular velocity data collected by the gyroscope, and the direction of gravity is estimated based on the acceleration data when the two images were taken; For the two segmented background images, the image rotation angle is adjusted according to the estimated rotation matrix, and then the image is scaled, and the initial feature point pair vector is calculated. The SIFT feature point pair accelerated matching method is used to obtain the final matching feature point pair; the image is divided into a grid, and the local homography matrix is ​​calculated using the feature point pair. By setting the anchor point and the overlap area boundary, the two images are smoothed and extrapolated to complete the splicing of the background image; Preprocessing the two segmented foreground images includes performing the same processing on the foreground image according to the scaling ratio obtained for the background image, rotating the foreground image accordingly to the background image, and dividing the foreground image into grids, calculating the local homography matrix between the grids according to the feature point pairs, and completing the splicing of the foreground image according to the local homography matrix; According to the stitched foreground and background images, the offset is calculated by finding the matching feature point pairs with high credibility in the stitching process. The position of the foreground image in the background image is calculated based on the offset, the foreground and background images are smoothly fused, and finally gravity correction is performed according to the gravity direction.

2. The method according to claim 1, characterized in that Segment the two images into foreground and background, including: Using U 2 -Net model separates the foreground target and divides the original image into foreground image and background image.

3. The method according to claim 1, characterized in that Estimate the 2D rotation matrix between the two images based on the angular velocity data collected by the gyroscope, including: According to the gyroscope data collected at a specified frequency, the angular velocity data of the device is obtained, and the rotation of the device is obtained by integrating the collected angular velocity data; based on the conversion between the local coordinates of the device and the camera coordinate system, the initial body coordinate system of the inertial sensor is selected as the fixed coordinate system, and the 2D rotation matrix between the two images is derived according to the rotation of the device, which is recorded as C; Estimate the direction of gravity based on the acceleration data when the two images were taken, including: The acceleration data of the camera when it is stationary is collected, and the angle with the gravity direction is calculated based on the gravity data of the two images. The gravity correction rotation matrix is ​​calculated based on the two angles and is recorded as R.

4. The method according to claim 1, characterized in that The SIFT feature point pair acceleration matching method is used to obtain the final matching feature point pair, including: The two background images are rotated and aligned to obtain two sets of feature points kp1 and kp2 and corresponding descriptors ds1 and ds2. The ideal position O of the feature point in the background image 2 corresponding to the feature point in the background image 1 is determined by using the pre-obtained vector relationship V. The candidate feature points within the tolerance range are found among the feature points of the background image 2 according to the preset tolerance range δ, and the valid matching pairs are selected by the preset distance threshold as the final matching feature point pairs.

5. The method according to claim 1, characterized in that The homography matrix is ​​calculated using feature point pairs, and both images are smoothed and extrapolated by setting anchor points and overlapping area boundaries, including: According to one of the matching point pairs in the two images obtained, it is converted into homogeneous coordinates, and the transformation relationship between the point pairs is established using the homography matrix. By constructing a linear equation system and minimizing the objective function Solve the homography matrix H, where a i is the coefficient matrix corresponding to each matching point, adding the constraint ‖h‖ 2 = 1 to ensure 8 degrees of freedom for isomorphic transformation; Set anchor points at the edge of the image and use homography linearization for smooth extrapolation; For any point q in the non-overlapping region, the anchor points in the boundary are linearized and a weighted average of the transformations is computed; when q is far away from an anchor point, all anchor points are given similar weights.

6. The method according to claim 5, characterized in that In the process of stitching the background images, similarity transformation is used in the non-overlapping areas of the two background images, including: Obtain the grid point coordinates of the image to be stitched and calculate the minimum distance from each grid point to the boundary of the overlapping area, and normalize the grid points based on the minimum distance; then use the Gaussian weighted kernel function to calculate the local affine transformation matrix, and maintain the numerical stability of the matrix through deconditioning and denormalization; For each grid point, the distance normalization weight is calculated based on its distance to the overlapping area. In the non-overlapping area, the affine transformation matrix is ​​extrapolated using local linearization and smooth interpolation methods, while the affine transformation matrix is ​​kept unchanged in the overlapping area. The affine transformation matrix and its inverse matrix are applied to the image stitching process to ensure smooth transition of the stitching edges and reduce distortion.

7. The method according to claim 1, characterized in that By finding matching feature point pairs with high credibility in the stitching process, the offset is calculated, and the position of the foreground image in the background image is calculated based on the offset, and the foreground and background images are smoothly fused, including: According to the feature point pair of the foreground image, the center of the feature point pair is selected as a reference, and the center of the feature point of foreground image 1 and foreground image 2 is calculated, recorded as g1 and g2. The parallax offset of the two foreground images is obtained by calculating the difference between g1 and g2, and the position g1 of the foreground image 1 in the image to be stitched is calculated, and the position g1 after the transformation is calculated. ′ , then in g1 ′ Add the position offset of g to the position of to get the current new foreground position; A foreground effective area mask is created according to the brightness of the foreground image, and the foreground is divided into a core area and an edge area through edge extension technology. In the edge area, distance transform is used to generate fusion weights, and the foreground pixels in the core area directly replace the background pixels through linear interpolation method, while the edge area achieves a natural transition through weighted interpolation of distance weights, completing the smooth fusion of the foreground image and the background image.

8. An image splicing device based on foreground and background segmentation, characterized in that: include: The foreground and background segmentation and attitude estimation module is used to segment the two images to be stitched into foreground and background, estimate the 2D rotation matrix between the two images based on the angular velocity data collected by the gyroscope, and estimate the gravity direction based on the acceleration data when the two images were taken; The background processing module is used to adjust the rotation angle of the two segmented background images according to the estimated rotation matrix, then scale the images, calculate the initial feature point pair vectors, and use the SIFT feature point pair accelerated matching method to obtain the final matching feature point pair; divide the images into grids, use the feature point pairs to calculate the homography matrix, and perform smooth extrapolation operations on both images by setting anchor points and overlapping area boundaries to complete the splicing of the background images; The foreground processing module is used to pre-process the two segmented foreground images, including performing the same processing on the foreground image according to the scaling ratio obtained for the background image, rotating the foreground image accordingly with the background image, and dividing the foreground image into grids, calculating the local homography matrix between the grids according to the feature point pairs, and completing the splicing of the foreground image according to the local homography matrix; The image fusion module is used to calculate the offset based on the already stitched foreground and background images by finding matching feature point pairs with high credibility during the stitching process, calculate the position of the foreground image in the background image based on the offset, smoothly fuse the foreground and background, and finally perform gravity correction according to the gravity direction.

9. A computer device, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the image stitching method based on foreground and background segmentation as described in any one of claims 1-7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image stitching method based on foreground and background segmentation according to any one of claims 1 to 7 are implemented.

Images