Panoramic image stitching method and computer readable storage medium

Abstract

This invention discloses a panoramic image stitching method and a computer-readable storage medium, relating to the field of image processing technology. The method includes: performing feature filtering on adjacent original images to obtain a first homography matrix; performing preliminary alignment of the adjacent original images based on the first homography matrix to obtain a first aligned adjacent image and its corresponding first image overlap region; performing a difference operation on the first image overlap region to obtain a motion binary mask; calculating the Euclidean distance from each pixel in the first image overlap region to the image center of the first aligned adjacent image; performing a mask forced zeroing operation based on the motion binary mask and the Euclidean distances to obtain a binary fusion mask; and optimizing the stitching line along the object contour based on the binary fusion mask and the adjacent original images to obtain a panoramic fused image. This solves the problem of semi-transparent ghosting caused by traditional fusion algorithms.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a panoramic image stitching method and a computer-readable storage medium. Background Technology

[0002] Current panoramic stitching and fusion solutions in the industry suffer from the following technical pain points: when capturing dynamic scenes with moving targets such as pedestrians and vehicles, the stitching results exhibit severe "ghosting" or "object truncation" phenomena. Existing optimization solutions (such as the traditional GraphCut stitching algorithm) attempt to find the optimal path, but their cost function relies solely on static color or gradient features, lacking the ability to perceive moving objects and thus failing to identify and avoid them, resulting in the stitching line passing through the middle of the moving target. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to overcome the shortcomings of the prior art and provide a panoramic image stitching method and a computer-readable storage medium.

[0004] This invention provides the following technical solution: In a first aspect, the present invention provides a panoramic image stitching method, the method comprising: The adjacent original images are scaled to obtain adjacent working images; the first homography matrix is ​​obtained by feature filtering of the adjacent working images. The adjacent original images are initially aligned according to the first homography matrix to obtain the first aligned adjacent images and their corresponding first image overlap regions; Perform a difference operation on the overlapping region of the first image to obtain a motion binary mask; Calculate the Euclidean distance from each pixel in the overlapping region of the first image to the image center of the first aligned adjacent image; Based on the Euclidean distance from each pixel in the overlapping region of the first image to the image center of the first aligned adjacent image, a mask forced zeroing operation is performed to obtain a binary fusion mask; Based on the binary fusion mask and the adjacent original images, the stitching line is optimized along the object contour to obtain a panoramic fusion image.

[0005] In an optional implementation, the first aligned adjacent images include a first aligned left image and a first aligned right image, the binary fusion mask includes a left binary mask and a right binary mask, and the step of performing a mask forced zeroing operation based on the Euclidean distance from each pixel in the overlapping area of ​​the motion binary mask and the first image to the image center of the first aligned adjacent image to obtain the binary fusion mask includes: Based on the preset mask value and the motion binary mask, the target region is determined from the overlapping region of the first image; From the Euclidean distances from each pixel in the overlapping region of the first image to the image center of the first aligned adjacent image, obtain the Euclidean distances from each pixel in the target region to the image center of the first aligned left image, and the Euclidean distances from each pixel in the target region to the image center of the first aligned right image. For each pixel in the target region, if the Euclidean distance from the pixel in the target region to the center of the first aligned left image is greater than or equal to the Euclidean distance from the pixel in the target region to the center of the first aligned right image, then the value of the pixel in the target region in the left binary mask is set to 0. If the Euclidean distance from a pixel in the target region to the center of the first aligned left image is less than the Euclidean distance from a pixel in the target region to the center of the first aligned right image, then the value of the pixel in the target region in the right binary mask is set to 0.

[0006] In an optional implementation, the step of optimizing the stitching line along the object contour based on the binary fusion mask and the adjacent original images to obtain a panoramic fusion image includes: The first homography matrix is ​​scaled to obtain the second homography matrix; The adjacent original images are initially aligned according to the second homography matrix to obtain the second aligned adjacent images and their corresponding second image overlap regions; The panoramic fused image is obtained by optimizing the stitching line along the object contour based on the binary fusion mask, the second aligned adjacent image and the corresponding second image overlapping area.

[0007] In an optional implementation, the second aligned adjacent images include a second aligned left image and a second aligned right image. The step of optimizing the stitching line along the object contour based on the binary fusion mask, the second aligned adjacent images, and their corresponding overlapping areas to obtain the panoramic fused image includes: Construct an energy function, the expression of which is: , In the formula, This indicates that pixel p in the overlapping region of the second image will be forcibly assigned to the label. The cost, A value of 0 indicates that pixel p in the overlapping region of the second image is in the left image after the second alignment. , A value of 1 indicates that pixel p in the overlapping region of the second image is in the image to the right after the second alignment. , Indicates adjacent pixels in the overlapping region of the second image In the label and tags Punishment at that time A value of 0 indicates that pixel q is in the left image after the second alignment. , A value of 1 indicates that pixel q is in the right image after the second alignment. , This represents the pixel value at pixel p in the second aligned left image. This represents the pixel value at pixel q in the left image after the second alignment. This represents the pixel value at pixel p in the second aligned image on the right. This represents the pixel value at pixel q in the right-hand image after the second alignment; A weighted undirected graph is constructed based on the overlapping region of the second image, and the energy function is solved by maximum flow and minimum cut based on the binary fusion mask and the weighted undirected graph to obtain the target label of each pixel in the overlapping region of the second image. Based on the target labels of each pixel in the overlapping region of the second image, generate the left image corresponding to the second aligned left image. Figure 2 Value mask, and the right image corresponding to the second aligned right image. Figure 2 Value mask; According to the left Figure 2 Value mask, the right Figure 2 The value mask, the second aligned left image, and the second aligned right image are stitched together to obtain the panoramic fused image.

[0008] In an optional implementation, the step of... Figure 2 Value mask, the right Figure 2 The panoramic fused image is obtained by stitching together the value mask, the second aligned left image, and the second aligned right image, including: According to the left Figure 2 Value mask and the right Figure 2 Value masking generates target weight map; The second aligned left image and the second aligned right image are weighted and fused according to the weight map to obtain the panoramic fused image.

[0009] In an optional implementation, the step of... Figure 2 Value mask and the right Figure 2 Value mask generation of target weight map includes: For the left Figure 2 Value mask and the right Figure 2 The value mask is morphologically expanded to obtain the expanded mask; The expanded mask is subjected to subpixel-level Gaussian blurring to obtain an initial weight map; Boundary safety locking is performed on the initial weight map to obtain the target weight map.

[0010] In an optional implementation, the step of establishing a weighted undirected graph based on the overlapping region of the second image includes: A source point and a sink point are preset, wherein the source point is used to indicate the second aligned left-side image, and the sink point is used to indicate the second aligned left-side image; Each pixel in the overlapping region of the second image, as well as the source point and the sink point, are used as nodes to determine the data edge and the smoothing edge; The weighted undirected graph is constructed based on the nodes, the data edges, and the smoothed edges.

[0011] In an optional implementation, the step of weighted fusing of the second aligned left image and the second aligned right image according to the weight map to obtain the panoramic fused image includes: Perform HSV space color safety compensation on the second aligned left image and the second aligned right image to obtain the compensated left image and the compensated right image; The compensated left image and the compensated right image are weighted and fused according to the weight map to obtain the panoramic fused image.

[0012] In an optional implementation, obtaining the first homography matrix by performing feature filtering on the adjacent working images includes: Based on the ORB operator, feature extraction is performed on the adjacent working images to obtain candidate feature points; The candidate feature points are matched based on the Hamming distance to obtain the matched feature points. The first homography matrix is ​​obtained by iteratively filtering the matching feature points based on the random sampling consensus algorithm.

[0013] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the panoramic image stitching method as described in any of the foregoing embodiments.

[0014] The panoramic image stitching method and computer-readable storage medium disclosed in this invention involve scaling adjacent original images to obtain adjacent working images; obtaining a first homography matrix by feature filtering of the adjacent working images; performing preliminary alignment of the adjacent original images based on the first homography matrix to obtain a first aligned adjacent image and its corresponding first image overlap region; performing a difference operation on the first image overlap region to obtain a motion binary mask; calculating the Euclidean distance from each pixel in the first image overlap region to the image center of the first aligned adjacent image; performing a mask forced zeroing operation based on the motion binary mask and the Euclidean distance from each pixel in the first image overlap region to the image center of the first aligned adjacent image to obtain a binary fusion mask; and optimizing the stitching line along the object contour based on the binary fusion mask and the adjacent original images to obtain a panoramic fused image. In this way, the dynamic displacement of the overlapping area of ​​the image is first identified. When a significant pixel difference is detected (i.e., a moving object is identified), the mask of the interference area is forced to be suppressed and zeroed by combining the Euclidean distance of each pixel to the center of the image, and a binary fusion mask is obtained. The binary fusion mask guides the stitching line algorithm to perform global path optimization along the physical contour of the moving target, thereby avoiding visually sensitive smooth misalignment areas or object cut-off points, and solving the problem of semi-transparent afterimages caused by traditional fusion algorithms. Attached Figure Description

[0015] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope of protection of the present invention. In the various drawings, similar components are numbered similarly.

[0016] Figure 1 A flowchart of the panoramic image stitching method proposed in this embodiment is shown; Figure 2 Another flowchart of the panoramic image stitching method proposed in this embodiment is shown; Figure 3 This illustration shows another flowchart of the panoramic image stitching method proposed in this embodiment; Figure 4 This diagram illustrates another step in the panoramic image stitching method proposed in this embodiment. Figure 5 A schematic diagram of another process of the panoramic image stitching method proposed in this embodiment is shown; Figure 6 This embodiment presents a schematic diagram of panoramic image stitching comparison. Figure 7 A schematic diagram of the panoramic image stitching device proposed in this embodiment is shown.

[0017] Explanation of reference numerals in the attached diagram: 700 - Panoramic image stitching device; 701 - Processing module; 702 - Alignment module; 703 - Differentiation module; 704 - Calculation module; 705 - Zeroing module; 706 - Fusion module. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0019] The components of the embodiments of the invention described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0020] In the following, the terms “comprising,” “having,” and their cognates, which may be used in various embodiments of the invention, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as excluding, firstly, the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations thereof.

[0021] Furthermore, the terms "first," "second," and "third" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0022] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.

[0023] Example 1 This disclosure provides a panoramic image stitching method.

[0024] Please see Figure 1 The panoramic image stitching method includes steps S101 to S106, and each step is described in detail below.

[0025] Step S101: Scale the adjacent original images to obtain adjacent working images; obtain the first homography matrix by performing feature filtering on the adjacent working images.

[0026] In this embodiment, the original image is captured in real time, and two adjacent original images are defined as adjacent original images.

[0027] Understandably, the original image may exhibit brightness attenuation at the lens edges, which can be addressed by applying a radial r2-based method. r The vignetting correction algorithm based on the 2-gain model pre-calculates the light intensity compensation coefficient and restores the edge brightness pixel by pixel in the image preprocessing stage to obtain the original image after removing the vignetting, which serves as the data basis for subsequent processing.

[0028] Furthermore, adjacent original images are scaled proportionally to a preset resolution (e.g., 1024×1024px) to obtain adjacent working images, which improves the subsequent calculation speed, reduces the complexity of feature matching calculation, and retains sufficient texture information for ORB (Oriented FAST and Rotated BRIEF) extraction; thus avoiding slow RANSAC iteration convergence or increased mismatch rate due to excessively high original image resolution.

[0029] Furthermore, feature filtering is performed on adjacent working images to obtain the first homography matrix.

[0030] In one specific embodiment, step S101 includes: extracting features from the adjacent working images based on the ORB operator to obtain candidate feature points; performing feature matching on the candidate feature points based on the Hamming distance to obtain matching feature points; and iteratively filtering the matching feature points based on the random sampling consensus algorithm to obtain the first homography matrix.

[0031] In this embodiment, ORB operator feature extraction involves using the Features from Accelerated Segment Test (FAST) algorithm to quickly identify pixels with drastic grayscale changes in two adjacent working images. Potential feature points are rapidly selected by comparing the brightness difference between a candidate pixel and its 16 surrounding pixels. Furthermore, to achieve rotational transformation, the algorithm calculates the grayscale centroid within the neighborhood of each feature point. .vector ( The direction of the geometric center is the principal direction of the feature point. , ,in, Moments within the neighborhood, , yes When taking (0,1) and (1,0) respectively, from The first-order component derived from the first-order component is combined with the first-order component to calculate the gray-level centroid and the principal direction of the ORB feature. Further, based on the calculated directions... The random point pairs of the BRIEF operator are rotated and translated to generate a 256-bit binary string with rotational description capability, thereby obtaining candidate feature points.

[0032] Hamming distance feature matching: Hamming distance is used as the metric for binary descriptors generated by ORB. Calculation logic: An XOR operation is performed on two 256-bit binary strings, and the number of "1"s in the result is counted. Furthermore, the CPU's POPCNT instruction set is used to accelerate the calculation, quickly finding the candidate points with the smallest and second smallest Hamming distances between adjacent images. Further, to eliminate ambiguous matches, only point pairs whose ratio of nearest neighbor distance to second nearest neighbor distance is less than a threshold (e.g., 0.75) are retained as matching feature points, ensuring the uniqueness and accuracy of the match.

[0033] The random sampling consensus algorithm iteratively filters: Four pairs of non-collinear feature points are randomly selected from the set of matching feature points. Further, using a direct linear transformation algorithm, the coordinates of the point pairs are substituted into a system of linear equations. Candidate matrices are obtained by Singular Value Decomposition (SVD). Furthermore, all matching feature points Substitution Mapped to another image plane to obtain Calculate the actual observation points With mapping point Euclidean distance between (Reprojection error): In the formula, Let x be the actual measured x-coordinate (image pixel coordinate system) of the i-th matching feature point on the second image. To use the current candidate matrix The predicted x-coordinate of the i-th feature point on the second image, derived by mapping, with the hat symbol ^ representing the model's predicted value; This represents the actual measured ordinate of the i-th matching feature point in the second image; For candidate matrix The predicted ordinate of the i-th feature point on the second image, obtained through geometric transformation. Further, an error threshold is set. (e.g., 3.0 pixels). If If the point is identified as an "inlier", the above steps are repeated (usually more than 2000 iterations), and the number of inliers with the maximum number of inliers is recorded. , which serves as the first homography matrix. Each pair of adjacent original images corresponds to one first homography matrix.

[0034] Understandably, robust, sparse but high-confidence feature correspondences are obtained while ensuring real-time performance (ORB is lightweight); RANSAC eliminates mismatches and significantly improves the geometric accuracy of the first homography matrix.

[0035] Step S102: The adjacent original images are initially aligned according to the first homography matrix to obtain the first aligned adjacent images and their corresponding first image overlap regions.

[0036] In this embodiment, the two images in adjacent original images are mapped to the same global canvas coordinate system by the first homography matrix to achieve preliminary alignment, thereby obtaining the first aligned adjacent images and their corresponding first image overlap areas.

[0037] It should be noted that the pre-calibration matrix extracted from the YAML file can be used. Preliminary alignment is performed on adjacent original images. This includes a pre-calibration matrix. The homography matrix is ​​a 3×3 matrix describing the position of a point in one image after matrix multiplication and transformation to the corresponding position in another image. The parameters are pre-calculated calibration data under ideal texture conditions (such as feature-rich scenes). Four images are stitched together each time, resulting in three homography matrices. The steps to obtain the calibration parameter YAML file are as follows: In a standard scene, a set of original images is captured using a fixed-position stitching camera. Feature detection algorithms are used to perform feature matching on adjacent images, and mismatched points are removed using the RANSAC algorithm to calculate the initial homography matrix. The calculated homography matrix is ​​serialized using a Python script, converting the file format into a readable YAML format.

[0038] Step S103: Perform a difference operation on the overlapping region of the first image to obtain a motion binary mask.

[0039] In this embodiment, an absdiff operation is performed on the overlapping region of the first image to compare the pixel value differences at the same spatial location within the overlapping region. When the pixel value difference exceeds a preset threshold, it is marked as an object displacement region. It accurately locates moving objects such as pedestrians and vehicles, and then outputs a motion binary mask. Here, 1 represents the object displacement region, and 0 represents the non-object displacement region.

[0040] Understandably, separating dynamic targets (pedestrians / vehicles) from the static background provides a binary spatial constraint for the stitching line to avoid moving areas, thereby solving the "ghosting / truncation" problem in the panoramic image stitching process.

[0041] Step S104: Calculate the Euclidean distance from each pixel in the overlapping region of the first image to the image center of the first aligned adjacent image.

[0042] In this embodiment, the first aligned adjacent images include the first aligned left image and the first aligned right image. The Euclidean distance from each pixel in the overlapping area of ​​the first images to the image center of the first aligned left image and the first aligned right image is calculated.

[0043] Understandably, pixels closer to the center have higher visual importance, smaller viewing angle distortion, and stronger texture reliability; this distance field provides a spatial weight basis for subsequent forced zeroing.

[0044] Step S105: Based on the motion binary mask and the Euclidean distance from each pixel in the overlapping region of the first image to the image center of the first aligned adjacent image, perform a mask forced zeroing operation to obtain a binary fusion mask.

[0045] In this embodiment, the object displacement region in the overlapping region of the first image is determined according to the motion binary mask, and then the Euclidean distance from each pixel in the object displacement region to the center of the first aligned left image and the first aligned right image is used to perform a mask forced zeroing operation to obtain a binary fusion mask.

[0046] Please see Figure 2 In one specific embodiment, the binary fusion mask includes a left binary mask and a right binary mask. Step S105 includes steps S1051 to S1053. Each step is described in detail below.

[0047] Step S1051: Determine the target region from the first image overlap region based on the preset mask value and the motion binary mask.

[0048] In this embodiment, the preset mask value is 1, and the motion binary mask in the overlapping area of ​​the first image is used to... The object's displacement region is taken as the target region.

[0049] Step S1052: Obtain the Euclidean distances from each pixel in the target region to the center of the first aligned left image and from each pixel in the target region to the center of the first aligned right image, based on the Euclidean distances from each pixel in the overlapping region of the first image to the center of the first aligned adjacent image.

[0050] In this embodiment, the Euclidean distances from each pixel in the overlapping region of the first image to the center of the first aligned left image and the first aligned right image are obtained. And the Euclidean distance from each pixel in the target region to the image center of the first aligned right image. .

[0051] Step S1053: For each pixel in the target area, if the Euclidean distance from the pixel in the target area to the center of the first aligned left image is greater than or equal to the Euclidean distance from the pixel in the target area to the center of the first aligned right image, then the value of the pixel in the target area in the left binary mask is set to 0; if the Euclidean distance from the pixel in the target area to the center of the first aligned left image is less than the Euclidean distance from the pixel in the target area to the center of the first aligned right image, then the value of the pixel in the target area in the right binary mask is set to 0.

[0052] In this embodiment, for each pixel in the target region, if This indicates that the object is closer to the center in the left image (resulting in better image quality). Therefore, the binary mask on the right should be applied. Forced assignment of 0 preserves the left side image after the first alignment; if Conversely, the left-side binary mask is then used. By forcibly assigning a value of 0, the right-side image after the first alignment is retained, which significantly reduces the risk of suture misalignment caused by unilateral image distortion.

[0053] Step S106: Based on the binary fusion mask and the adjacent original images, the stitching line is optimized along the object contour to obtain a panoramic fusion image.

[0054] In this embodiment, a panoramic fused image is obtained by using a binary fusion mask as a constraint guide and combining the stitching lines of adjacent original images along the object contour to avoid visually sensitive smooth misalignment areas or object cut-off points, thus solving the problem of semi-transparent afterimages caused by traditional fusion algorithms.

[0055] Please see Figure 3 In one specific embodiment, step S106 includes steps S1061 to S1063, and each step is described in detail below.

[0056] Step S1061: Scale the first homography matrix to obtain the second homography matrix.

[0057] In this embodiment, by adjusting the first homography matrix Perform a scaling transformation on the first homography matrix. Restored to the original high-resolution space, the second homography matrix is ​​obtained. The scaling transformation expression is as follows: In the formula, The scaling matrix for downsampling the image.

[0058] Step S1062: Perform preliminary alignment of the adjacent original images according to the second homography matrix to obtain the second aligned adjacent images and their corresponding second image overlap regions.

[0059] In this embodiment, the two images in adjacent original images are mapped to the same global canvas coordinate system using the second homography matrix to achieve preliminary alignment. Then, the adjacent images after the second alignment and their corresponding overlapping areas are obtained, providing a sub-pixel level alignment basis for the final stitching. This ensures that the pixel value difference in the energy function truly reflects the color / texture difference, rather than the registration residual, thereby improving the semantic accuracy of the segmentation.

[0060] Step S1063: Based on the binary fusion mask, the second aligned adjacent images and their corresponding overlapping areas, the stitching line is optimized along the object contour to obtain the panoramic fusion image.

[0061] In this embodiment, a binary mask is used to constrain available pixels, and the optimal stitching line is determined along the object contour in the overlapping area of ​​the second image. The optimal stitching line is then used to fuse the adjacent images after the second alignment to generate a panoramic fused image.

[0062] Please see Figure 4 In one specific embodiment, the second aligned adjacent images include the second aligned left image and the second aligned right image. Step S1064 includes steps S401 to S404. Each step is described in detail below.

[0063] Step S401: Construct the energy function.

[0064] In this embodiment, the expression for the energy function is: , In the formula, data items This indicates that the pixels in the overlapping area of ​​the second image will be... p Forced assignment to label The cost, A value of 0 indicates pixels in the overlapping region of the second image. p Left image after second alignment , A value of 1 indicates a pixel in the overlapping region of the second image. p Right side image after second alignment Smoothing term Indicates adjacent pixels in the overlapping region of the second image In the label and tags Punishment at that time A value of 0 indicates pixels in the overlapping region of the second image. q Left image after second alignment , A value of 1 indicates a pixel in the overlapping region of the second image. q Right side image after second alignment , This indicates the pixels in the left image after the second alignment. p Pixel value at that location, This indicates the pixels in the left image after the second alignment. q Pixel value at that location, Indicates the pixels in the right-hand image after the second alignment. p Pixel value at that location, Indicates the pixels in the right-hand image after the second alignment. q Pixel value at; Step S402: A weighted undirected graph is established based on the overlapping region of the second image, and the energy function is solved by maximum flow and minimum cut based on the binary fusion mask and the weighted undirected graph to obtain the target label of each pixel in the overlapping region of the second image.

[0065] In this embodiment, a weighted undirected graph is established based on the pixels in the overlapping region of the second image.

[0066] Specifically, a source point and a sink point are preset, with the source point indicating the left side of the second aligned image and the sink point indicating the left side of the second aligned image; each pixel in the overlapping area of ​​the second image, as well as the source point and the sink point, are used as nodes to determine data edges and smoothing edges; a weighted undirected graph is constructed based on the nodes, data edges, and smoothing edges.

[0067] Furthermore, the binary fusion mask is used as a data constraint, so that the stitching line naturally avoids moving objects during the optimization process along the object contour. At the same time, the smoothing term is based on the pixel neighborhood gradient, which guides the stitching line to follow the object contour (rather than cross it). The Boykov-Kolmogorov algorithm is used to solve the energy function by maximum flow and minimum cut, so as to obtain the target label of each pixel in the overlapping area of ​​the second image, that is, the position of the optimal stitching line.

[0068] Understandably, gradient information is introduced into the smoothing term. When the gradient directions of the two images are consistent and their magnitudes are similar at a certain point, the cutting cost at that point is the lowest. At the same time, the algorithm will automatically bypass areas with huge color differences (such as moving pedestrians and vehicles) to ensure that the stitching line does not cross dynamic targets.

[0069] Step S403: Generate the left image corresponding to the second aligned left image based on the target labels of each pixel in the overlapping region of the second image. Figure 2 Value mask, and the right image corresponding to the second aligned right image. Figure 2 Value mask.

[0070] In this embodiment, based on the target labels of each pixel in the overlapping region of the second image, left and right images after alignment are respectively generated for the second aligned left image and the second aligned right image. Figure 2 Value mask and right Figure 2 Value mask. For example, one side of the suture line is 1, and the other side is 0.

[0071] Step S404, according to the left Figure 2 Value mask, the right Figure 2 The value mask, the second aligned left image, and the second aligned right image are stitched together to obtain the panoramic fused image.

[0072] In this embodiment, according to the left Figure 2 Value mask, right Figure 2 The value mask is used to fuse the left image after the second alignment and the right image after the second alignment to obtain a panoramic fused image.

[0073] Please see Figure 5 In one specific embodiment, step S404 includes steps S4041 to S4042, and each step is described in detail below.

[0074] Step S4041, according to the left Figure 2 Value mask and the right Figure 2 Value mask generates target weight map.

[0075] In this embodiment, according to the left Figure 2 Value mask and right Figure 2 Value mask generates target weight map, achieving subpixel smooth transition of the stitching line, balancing center sharpness and edge naturalness, and completely eliminating ghosting and black borders.

[0076] In one specific embodiment, step S4041 includes: [the following steps are described in the original text, but are not directly related to the previous sentence] Figure 2 Value mask and the right Figure 2 Morphological dilation is performed on the value mask to obtain a dilated mask; subpixel-level Gaussian blur is applied to the dilated mask to obtain an initial weight map; boundary safety locking is applied to the initial weight map to obtain the target weight map.

[0077] In this embodiment, for the left Figure 2 Value mask and right Figure 2 The value mask is expanded over a very small range to fill the tiny gaps that may have been generated during the discretization process, resulting in an expanded mask.

[0078] The dilated mask is smoothed using a sub-pixel-level Gaussian kernel to obtain an initial weight map with an extremely narrow gradient band. The Gaussian kernel size is limited to... arrive Between. This operation is not intended to achieve long color transitions, but rather to perform anti-aliasing only at the edges. The feathering width is controlled within. Within the pixel range.

[0079] Further, a logical AND operation is performed between the initial weight map and the "original image effective region" to achieve boundary security locking, thereby obtaining the target weight map. This ensures that the weights after blurring do not spread outside the image, preventing black borders from appearing at the seams.

[0080] Step S4042: Perform weighted fusion of the second aligned left image and the second aligned right image according to the weight map to obtain the panoramic fused image.

[0081] In this embodiment, the panoramic fused image The calculation formula is: .because In most areas, the value is 0 or 1, with a gradient only present in a very narrow range of the seam. The final effect is that the center of the image retains its original sharpness, and the seam disappears completely under the naked eye.

[0082] In one specific embodiment, step S4042 includes: performing HSV space color safety compensation on the second aligned left image and the second aligned right image to obtain a compensated left image and a compensated right image; and performing weighted fusion on the compensated left image and the compensated right image according to the weight map to obtain the panoramic fused image.

[0083] In this embodiment, the second aligned left image and the second aligned right image are converted to the HSV color space for safe brightness alignment, resulting in a compensated left image and a compensated right image. Specifically, hue is fixed: the H channel remains absolutely unchanged to ensure no hue shift occurs; brightness gain: the gain coefficient g is calculated based on the brightness difference in the overlapping area and applies only to the V (Value) channel; saturation roll-off: the S (Saturation) channel undergoes global adaptive desaturation processing of 0.85 to 0.9 to eliminate color temperature differences at the seams.

[0084] Please see Figure 6The green box shows the panoramic fusion image generated by the panoramic image stitching method of this embodiment, while the red box shows the panoramic fusion image generated by the traditional panoramic image stitching method. It can be seen that the panoramic fusion image generated by the traditional panoramic image stitching method exhibits a "ghosting" phenomenon, while the panoramic fusion image generated by the panoramic image stitching method of this embodiment does not exhibit a "ghosting" phenomenon.

[0085] The panoramic image stitching method proposed in this embodiment scales adjacent original images to obtain adjacent working images; a first homography matrix is ​​obtained by feature filtering of the adjacent working images; the adjacent original images are initially aligned according to the first homography matrix to obtain a first aligned adjacent image and its corresponding first image overlap region; a difference operation is performed on the first image overlap region to obtain a motion binary mask; the Euclidean distance from each pixel in the first image overlap region to the image center of the first aligned adjacent image is calculated; a mask forced zeroing operation is performed according to the motion binary mask and the Euclidean distance from each pixel in the first image overlap region to the image center of the first aligned adjacent image to obtain a binary fusion mask; and the stitching line is optimized along the object contour according to the binary fusion mask and the adjacent original images to obtain a panoramic fused image. In this way, the dynamic displacement of the overlapping area of ​​the image is first identified. When a significant pixel difference is detected (i.e., a moving object is identified), the mask of the interference area is forced to be suppressed and zeroed by combining the Euclidean distance of each pixel to the center of the image, and a binary fusion mask is obtained. The binary fusion mask guides the stitching line algorithm to perform global path optimization along the physical contour of the moving target, thereby avoiding visually sensitive smooth misalignment areas or object cut-off points, and solving the problem of semi-transparent afterimages caused by traditional fusion algorithms.

[0086] Example 2 Furthermore, this disclosure provides a panoramic image stitching device 700, please refer to [link to previous document]. Figure 7 The device includes: Processing module 701 is used to scale adjacent original images to obtain adjacent working images; and to obtain a first homography matrix by performing feature filtering on the adjacent working images. Alignment module 702 is used to perform preliminary alignment of the adjacent original images according to the first homography matrix to obtain the first aligned adjacent images and their corresponding first image overlap regions; Difference module 703 is used to perform difference operations on the overlapping region of the first image to obtain a motion binary mask; Calculation module 704 is used to calculate the Euclidean distance from each pixel in the overlapping region of the first image to the image center of the first aligned adjacent image; The zeroing module 705 is used to perform a mask forced zeroing operation based on the Euclidean distance from each pixel in the overlapping area of ​​the motion binary mask and the first image to the image center of the first aligned adjacent image, so as to obtain a binary fusion mask. The fusion module 706 is used to optimize the stitching line along the object contour based on the binary fusion mask and the adjacent original images to obtain a panoramic fusion image.

[0087] Optionally, the first aligned adjacent images include a first aligned left image and a first aligned right image, and the binary fusion mask includes a left binary mask and a right binary mask. The zeroing module 705 is further configured to determine a target region from the overlapping region of the first images based on a preset mask value and the motion binary mask; and to obtain, from the Euclidean distances from each pixel in the overlapping region of the first images to the image center of the first aligned adjacent images, the Euclidean distances from each pixel in the target region to the image center of the first aligned left image, and the Euclidean distances from each pixel in the target region to the image center of the first aligned right image. The Euclidean distance from the center; for each pixel in the target region, if the Euclidean distance from the pixel in the target region to the center of the first aligned left image is greater than or equal to the Euclidean distance from the pixel in the target region to the center of the first aligned right image, then the value of the pixel in the target region in the left binary mask is set to 0; if the Euclidean distance from the pixel in the target region to the center of the first aligned left image is less than the Euclidean distance from the pixel in the target region to the center of the first aligned right image, then the value of the pixel in the target region in the right binary mask is set to 0.

[0088] Optionally, the fusion module 706 is further configured to perform a scaling transformation on the first homography matrix to obtain a second homography matrix; perform preliminary alignment of the adjacent original images according to the second homography matrix to obtain a second aligned adjacent image and its corresponding second image overlap region; and perform stitching line optimization along the object contour according to the binary fusion mask, the second aligned adjacent image and its corresponding second image overlap region to obtain the panoramic fused image.

[0089] Optionally, the second aligned adjacent images include the second aligned left image and the second aligned right image. The fusion module 706 is further used to construct an energy function, the expression of which is: , In the formula, This indicates that the pixels in the overlapping area of ​​the second image will be... p Forced assignment to label The cost, A value of 0 indicates pixels in the overlapping region of the second image. p Left image after second alignment , A value of 1 indicates a pixel in the overlapping region of the second image. p Right side image after second alignment , Indicates adjacent pixels in the overlapping region of the second image In the label and tags Punishment at that time A value of 0 indicates a pixel. q Left image after second alignment , A value of 1 indicates a pixel. q Right side image after second alignment , This indicates the pixels in the left image after the second alignment. p Pixel value at that location, This indicates the pixels in the left image after the second alignment. q Pixel value at that location, Indicates the pixels in the right-hand image after the second alignment. p Pixel value at that location, Indicates the pixels in the right-hand image after the second alignment. q The pixel values ​​at the location are determined; a weighted undirected graph is constructed based on the overlapping region of the second image, and the energy function is solved by maximum flow and minimum cut based on the binary fusion mask and the weighted undirected graph to obtain the target label of each pixel in the overlapping region of the second image; the left image corresponding to the second aligned left image is generated based on the target label of each pixel in the overlapping region of the second image. Figure 2 Value mask, and the right image corresponding to the second aligned right image. Figure 2 Value mask; according to the left Figure 2 Value mask, the right Figure 2 The value mask, the second aligned left image, and the second aligned right image are stitched together to obtain the panoramic fused image.

[0090] Optionally, the fusion module 706 is further configured to, based on the left Figure 2 Value mask and the right Figure 2 A target weight map is generated using a value mask; the second aligned left image and the second aligned right image are then weighted and fused according to the weight map to obtain the panoramic fused image.

[0091] Optionally, the fusion module 706 is also used for the left Figure 2 Value mask and the right Figure 2Morphological dilation is performed on the value mask to obtain a dilated mask; subpixel-level Gaussian blur is applied to the dilated mask to obtain an initial weight map; boundary safety locking is applied to the initial weight map to obtain the target weight map.

[0092] Optionally, the fusion module 706 is further configured to preset a source point and a sink point, wherein the source point is used to indicate the second aligned left-side image, and the sink point is used to indicate the second aligned left-side image; to use each pixel in the overlapping area of ​​the second image, as well as the source point and the sink point, as nodes to determine data edges and smoothing edges; and to construct the weighted undirected graph based on the nodes, the data edges, and the smoothing edges.

[0093] Optionally, the fusion module 706 is further configured to perform HSV space color safety compensation on the second aligned left image and the second aligned right image to obtain a compensated left image and a compensated right image; and to perform weighted fusion on the compensated left image and the compensated right image according to the weight map to obtain the panoramic fused image.

[0094] Optionally, the processing module 701 is further configured to extract features from the adjacent working images based on the ORB operator to obtain candidate feature points; perform feature matching on the candidate feature points based on the Hamming distance to obtain matching feature points; and iteratively filter the matching feature points based on the random sampling consensus algorithm to obtain the first homography matrix.

[0095] The apparatus provided in this disclosure can perform the steps of the panoramic image stitching method provided in Embodiment 1. To avoid repetition, the steps will not be repeated.

[0096] The panoramic image stitching device proposed in this embodiment scales adjacent original images to obtain adjacent working images; a first homography matrix is ​​obtained by feature filtering of the adjacent working images; the adjacent original images are initially aligned according to the first homography matrix to obtain a first aligned adjacent image and its corresponding first image overlap region; a difference operation is performed on the first image overlap region to obtain a motion binary mask; the Euclidean distance from each pixel in the first image overlap region to the image center of the first aligned adjacent image is calculated; a mask forced zeroing operation is performed according to the motion binary mask and the Euclidean distance from each pixel in the first image overlap region to the image center of the first aligned adjacent image to obtain a binary fusion mask; and the stitching line is optimized along the object contour according to the binary fusion mask and the adjacent original images to obtain a panoramic fused image. In this way, the dynamic displacement of the overlapping area of ​​the image is first identified. When a significant pixel difference is detected (i.e., a moving object is identified), the mask of the interference area is forced to be suppressed and zeroed by combining the Euclidean distance of each pixel to the center of the image, and a binary fusion mask is obtained. The binary fusion mask guides the stitching line algorithm to perform global path optimization along the physical contour of the moving target, thereby avoiding visually sensitive smooth misalignment areas or object cut-off points, and solving the problem of semi-transparent afterimages caused by traditional fusion algorithms.

[0097] Example 3 Furthermore, this disclosure provides a computer device including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the panoramic image stitching method described in Embodiment 1.

[0098] The device provided in this embodiment can perform the steps of the panoramic image stitching method provided in Embodiment 1. To avoid repetition, the steps will not be repeated.

[0099] Example 4 This disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the panoramic image stitching method described in Embodiment 1.

[0100] In this embodiment, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0101] The computer-readable storage medium provided in this embodiment can implement the panoramic image stitching method provided in Embodiment 1. To avoid repetition, it will not be described again here.

[0102] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.

[0103] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0104] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A panoramic image stitching method, characterized in that, The method includes: The adjacent original images are scaled to obtain adjacent working images; the first homography matrix is ​​obtained by feature filtering of the adjacent working images. The adjacent original images are initially aligned according to the first homography matrix to obtain the first aligned adjacent images and their corresponding first image overlap regions; Perform a difference operation on the overlapping region of the first image to obtain a motion binary mask; Calculate the Euclidean distance from each pixel in the overlapping region of the first image to the image center of the first aligned adjacent image; Based on the Euclidean distance from each pixel in the overlapping region of the first image to the image center of the first aligned adjacent image, a mask forced zeroing operation is performed to obtain a binary fusion mask; Based on the binary fusion mask and the adjacent original images, the stitching line is optimized along the object contour to obtain a panoramic fusion image.

2. The panoramic image stitching method according to claim 1, characterized in that, The first aligned adjacent images include a first aligned left image and a first aligned right image. The binary fusion mask includes a left binary mask and a right binary mask. The step of performing a mask forced zeroing operation based on the Euclidean distance from each pixel in the overlapping area of ​​the motion binary mask and the first image to the image center of the first aligned adjacent image to obtain the binary fusion mask includes: Based on the preset mask value and the motion binary mask, the target region is determined from the overlapping region of the first image; From the Euclidean distances from each pixel in the overlapping region of the first image to the image center of the first aligned adjacent image, obtain the Euclidean distances from each pixel in the target region to the image center of the first aligned left image, and the Euclidean distances from each pixel in the target region to the image center of the first aligned right image. For each pixel in the target region, if the Euclidean distance from the pixel in the target region to the center of the first aligned left image is greater than or equal to the Euclidean distance from the pixel in the target region to the center of the first aligned right image, then the value of the pixel in the target region in the left binary mask is set to 0. If the Euclidean distance from a pixel in the target region to the center of the first aligned left image is less than the Euclidean distance from a pixel in the target region to the center of the first aligned right image, then the value of the pixel in the target region in the right binary mask is set to 0.

3. The panoramic image stitching method according to claim 1, characterized in that, The step of optimizing the stitching line along the object contour based on the binary fusion mask and the adjacent original images to obtain a panoramic fusion image includes: The first homography matrix is ​​scaled to obtain the second homography matrix; The adjacent original images are initially aligned according to the second homography matrix to obtain the second aligned adjacent images and their corresponding second image overlap regions; The panoramic fused image is obtained by optimizing the stitching line along the object contour based on the binary fusion mask, the second aligned adjacent image and the corresponding second image overlapping area.

4. The panoramic image stitching method according to claim 3, characterized in that, The second aligned adjacent images include the second aligned left image and the second aligned right image. The step of optimizing the stitching line along the object contour based on the binary fusion mask, the second aligned adjacent images, and their corresponding overlapping areas to obtain the panoramic fused image includes: Construct an energy function, the expression of which is: , In the formula, This indicates that the pixels in the overlapping area of ​​the second image will be... p Forced assignment to label The cost, A value of 0 indicates pixels in the overlapping region of the second image. p Left image after second alignment , A value of 1 indicates a pixel in the overlapping region of the second image. p Right side image after second alignment , Indicates adjacent pixels in the overlapping region of the second image In the label and tags Punishment at that time A value of 0 indicates a pixel. q Left image after second alignment , A value of 1 indicates a pixel. q Right side image after second alignment , This indicates the pixels in the left image after the second alignment. p Pixel value at that location, This indicates the pixels in the left image after the second alignment. q Pixel value at that location, Indicates the pixels in the right-hand image after the second alignment. p Pixel value at that location, Indicates the pixels in the right-hand image after the second alignment. q Pixel value at; A weighted undirected graph is constructed based on the overlapping region of the second image, and the energy function is solved by maximum flow and minimum cut based on the binary fusion mask and the weighted undirected graph to obtain the target label of each pixel in the overlapping region of the second image. Generate a left-image binary mask corresponding to the second aligned left-image and a right-image binary mask corresponding to the second aligned right-image based on the target labels of each pixel in the overlapping region of the second image; The panoramic fused image is obtained by stitching together the left binary mask, the right binary mask, the second aligned left image, and the second aligned right image.

5. The panoramic image stitching method according to claim 4, characterized in that, The step of stitching images together based on the left binary mask, the right binary mask, the second aligned left image, and the second aligned right image to obtain the panoramic fused image includes: A target weight map is generated based on the binary mask in the left image and the binary mask in the right image; The second aligned left image and the second aligned right image are weighted and fused according to the weight map to obtain the panoramic fused image.

6. The panoramic image stitching method according to claim 5, characterized in that, The step of generating a target weight map based on the binary mask in the left image and the binary mask in the right image includes: Morphological dilation is performed on the binary mask in the left image and the binary mask in the right image to obtain the dilated mask; The expanded mask is subjected to subpixel-level Gaussian blurring to obtain an initial weight map; Boundary safety locking is performed on the initial weight map to obtain the target weight map.

7. The panoramic image stitching method according to claim 4, characterized in that, The step of establishing a weighted undirected graph based on the overlapping region of the second image includes: A source point and a sink point are preset, wherein the source point is used to indicate the second aligned left-side image, and the sink point is used to indicate the second aligned left-side image; Each pixel in the overlapping region of the second image, as well as the source point and the sink point, are used as nodes to determine the data edge and the smoothing edge; The weighted undirected graph is constructed based on the nodes, the data edges, and the smoothed edges.

8. The panoramic image stitching method according to claim 5, characterized in that, The step of weighted fusing the second aligned left image and the second aligned right image according to the weight map to obtain the panoramic fused image includes: Perform HSV space color safety compensation on the second aligned left image and the second aligned right image to obtain the compensated left image and the compensated right image; The compensated left image and the compensated right image are weighted and fused according to the weight map to obtain the panoramic fused image.

9. The panoramic image stitching method according to claim 1, characterized in that, The step of obtaining the first homography matrix by performing feature filtering on the adjacent working images includes: Based on the ORB operator, feature extraction is performed on the adjacent working images to obtain candidate feature points; The candidate feature points are matched based on the Hamming distance to obtain the matched feature points. The first homography matrix is ​​obtained by iteratively filtering the matching feature points based on the random sampling consensus algorithm.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the panoramic image stitching method as described in any one of claims 1 to 9.

Images