Panoramic image splicing method, system and device

Abstract

The invention discloses a panoramic image splicing method, system and device, and relates to the technical field of image processing, and the method comprises the steps: collecting images around a vehicle at different visual angles; using the MASK masks to generate overlapping region masks of the different view angle images, and performing local feature detection on the overlapping region masks to generate a transformation matrix; aligning the images of different visual angles according to the transformation matrix; a Canny edge detection operator is fused in splicing seam energy detection, the energy weight of the image edge area of the aligned image is enhanced, and a splicing seam is guided to actively avoid an object with remarkable structural features; b spline interpolation is introduced to optimize an initial splicing seam path; constructing a Laplacian pyramid along Gaussian blur of a splicing seam, and sequentially carrying out gradual fusion to generate a seamless panoramic image; according to the method, the basic function of eliminating the visual blind area is realized, a clear and visual panoramic view can be provided for a driver, and the driving safety is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of panoramic image stitching, and particularly relates to a panoramic image stitching method, system and device. BACKGROUND

[0002] Image stitching algorithm plays a crucial core role in the automobile panoramic surround view system. In the evolution of image stitching technology, some scholars have proposed a theoretical framework for image stitching, which accurately estimates and adjusts eight key parameters in the mathematical model through the design of an iterative optimization-based algorithm. This method has a wide range of applications and high computational efficiency, which is reflected in its fast convergence characteristics. Subsequently, some scholars further proposed an adaptive stitching architecture based on motion segmentation, which effectively solves the distortion problem when processing moving objects. However, this method has high computational complexity and is sensitive to scene depth changes.

[0003] The SURF algorithm proposed by some scholars uses integral image and Hessian matrix to quickly detect feature points, reduces the amount of calculation by simplifying and approximating calculation, and improves the calculation efficiency compared with SIFT. However, this approximate calculation strategy speeds up the processing speed at the expense of the completeness of the feature representation, and is less stable than SIFT when dealing with changes in light and angle. The approximate calculation of the Hessian matrix based on the box filter is not sensitive to fine texture changes, and it is difficult to ensure the high-precision matching required by panoramic stitching in the complex and changeable environment around the vehicle, and the calculation amount still exceeds the real-time processing capability of the vehicle-mounted processor. Some scholars have greatly reduced the calculation amount and memory occupation by combining FAST corner detection and BRIEF descriptor, but the FAST detector is sensitive to noise and only focuses on local pixel changes while ignoring global structural information, resulting in low matching accuracy when dealing with geometric deformation in complex environments, especially in scenes with a large number of repetitive textures and perspective deformation around the vehicle, which is prone to false matching and does not meet the high-precision requirements of the panoramic surround view system. In the field of image fusion, the weighted average method is widely used due to its simple linear calculation characteristics, but this method only considers the linear combination of pixel values and ignores the image structure information, which cannot effectively handle the nonlinear light difference and geometric distortion between images, resulting in obvious seams and "ghosting" in the fusion area. Some scholars have proposed a gradient domain fusion technology based on Poisson equation to maintain image structure continuity by solving partial differential equations in the gradient domain, which can effectively eliminate the stitching seams, but the calculation complexity of solving large linear equations increases quadratically or cubically, and the calculation amount increases dramatically with the increase of image resolution, making it difficult to realize real-time operation on the vehicle-mounted processor. Some scholars have proposed an image stitching technology that maximizes gradient consistency to find the optimal fusion result, which requires iterative optimization to solve large-scale sparse linear equations. Although the stitching effect is natural, the multiple iterations and matrix operations in the optimization process result in long calculation time, which does not meet the low delay requirement in the vehicle-mounted environment. Some scholars have also proposed a multi-band fusion method that creates a transition band with variable width in the overlapping area to smooth the boundary, but this method needs to pre-calculate the global energy function and find the optimal path, which is prone to blur or ghosting at the edge of the moving object in dynamic scenes due to the inconsistency of the content in the overlapping area caused by the sampling time difference, reducing the reliability of the surround view system.

[0004] In summary, the traditional stitching method relies on a global transformation model, and when dealing with scenes with complex geometric structures, the geometric error of the stitching result is large, and the stitching gap is obvious, which makes it difficult to meet the high requirements of the assisted driving system on image quality and real-time performance. SUMMARY

[0005] To address the shortcomings of existing technologies in processing scenes with complex geometric structures, which suffer from large geometric errors and obvious stitching gaps, this invention proposes a panoramic image stitching method, system, and apparatus. It uses a mask to determine the overlapping area, employs a multi-resolution fusion algorithm based on improved stitching gaps for image fusion, and smoothly transitions the registered images in the overlapping area, eliminating seams and improving brightness differences between images, thereby generating a seamless panoramic image and solving the problems of existing technologies.

[0006] A panoramic image stitching method includes the following steps: Collect images of the vehicle's surroundings from different perspectives; MASK masking is applied to images around the vehicle from different viewpoints to generate overlapping region masks; local feature detection is performed on the overlapping region masks to generate an image transformation matrix; based on the image transformation matrix, the images around the vehicle from different viewpoints are aligned and registered. In the overlapping areas of images from different perspectives after alignment and registration, the comprehensive energy function is calculated by the pixel color difference and structural gradient difference in the overlapping area. Based on the comprehensive energy function, the image is guided to avoid objects with structural features at the seam. The dynamic programming algorithm is used to search for the initial seam path in the overlapping area. The initial seam path is optimized by B-spline interpolation to obtain a smooth seam. Based on smooth stitching seams, a multi-resolution pyramid fusion strategy is used to fuse the registered images, outputting a seamless panoramic image.

[0007] Furthermore, the step of performing local feature detection on the overlapping region mask to generate an image transformation matrix specifically includes the following steps: The SIFT algorithm is used to find key points that can uniquely identify the location within the overlapping region of adjacent images from different viewpoints; a descriptor is generated for each detected key point. Principal component analysis (PCA) was used to reduce the dimensionality of the 128-dimensional SIFT descriptor eigenvectors, compressing the 128-dimensional eigenvectors into 32-dimensional ones. For a descriptor in the source image, calculate the distance between the descriptor and all descriptors in the target image, and find the descriptor with the closest distance as the matching point; The Random Sampling Consensus (RANSAC) algorithm is introduced to remove erroneous matching points during the matching process and output the optimal image transformation matrix.

[0008] Furthermore, the calculation process of the comprehensive energy function is specifically expressed as follows: ; in, and Overlapping regions Pixel color difference and structural gradient difference; color difference , and These represent the pixel grayscale values ​​of the two images respectively; structural gradient difference. , and They represent direction and Sobel gradient operator for the direction.

[0009] Furthermore, the step of using a dynamic programming algorithm to search for the initial seam path within the overlapping area specifically includes using a dynamic programming algorithm to search from the top of the overlapping area, calculating the cumulative cost row by row, and selecting the path with the minimum cost as the initial seam path.

[0010] Furthermore, the process of fusing the registered images using a multi-resolution pyramid fusion strategy based on smooth stitching seams to output a seamless panoramic image specifically includes the following steps: The registered image is subjected to layer-by-layer Gaussian blurring and downsampling to obtain a Gaussian pyramid; wherein the construction process of the Gaussian pyramid is represented as follows: , This represents the i-th level of the pyramid. This indicates a downsampling operation. Indicates the Gaussian blur operation; Based on the Gaussian pyramid, a Laplacian pyramid is constructed from the registered image; the construction process of the Laplacian pyramid is represented as follows: ,in, This is the current layer of the Pyramid of Laplace. This is the current layer of the Gaussian pyramid. It is the result of upsampling the previous layer image; A weighted map corresponding to the resolution of each layer of the Laplace pyramid is generated based on the smooth seams. The Laplacian pyramid of each pair of registered images is weighted and fused using a weighted graph; the fusion process is represented as follows: ; It is the current layer of the weighted and merged Laplace pyramid. and These are the current Laplacian pyramid levels for the two images. This is the weight graph of the current layer; By upsampling and reconstructing the weighted and fused Laplacian pyramid layer by layer, a seamless panoramic image is obtained.

[0011] Furthermore, the initial splice path is optimized by B-spline interpolation to obtain a smooth splice; specifically, the sawtooth abrupt changes and acute inflection points of the initial splice path are eliminated by utilizing the continuity of the second derivative of the B-spline curve to generate a smooth splice.

[0012] Furthermore, images of the vehicle's surroundings from different perspectives were captured using four fisheye cameras.

[0013] The present invention also includes a panoramic image stitching system, comprising: The acquisition module is used to acquire images of the vehicle's surroundings from different perspectives; The registration module is used to perform a masking operation on images around the vehicle from different viewpoints to generate an overlapping region mask for the images around the vehicle from different viewpoints; perform local feature detection on the overlapping region mask to generate an image transformation matrix; and align and register the images around the vehicle from different viewpoints based on the image transformation matrix. The stitching optimization module is used to calculate the comprehensive energy function by using the pixel color difference and structural gradient difference in the overlapping area of ​​the image under different viewpoints after alignment and registration. Based on the comprehensive energy function, the image is guided to avoid objects with structural features at the stitching seam. The module also uses a dynamic programming algorithm to search for the initial stitching seam path in the overlapping area and performs B-spline interpolation optimization on the initial stitching seam path to obtain a smooth stitching seam. The fusion module is used to fuse registered images based on smooth stitching seams and employs a multi-resolution pyramid fusion strategy to output seamless panoramic images.

[0014] The present invention also includes a panoramic image stitching computer device, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the panoramic image stitching method.

[0015] The present invention also includes a readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, are used to perform the steps of the panoramic image stitching method.

[0016] This invention provides a panoramic image stitching method, which has the following beneficial effects: This invention uses a mask for local feature detection, strictly limiting feature matching to the theoretically overlapping area. This significantly reduces interference from erroneous feature points in non-overlapping areas, laying a solid foundation for generating a high-precision transformation matrix. It ensures the accuracy of image alignment from the source, avoiding ghosting and misalignment caused by registration errors. By calculating a comprehensive energy function, it guides the stitching seam to avoid significant structural features, thus maintaining the integrity and visual continuity of key objects in the scene, resulting in a more natural and reliable output. A dynamic programming algorithm is employed to select the minimum-cost path and search for the initial stitching seam path. By combining B-spline interpolation, the initial stitching seam path is optimized to eliminate the jagged path generated by traditional dynamic programming and avoid geometric deformation artifacts after stitching. The optimized stitching seam is used for image fusion, which smoothly transitions the registered images in the overlapping area, eliminating seams and improving brightness differences between images, thereby generating a seamless panoramic image. This method achieves smooth transition and no ghosting in terms of stitching effect, eliminates geometric errors in the stitching result, and shows good adaptability in real road environments. It can provide drivers with a clear and intuitive panoramic view and effectively improve driving safety. Attached Figure Description

[0017] Figure 1 This is a flowchart of the panoramic image stitching method in an embodiment of the present invention; Figure 2 This is a comparison chart of experimental results for the first group of combined improvement points in this embodiment of the invention; Figure 2 (a) in the figure is the original SIFT experimental result graph; Figure 2 (b) in the figure shows the experimental results of SIFT+MASK; Figure 2 (c) in the figure shows the experimental results of PCA-SIFT+MASK; Figure 2 (d) in the figure represents the experimental results of the complete improved algorithm; Figure 3 This is a comparison chart of experimental results for the second set of combined improvement points in this embodiment of the invention; Figure 3 (a) in the figure is the original SIFT experimental result graph; Figure 3 (b) in the figure shows the experimental results of SIFT+MASK; Figure 3 (c) in the figure shows the experimental results of PCA-SIFT+MASK; Figure 3 (d) in the figure represents the experimental results of the complete improved algorithm; Figure 4 This is a comparison diagram of the first set of fusion results in an embodiment of the present invention; Figure 4 (a) in the figure shows the experimental results of the direct fusion algorithm; Figure 4 (b) in the figure shows the experimental results of the weighted average algorithm; Figure 4 (c) in the figure shows the experimental results of the multi-resolution fusion algorithm;Figure 4 (d) in the figure represents the experimental results of the algorithm of this invention; Figure 5 This is a comparison diagram of the second set of fusion results in an embodiment of the present invention; Figure 5 (a) in the figure shows the experimental results of the direct fusion algorithm; Figure 5 (b) in the figure shows the experimental results of the weighted average algorithm; Figure 5 (c) in the figure shows the experimental results of the multi-resolution fusion algorithm; Figure 5 (d) in the figure represents the experimental results of the algorithm of this invention; Figure 6 These are images collected from a real vehicle experiment in an embodiment of the present invention; Figure 6 (a) in the diagram is a schematic of the calibration cloth being laid in front of the vehicle body; Figure 6 (b) is a schematic diagram of the calibration cloth being laid at the rear of the vehicle body; Figure 6 (c) in the diagram is a schematic of the calibration cloth being laid on the left side of the vehicle body; Figure 6 (d) in the diagram is a schematic of the calibration cloth being laid on the right side of the vehicle body; Figure 7 This is a panoramic view of a car in an embodiment of the present invention; Figure 8 This is a diagram showing the original image acquisition result of a dynamic road scene in an embodiment of the present invention; Figure 8 (a) in the diagram is a front view of the vehicle body; Figure 8 (b) in the diagram is a rear view of the vehicle. Figure 8 (c) in the diagram is a schematic view of the left side of the vehicle body; Figure 8 (d) in the diagram represents the right side of the vehicle body; Figure 9 This is a panoramic view of a dynamic road scene in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0019] Traditional SIFT algorithms, with their advantages of scale, rotation, and illumination invariance, exhibit high robustness in feature detection and matching tasks. However, they still have limitations in many aspects: for example, detection in the global scope is easily affected by noise and non-critical areas, leading to low detection efficiency and mismatches; although 128-dimensional feature vectors have strong discriminative power, the high dimension increases the storage and computational burden; and there are too many mismatches in coarse matching, affecting matching accuracy.

[0020] This invention proposes a panoramic image stitching method, such as... Figure 1 As shown, the method specifically includes the following steps: S1, Data Acquisition.

[0021] During the data acquisition phase, images of the vehicle's surroundings were captured from different perspectives using four fisheye cameras, forming a complete field-of-view dataset.

[0022] S2. Image preprocessing: In the image preprocessing stage, the fisheye camera is first analyzed to ensure that the camera distortion can be correctly identified and calibrated during the panoramic image stitching process. Then, the camera is calibrated to obtain the camera's intrinsic and extrinsic parameters and distortion coefficients, and distortion correction is performed to eliminate image distortion caused by the fisheye lens. Finally, the image is transformed into a bird's-eye view suitable for stitching through perspective transformation.

[0023] S3. Traditional SIFT algorithms, with their advantages of scale, rotation, and illumination invariance, exhibit high robustness in feature detection and matching tasks. However, they still have limitations in many aspects: for example, when performing detection globally, they are easily affected by noise and non-critical areas, leading to low detection efficiency and mismatches; although 128-dimensional feature vectors have strong discriminative power, the high dimension increases the storage and computational burden; and there are too many mismatches in coarse matching, affecting matching accuracy.

[0024] This invention uses an improved SIFT-based feature detection and matching algorithm for image registration. It aligns images from different viewpoints through geometric transformations to ensure accurate matching of their overlapping areas. Specifically, it includes the following steps: (1) During the input preparation stage, the system receives four bird’s-eye view images that have been distorted and transformed from top to bottom. These images have been unified to the bird’s-eye view coordinate system, providing a geometrically consistent basis for registration.

[0025] (2) In the feature detection stage, global detection is changed to local detection, and the MASK mask is used to determine the overlapping area. The overlapping area of ​​adjacent cameras is determined according to the vehicle geometry. Taking left-front registration as an example, a mask matrix is ​​created, and the overlapping area on the left side of the front view image and the upper side of the left view image is marked as 255, and the remaining area is marked as 0. Then, SIFT feature detection is performed on the limited area.

[0026] (3) In the feature descriptor generation stage, the detected feature points are subjected to dimensionality reduction of the feature vectors through principal component analysis (PCA), compressing the 128-dimensional feature vectors into 32-dimensional vectors. This process not only retains a large amount of original information, but also reduces memory usage and improves processing speed.

[0027] (4) In the feature matching stage, the Random Sampling Consensus (RANSAC) algorithm is introduced to eliminate erroneous matches. First, a brute-force matcher is used for initial matching, configuring the Euclidean distance metric and ratio test threshold. These matches may contain many erroneous matching points. Then, the RANSAC algorithm randomly selects 4 pairs of matching points each time to calculate the homography matrix. Inliers are filtered by reprojection error, generally with an error threshold of 3 pixels. Only points with a reprojection error less than this threshold are considered inliers. The above process is repeated, and the optimal transformation model is found after a maximum of 2000 iterations. The model with the highest proportion of inliers is selected as the final result.

[0028] Using the optimal homography matrix obtained from RANSAC, all pixels in the source image are mapped to the target image coordinate system according to the rules defined by the matrix (including rotation, scaling, translation, etc.). OpenCV functions are then used to reproject the source image pixels to the new positions, ultimately achieving precise alignment of images from different perspectives on the same plane, realizing geometric registration, and laying the foundation for subsequent image fusion.

[0029] S4. Calculate the comprehensive energy function for each overlapping region, including color difference, gradient difference, and edge weights. The energy function expression is defined by fusing color and geometric feature constraints as follows: ,in, and Overlapping regions The pixel color difference and structural gradient difference.

[0030] Color difference can be obtained by subtracting the pixel values ​​of the overlapping areas of the two images: ,in, and These represent the pixel grayscale values ​​of the two images, respectively.

[0031] Structural gradient difference The expression: in, and They represent direction and Sobel gradient operator for the direction.

[0032] The smaller the energy function, the smaller the difference between pixels, and the more suitable it is as a point on the seam of the stitching.

[0033] The Canny edge detection operator fusion is a further improvement on this approach. By detecting significant edges in the image and assigning them higher energy weights, it guides the seams to actively avoid objects with significant structural features, solving the problem of the original algorithm easily cutting out important targets and causing content loss. A dynamic programming algorithm is used to search for the optimal path: starting from the top of the overlapping region, the cumulative cost is calculated row by row, the path with the minimum cost is selected, and finally, the initial seam path is obtained through backtracking.

[0034] S5. B-spline interpolation is introduced to optimize the initial splice path. The path abrupt change is eliminated by the second derivative, eliminating the jagged path generated by traditional dynamic programming and avoiding geometric deformation artifacts after splicing.

[0035] S6. Combining a multi-resolution fusion algorithm, the image is decomposed into multiple different frequency levels for independent detail processing, preserving more texture details and enhancing image quality. First, a 5-layer Gaussian pyramid and a Laplacian pyramid are constructed.

[0036] Gauss Pyramid using the formula Build, in This represents the i-th level of the pyramid. This indicates a downsampling operation. This indicates a Gaussian blur operation. The input image is subjected to layer-by-layer Gaussian blur and downsampling to obtain a Gaussian pyramid, which mainly contains low-frequency information about the image, such as overall brightness distribution and color variations.

[0037] The Pyramid of Laplace is derived from the formula It is concluded that, This is the current layer of the Pyramid of Laplace. This is the current layer of the Gaussian pyramid. It is the result of upsampling the previous layer image. In fact, it is the error between the current layer of the Gaussian pyramid and the adjacent layers after upsampling, mainly preserving high-frequency detail information, such as edges and textures.

[0038] Smoothing weights are created based on the optimized seam locations, and each layer of the pyramid is weighted and fused. For each layer of the Laplacian pyramid (from bottom to top), the two images are weighted and fused using a weighted graph, as shown in the formula below: .in, It is the current layer of the weighted and merged Laplace pyramid. and These are the current Laplacian pyramid layers of the two images. It is the weight graph of the current layer.

[0039] Finally, the weighted and fused Laplacian pyramid was upsampled and reconstructed layer by layer. Starting from the top layer, the image details were gradually added back, and a seamless panoramic image was finally reconstructed.

[0040] Traditional dynamic programming-based optimal stitching seam algorithms primarily rely on pixel differences to determine the path, but they do not fully consider the integrity and continuity of object edges in the image. Stitching seams that directly pass through objects can lead to a visually disjointed appearance. Furthermore, the path chosen by backtracking to minimize cost may be jagged or sharp-angled, introducing geometric deformation artifacts during the fusion process. While obtaining the optimal stitching seam can improve the transition and seam elimination in the stitching area, it still falls short in handling details across different regions. Therefore, this invention combines a multi-resolution fusion algorithm to further improve the image fusion precision, smoothly transitioning the registered images in overlapping areas, eliminating seams, and improving brightness differences between images, thereby generating a seamless panoramic image.

[0041] Experimental Analysis: (1) Validation of the improved SIFT-based feature detection and matching algorithm.

[0042] This invention selects two representative datasets for testing, as shown in Table 1. The first set is the boat sequence from the ACR dataset, characterized by rich texture and edge information and high scene complexity. The second set is a self-built dataset, characterized by large homogeneous regions and a small number of significant structures, with lower feature point density, more closely resembling the actual driving environment. By progressively adding improvement points, the collaborative mechanism of each improvement point is quantitatively analyzed in the feature matching process. The experimental results of combining improvement points are shown below. Figure 2 and Figure 3 As shown.

[0043] Table 1 Comparison of Experimental Results for Combined Improvement Points Experimental results show that the improved algorithm proposed in this invention has achieved significant progress in all key indicators. Specifically, SIFT+MASK, by introducing masking technology to limit the feature detection area, reduces the number of feature points from 3433 to 2792 in the first group, a reduction of 18.7%; and from 1561 to 1404 in the second group, a reduction of 10%. This indicates that masking technology can effectively reduce redundant computation and improve matching accuracy.

[0044] Based on this, PCA-SIFT+MASK optimizes performance through 32-dimensional dimensionality reduction: the number of features in the first and second groups is reduced by 40.4% and 47.1% respectively, and the resource consumption is reduced to 0.68MB and 0.63MB respectively. At the same time, the proportion of interior points is increased to 82% and 80%, proving that dimensionality reduction effectively preserves key information.

[0045] Ultimately, the fully improved algorithm, which integrates all the above optimization measures and incorporates RANSAC, achieves the best performance across all metrics: the interior point ratios of the first and second groups reach 85% and 83% respectively, representing improvements of 13.3% and 15.3% compared to the original SIFT; the number of features is reduced by approximately 36% and 26%; the computation time is reduced to 60 milliseconds and 15 milliseconds respectively, representing improvements of 57.1% and 60.5% compared to the original SIFT; and the source occupancy is only 0.48MB and 0.45MB, representing reductions of 57.9% and 62.2% compared to the original SIFT.

[0046] These results demonstrate that the synergistic effect between the improved modules is key to the performance enhancement: the mask focuses on overlapping regions, providing high-information-density input for PCA; while RANSAC compensates for the descriptor discrimination loss that may result from dimensionality reduction through geometric consistency constraints. The complete improved algorithm significantly outperforms the original SIFT in terms of accuracy, robustness, and resource efficiency, effectively solving the problems of redundant computation, high storage overhead, and mismatch sensitivity of traditional SIFT, and meeting the low resource consumption and high reliability requirements of automotive panoramic surround view systems for real-time image matching.

[0047] (2) Verification based on the improved multi-resolution fusion algorithm for splicing seams.

[0048] Based on successful image registration, this invention conducts in-depth research and optimization of the fusion process. The experiment uses a publicly available dataset from the Internet, which gathers images from various scenes, covering different lighting conditions, viewpoints, and resolutions, and can simulate various complex scenes in the real world.

[0049] This experiment selected two sets of images as experimental data to verify the method proposed in this invention. The first set is at a resolution of 320. The 240-pixel scene, containing complex structures (buildings), uniform regions (parking lots), and key targets (vehicles), is suitable for testing the algorithm's ability to handle structural edges and preserve details. The second set uses a 480-pixel resolution. A 320-pixel scene, including pedestrians, trees, and road markings, is suitable for evaluating the algorithm's performance when handling dynamic objects and irregular textures. Experimental results are as follows: Figure 4 and Figure 5 As shown.

[0050] From a subjective visual perspective, direct fusion algorithms, by simply stitching images together, result in obvious boundary problems, with noticeable seams and brightness differences. While weighted average fusion alleviates hard boundaries, the global averaging operation causes loss of high-frequency details and a decrease in image sharpness. The multi-resolution pyramid algorithm preserves details and smooths transitions well, but multi-scale information misalignment produces ghosting issues, resulting in unnatural repetitions and blurred images. This invention's improved optimal stitching seam algorithm combines the advantages of multi-resolution fusion, optimizes the shortcomings of the first three algorithms, effectively reduces stitching boundaries, preserves sharpness, and solves the ghosting problem, making the final stitched image smoother and more natural.

[0051] In addition, this invention selects four objective indicators—structural similarity index (SSIM), information entropy (IE), root mean square error (RMSE), and peak signal-to-noise ratio (PSNR)—to evaluate the quality of the fused image, and the results are shown in Table 2.

[0052] Table 2 Comparison of Fusion Algorithm Results Based on comparative analysis of experimental data, the improved algorithm of this invention demonstrates significant advantages in key performance indicators: Regarding boundary issues, the SSIM value reached 0.90 in the first set of experiments, an improvement of over 18.4% compared to the direct fusion algorithm; in the second set of experiments, the SSIM value reached 0.93, an improvement of 19.2% compared to the direct fusion algorithm, indicating that the improved algorithm effectively enhances the structural continuity of the stitched region, making the image transition more natural. In terms of image fusion quality, the IE reached 7.8 in both sets of experiments, an improvement of 6.8% and 5.4% compared to the weighted average algorithm, indicating that the fused region contains richer information. Regarding image detail and ghosting removal, the MSE value decreased to 18.1 in the first set, a reduction of 22.0% compared to the multi-resolution fusion algorithm; in the second set of experiments, the MSE value decreased to 17.8, a reduction of 26.1% compared to the multi-resolution fusion algorithm, while the PSNR indicators reached 37.1dB and 39.5dB respectively, far exceeding other algorithms, indicating superior overall fusion quality. This set of key data fully verifies the comprehensive advantages of the algorithm of this invention in maintaining structural continuity, optimizing information fusion, and reducing dynamic artifacts, and verifies the effectiveness of the algorithm, especially in scenes with complex textures and dynamic objects.

[0053] (3) Static scene real vehicle verification.

[0054] Based on laboratory verification, this invention further validated the algorithm's performance in a real-world environment through static vehicle-based experiments. A school test vehicle was selected for the experiment, with calibration cloths laid out in all four directions (front, rear, left, and right). The front and rear calibration cloths were 4.4m long. A 1.2m rectangular layout is laid at the front and rear of the vehicle body, with 1.6m markers selected for the left and right sides. The 1.2m cameras are positioned on both sides of the vehicle. The fisheye cameras are mounted below the front and rear license plates and below the side mirrors to minimize obstruction of the shooting angle by the vehicle's structure. Figure 6 As shown.

[0055] A feature-point-based image registration method is adopted, using an improved SIFT algorithm for feature point detection and matching. In the feature detection stage, a mask is used to limit the detection range, performing detection only in overlapping image regions, thus improving detection efficiency and reducing false matches. In the feature matching stage, the RANSAC algorithm is introduced to eliminate incorrect matches, improving matching accuracy and robustness. Then, a multi-resolution fusion algorithm based on an improved stitching seam is used to fuse the images. The fusion result is shown below. Figure 7 As shown, static scenario testing verified the feasibility of the invention in a real-world vehicle application environment, laying the foundation for further dynamic scenario testing.

[0056] (4) Dynamic scenario real vehicle verification.

[0057] Real-world road testing is the final stage of algorithm validation, designed to evaluate the algorithm's performance in complex and dynamically changing real-world driving environments. This section details the testing process and result analysis of the proposed panoramic surround-view system in real-world scenarios, verifying the system's reliability and stability in practical applications. The original image acquisition results for dynamic road scenes are shown below. Figure 8 As shown, the panoramic surround view effect in the road scene is as follows: Figure 9 As shown.

[0058] Experimental results show that the overall algorithm of the automotive panoramic surround view system studied in this invention is feasible, and it achieves smooth transitions and no ghosting in terms of stitching effect, largely realizing the basic function of eliminating blind spots. The system exhibits good adaptability in real road environments, providing drivers with a clear and intuitive panoramic view, effectively improving driving safety.

[0059] Besides visual effects, this invention also focuses on the real-time performance and resource consumption of the algorithm, which are key indicators for automotive applications. For automotive applications, real-time performance and resource consumption are important metrics for evaluating the practical value of an algorithm. The time performance and resource consumption of the algorithm in real-world road tests were analyzed. On the test platform (RTX 3060 GPU), taking a dynamic road scene test as an example, the time consumption of each stage of the processing flow is shown in Table 3.

[0060] Table 3 Time Consumption Table for Dynamic Road Scenes As shown in Table 3, the improved algorithm achieved a processing frame rate of approximately 4.2 FPS (238 ms / frame) on the test platform, a significant improvement compared to the traditional algorithm. In contrast, the traditional panoramic surround-view algorithm typically requires 500-600 ms of processing time, demonstrating an overall performance improvement of approximately 55%. This processing speed essentially meets the real-time requirements on the current test platform.

[0061] Table 4 shows the resource consumption of each stage of the processing flow in the dynamic road scenario.

[0062] Table 4 Resource Consumption Analysis of Dynamic Road Scenarios From an overall resource efficiency perspective, the current implementation achieves a peak memory usage of approximately 1.2GB, GPU utilization of approximately 60%, and an average CPU utilization of 32% during processing. Compared to traditional methods that typically require 2-3GB of memory and 80-90% GPU utilization, this invention demonstrates a significant improvement in resource efficiency. This optimization primarily stems from the feature descriptor dimensionality reduction and masking constraints proposed in Chapter 3, as well as the multi-resolution processing strategy presented in Chapter 4. These improvements not only reduce computational load but also lower memory consumption.

[0063] Based on the same inventive concept, this invention also proposes a panoramic image stitching system, comprising: The acquisition module is used to acquire images of the vehicle's surroundings from different perspectives.

[0064] The registration module is used to perform a masking operation on images around the vehicle from different viewpoints to generate overlapping region masks for the images around the vehicle from different viewpoints; perform local feature detection on the overlapping region masks to generate an image transformation matrix; and align and register the images around the vehicle from different viewpoints based on the image transformation matrix.

[0065] The stitching optimization module is used to calculate the comprehensive energy function by using the pixel color difference and structural gradient difference in the overlapping area of ​​the image under different viewpoints after alignment and registration. Based on the comprehensive energy function, the image is guided to avoid objects with structural features at the stitching seam. The module also uses a dynamic programming algorithm to search for the initial stitching seam path in the overlapping area and performs B-spline interpolation optimization on the initial stitching seam path to obtain a smooth stitching seam.

[0066] The fusion module is used to fuse registered images based on smooth stitching seams and employs a multi-resolution pyramid fusion strategy to output seamless panoramic images.

[0067] The present invention also includes a panoramic image stitching computer device, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the panoramic image stitching method.

[0068] The present invention also includes a readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, are used to perform the steps of the panoramic image stitching method.

[0069] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A panoramic image stitching method, characterized in that, Includes the following steps: Collect images of the vehicle's surroundings from different perspectives; MASK masking is applied to images around the vehicle from different viewpoints to generate overlapping region masks for the images around the vehicle from different viewpoints; local feature detection is performed on the overlapping region masks to generate an image transformation matrix. Based on the image transformation matrix, the images around the vehicle from different perspectives are aligned and registered; In the overlapping areas of images from different perspectives after alignment and registration, the comprehensive energy function is calculated by the pixel color difference and structural gradient difference in the overlapping area. Based on the comprehensive energy function, the image is guided to avoid objects with structural features at the seam. The dynamic programming algorithm is used to search for the initial seam path in the overlapping area. The initial seam path is optimized by B-spline interpolation to obtain a smooth seam. Based on smooth stitching seams, a multi-resolution pyramid fusion strategy is used to fuse the registered images, outputting a seamless panoramic image.

2. The panoramic image stitching method according to claim 1, characterized in that, The process of performing local feature detection on the overlapping region mask to generate an image transformation matrix specifically includes the following steps: The SIFT algorithm is used to find key points that can uniquely identify the location within the overlapping region of adjacent images from different viewpoints; a descriptor is generated for each detected key point. Principal component analysis (PCA) was used to reduce the dimensionality of the 128-dimensional SIFT descriptor eigenvectors, compressing the 128-dimensional eigenvectors into 32-dimensional ones. For a descriptor in the source image, calculate the distance between the descriptor and all descriptors in the target image, and find the descriptor with the closest distance as the matching point; The Random Sampling Consensus (RANSAC) algorithm is introduced to remove erroneous matching points during the matching process and output the optimal image transformation matrix.

3. The panoramic image stitching method according to claim 1, characterized in that, The calculation process of the comprehensive energy function is specifically expressed as follows: ； in, and Overlapping regions Pixel color difference and structural gradient difference; color difference , and These represent the pixel grayscale values ​​of the two images respectively; structural gradient difference. , and They represent direction and Sobel gradient operator for the direction.

4. The panoramic image stitching method according to claim 1, characterized in that, The method of using dynamic programming to search for the initial seam path in the overlapping area specifically includes using dynamic programming to search from the top of the overlapping area, calculating the cumulative cost row by row, and selecting the path with the minimum cost as the initial seam path.

5. A panoramic image stitching method according to claim 1, characterized in that, The initial splice path is optimized by B-spline interpolation to obtain a smooth splice. Specifically, the zigzag abrupt changes and acute inflection points of the initial splice path are eliminated by utilizing the continuity of the second derivative of the B-spline curve to generate a smooth splice.

6. The panoramic image stitching method according to claim 1, characterized in that, The method of fusing registered images based on smooth stitching seams using a multi-resolution pyramid fusion strategy to output a seamless panoramic image includes the following steps: The registered image is subjected to layer-by-layer Gaussian blurring and downsampling to obtain a Gaussian pyramid; wherein the construction process of the Gaussian pyramid is represented as follows: , Let i represent the i-th level of the pyramid. This indicates a downsampling operation. Indicates the Gaussian blur operation; Based on the Gaussian pyramid, a Laplacian pyramid is constructed from the registered image; the construction process of the Laplacian pyramid is represented as follows: ,in, This is the current layer of the Pyramid of Laplace. This is the current layer of the Gaussian pyramid. It is the result of upsampling the previous layer image; A weighted map corresponding to the resolution of each layer of the Laplace pyramid is generated based on the smooth seams. The Laplacian pyramid of each pair of registered images is weighted and fused using a weighted graph; the fusion process is represented as follows: ; It is the current layer of the weighted and merged Laplace pyramid. and These are the current Laplacian pyramid levels for the two images. This is the weight graph of the current layer; By upsampling and reconstructing the weighted and fused Laplacian pyramid layer by layer, a seamless panoramic image is obtained.

7. A panoramic image stitching method according to claim 1, characterized in that, Images of the vehicle's surroundings from different perspectives were captured using four fisheye cameras.

8. A panoramic image stitching system, characterized in that, include: The acquisition module is used to acquire images of the vehicle's surroundings from different perspectives; The registration module is used to perform a masking operation on images around the vehicle from different viewpoints to generate an overlapping region mask for the images around the vehicle from different viewpoints; local feature detection is performed on the overlapping region mask to generate an image transformation matrix. Based on the image transformation matrix, the images around the vehicle from different perspectives are aligned and registered; The stitching optimization module is used to calculate the comprehensive energy function by using the pixel color difference and structural gradient difference in the overlapping area of ​​the image under different viewpoints after alignment and registration. Based on the comprehensive energy function, the image is guided to avoid objects with structural features at the stitching seam. The module also uses a dynamic programming algorithm to search for the initial stitching seam path in the overlapping area and performs B-spline interpolation optimization on the initial stitching seam path to obtain a smooth stitching seam. The fusion module is used to fuse registered images based on smooth stitching seams and employs a multi-resolution pyramid fusion strategy to output seamless panoramic images.

9. A panoramic image stitching computer device, characterized in that, include: A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the panoramic image stitching method according to any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which includes program instructions that, when executed by a processor, perform the steps of the panoramic image stitching method according to any one of claims 1-7.

Images