Panoramic image stitching method and device and electronic equipment

Abstract

The invention provides a panoramic image splicing method and device and electronic equipment, and the method comprises the steps: obtaining a plurality of images and metadata thereof, carrying out the preprocessing of the plurality of images according to the metadata, and obtaining a plurality of preprocessed images; extracting image key points and feature descriptors of the image key points from the plurality of preprocessed images through a SuperPoint neural network model; performing approximate nearest neighbor search matching on the image key points according to the feature descriptors of the image key points to obtain a plurality of initial matching pairs; eliminating matching pairs with spatial consistency smaller than a threshold value from the plurality of initial matching pairs through a random sampling consistency algorithm to obtain final matching pairs; based on the final matching pair, performing global binding adjustment on camera transformation parameters corresponding to the plurality of images through a nonlinear optimization algorithm to obtain alignment parameters, and transforming the plurality of images according to the alignment parameters to obtain a plurality of transformed images; and performing multi-band fusion rendering on the plurality of transformed images to obtain a panoramic image.

Description

Technical Field

[0001] This application relates to the technical fields of computer vision, image stitching and panoramic image generation, and more specifically, to a panoramic image stitching method, apparatus and electronic device. Background Technology

[0002] Current panoramic image stitching methods rely on manually designed feature extraction algorithms (such as SIFT and SURF), which are sensitive to changes in lighting, viewpoint differences, and low-texture areas (such as the sky). This results in sparse keypoint distribution and low matching accuracy, directly affecting subsequent geometric correction and image fusion. For example, in low-texture areas, traditional methods often fail to establish reliable matching pairs due to a lack of effective feature points, making it impossible to balance detail preservation and seam elimination. This leads to broken seams or missing content in the final panoramic image. Therefore, the panoramic images fused by current methods have poor fusion quality. Summary of the Invention

[0003] The purpose of this application is to provide a panoramic image stitching method, apparatus, and electronic device to improve the problem of poor fusion quality of the obtained panoramic images.

[0004] This application provides a panoramic image stitching method, comprising: acquiring multiple images and metadata of each image, and preprocessing the multiple images according to the metadata to obtain preprocessed multiple images, wherein the multiple images have partially overlapping fields of view; extracting image key points and feature descriptors of the image key points with scale and rotation robustness from the preprocessed multiple images using a SuperPoint neural network model; performing approximate nearest neighbor search matching on the image key points according to the feature descriptors of the image key points to obtain multiple initial matching pairs; removing matching pairs with spatial consistency less than a threshold from the multiple initial matching pairs using the geometric constraints of a random sampling consensus algorithm to obtain final matching pairs; based on the final matching pairs, globally binding and adjusting the camera transformation parameters corresponding to the multiple images using a nonlinear optimization algorithm to obtain alignment parameters, and transforming the multiple images according to the alignment parameters to obtain multiple transformed images; and performing multi-band fusion rendering on the multiple transformed images to obtain a panoramic image. In the implementation of the above scheme, by combining metadata preprocessing and the SuperPoint neural network model to extract image key points and feature descriptors with scale and rotation robustness, the robustness and accuracy of image key point extraction can be significantly improved. Furthermore, by performing fusion rendering on the transformed image, stable feature matching performance can still be maintained when processing images with different lighting, viewing angles and scales, effectively eliminating brightness differences and color discontinuities in the stitching area, smoothly transitioning the edges and textures of different images, and finally generating a seamless panoramic image with good visual effects, thereby improving the quality of panoramic image stitching.

[0005] Optionally, in this embodiment, the SuperPoint neural network model includes: a shared encoder, a detection branch, and a description branch. Extracting scale- and rotation-robust image keypoints and their feature descriptors from multiple preprocessed images using the SuperPoint neural network model includes: for each of the preprocessed images, using the shared encoder to extract features; using the detection branch to detect image keypoints; using the description branch to extract a description map from the image features to obtain a dense description map; and sampling the dense description map based on the image keypoints to obtain feature descriptors for the image keypoints. In the implementation of the above scheme, through the sampling mechanism of the dense description map, the model can dynamically extract corresponding feature descriptors based on the detected keypoints. This adaptive method enhances the flexibility and specificity of feature description. Compared to the method of extracting features from fixed regions, this method can better adapt to images with different scales and rotational changes, improving the accuracy of feature matching. Furthermore, the end-to-end design of the entire SuperPoint neural network model automates the entire process from image input to key point and feature descriptor output, reducing the need for manual intervention and parameter tuning. This integrated solution not only simplifies system complexity but also improves the model's generalization ability and adaptability.

[0006] Optionally, in this embodiment, using a detection branch to detect key points in image features includes: using the detection branch to detect key points in image features to obtain a detection feature map; converting the detection feature map into a heatmap, and performing non-maximum suppression on the heatmap according to a preset suppression threshold to obtain image key points. In the implementation of the above scheme, by combining detection branch and non-maximum suppression techniques, high detection efficiency can be maintained while achieving accurate localization of image key points, balancing detection speed and accuracy, making it suitable for application scenarios with high real-time requirements. Specifically, by converting the detection feature map into a heatmap, the probability distribution of key points in each region of the image can be visually displayed, thereby more accurately locating the key point positions and improving the accuracy and visualization effect of key point detection. Furthermore, by performing non-maximum suppression on the heatmap according to a preset suppression threshold, low-probability noise points can be effectively filtered out, while high-probability key points are retained, thereby reducing the false detection rate and improving the robustness of key point detection.

[0007] Optionally, in this embodiment, the multiple images include: a first image and a second image; approximate nearest neighbor search matching is performed on the key points of the images based on their feature descriptors, including: constructing index parameters and search parameters for the FLANN fast library of approximate nearest neighbors based on a KD-tree; constructing a FLANN matcher using the index parameters and search parameters, and matching the feature descriptors of the first image and the second image using the FLANN matcher and the K-nearest neighbor algorithm to obtain multiple matching results; filtering the multiple matching results to obtain multiple initial matching pairs. In the implementation of the above scheme, by using the FLANN matcher and the K-nearest neighbor algorithm to match the feature descriptors of the first image and the second image, the number of mismatches can be effectively reduced, and the accuracy and reliability of matching can be improved, thereby providing higher quality initial matching pairs for subsequent image processing tasks. Furthermore, by filtering the multiple matching results, inaccurate matching pairs can be further eliminated, and high-quality initial matching pairs can be retained, thereby improving the overall accuracy and robustness of image matching.

[0008] Optionally, in this embodiment, after obtaining the final matching pair, the method further includes: pre-screening and sorting multiple initial matching pairs based on the confidence level of image key points and the similarity between feature descriptors to generate sorted candidate matching pairs; using the PROSAC algorithm to perform progressive iterative sampling on the sorted candidate matching pairs to calculate the reprojection error of the final matching pair; and removing matching pairs from the final matching pairs whose reprojection error is greater than a preset error threshold. In the implementation of the above scheme, by using the PROSAC algorithm to perform progressive iterative sampling on the sorted candidate matching pairs, the pre-sorting information can be fully utilized, gradually expanding from high-confidence matching. Compared with random sampling methods, it can converge to the optimal solution faster. This ordered sampling method greatly reduces the waste of computational resources while improving matching accuracy and algorithm stability. Furthermore, by calculating and removing matching pairs whose reprojection error is greater than the threshold, accurate screening of the matching results is achieved, ensuring the geometric consistency of the final matching pairs. This step effectively eliminates mismatches caused by factors such as viewpoint changes and occlusion, significantly improving the reliability and robustness of the matching results.

[0009] Optionally, in this embodiment, multi-band fusion rendering of multiple transformed images includes: multi-band fusion of multiple transformed images to obtain a fused image; dynamic range synthesis and tone mapping of the fused image to obtain a tone-mapped image; and color consistency correction of the tone-mapped image to obtain a panoramic image. In the implementation of the above scheme, multi-band fusion processing of multiple transformed images effectively preserves image detail information from different frequency bands, avoiding detail loss or blurring caused by single-band processing, thereby significantly improving the overall clarity and detail of the fused image. Furthermore, by performing dynamic range synthesis and tone mapping on the fused image, the brightness distribution and color contrast of the image can be adaptively adjusted, ensuring balanced representation of both bright and dark areas in high dynamic range scenes, thereby enhancing the visual realism and viewing comfort of the image.

[0010] Optionally, in this embodiment, multi-band fusion of multiple transformed images includes: constructing a Laplacian pyramid and a weight pyramid for each transformed image; and performing multi-band weighted fusion of the transformed image based on the Laplacian pyramid and the weight pyramid. In the implementation of the above scheme, by using a weight pyramid for multi-band weighted fusion, the fusion weights can be dynamically adjusted according to the importance of different frequency bands, making important features more prominent in the fusion result. This adaptive weighting strategy overcomes the limitations of traditional fixed-weight fusion methods and can better adapt to the feature distribution of different image content, thus producing a more natural and visually perceptual fusion effect. Furthermore, by combining the Laplacian pyramid and the weight pyramid, the synergistic effect of frequency band separation and weighted fusion is achieved. The Laplacian pyramid provides accurate frequency band decomposition, while the weight pyramid achieves fine-grained frequency band control. The combination of the two enables the fusion process to achieve optimal balance at each frequency band level, thereby producing a fusion effect that surpasses simple linear superposition.

[0011] Optionally, in this embodiment, after obtaining the fused image, the method further includes: detecting sky regions and non-sky regions from the fused image; fusing the sky regions using a first fusion method to obtain a fused sky region; and fusing the non-sky regions using a second fusion method, wherein the fusion weight of the first fusion method is reduced relative to the fusion weight of the second fusion method; and performing repair processing on the fused sky regions. In the implementation of the above scheme, by detecting sky regions and non-sky regions from the fused image and processing them using different fusion methods respectively, optimization can be performed on the characteristics of different regions, thereby improving the overall image fusion quality. This partitioned processing method avoids the local distortion problem caused by applying uniform fusion parameters to the entire image in traditional methods, ensuring that both sky and non-sky regions achieve the best visual effect. Furthermore, by performing repair processing on the fused sky regions, imperfections or noise that may be introduced during the fusion process can be further eliminated, improving the purity and realism of the sky regions. This step compensates for any shortcomings that may exist during the fusion process, making the final image visually more coherent and natural.

[0012] Optionally, in this embodiment, the fused sky region is repaired by: determining whether the area of ​​the fused sky region is less than a preset area threshold; if so, the fused sky region is directly repaired using the OpenCV inpaint function; otherwise, a pre-trained SkyGAN generative adversarial network model is called to repair the fused sky region. In the implementation of the above scheme, different repair methods are dynamically selected based on the size of the sky region, thus optimizing the allocation of computational resources. The lightweight OpenCV inpaint function is used for small areas, reducing computational overhead; while the more complex SkyGAN model is used for large areas, ensuring repair quality and improving overall processing efficiency while maintaining effectiveness. Furthermore, by combining traditional image processing algorithms (OpenCV inpaint) with deep learning models (SkyGAN), a hybrid repair strategy is formed. This combination not only leverages the efficiency advantages of traditional algorithms in handling simple tasks but also utilizes the powerful generative capabilities of deep learning in complex scenes, thereby achieving the most suitable repair scheme for sky regions of different sizes.

[0013] This application also provides a panoramic image stitching device, comprising: an image data acquisition and processing module, used to acquire multiple images and metadata of each image, and preprocess the multiple images according to the metadata to obtain preprocessed multiple images, wherein the multiple images have partially overlapping fields of view; a key point descriptor extraction module, used to extract image key points with scale and rotation robustness and feature descriptors of image key points from the preprocessed multiple images using a SuperPoint neural network model; an image nearest neighbor search and matching module, used to perform approximate nearest neighbor search and matching on the image key points according to the feature descriptors of the image key points to obtain multiple initial matching pairs; an initial matching data elimination module, used to eliminate matching pairs with spatial consistency less than a threshold from the multiple initial matching pairs through geometric constraints of a random sampling consensus algorithm to obtain final matching pairs; an image alignment adjustment and transformation module, used to globally bundle and adjust the camera transformation parameters corresponding to the multiple images based on the final matching pairs through a nonlinear optimization algorithm to obtain alignment parameters, and transform the multiple images according to the alignment parameters to obtain multiple transformed images; and an image frequency band fusion rendering module, used to perform multi-frequency band fusion rendering on the multiple transformed images to obtain a panoramic image.

[0014] Optionally, in this embodiment, the SuperPoint neural network model includes: a shared encoder, a detection branch, and a description branch; the keypoint descriptor extraction module includes: an image feature extraction submodule, used to extract features from each of the preprocessed images using the shared encoder to obtain image features; an image keypoint detection submodule, used to use the detection branch to detect image keypoints from the image features to obtain image keypoints; an image descriptor extraction module, used to use the description branch to extract a description map from the image features to obtain a dense description map; and a feature descriptor sampling module, used to sample the dense description map based on the image keypoints to obtain feature descriptors for the image keypoints.

[0015] Optionally, in this embodiment of the application, the image key point detection submodule includes: a detection feature map acquisition unit, used to use a detection branch to detect image key points on image features and obtain a detection feature map; and an image key point acquisition unit, used to convert the detection feature map into a heatmap and perform non-maximum suppression on the heatmap according to a preset suppression threshold to obtain image key points.

[0016] Optionally, in this embodiment, the multiple images include: a first image and a second image; the image nearest neighbor search matching module includes: a matching result acquisition submodule, used to construct index parameters and search parameters of the approximate nearest neighbor fast library FLANN based on KD-tree; construct a FLANN matcher using the index parameters and search parameters, and match the feature descriptors of the first image and the second image using the FLANN matcher and the K-nearest neighbor algorithm to obtain multiple matching results; and an initial matching pair acquisition submodule, used to filter the multiple matching results to obtain multiple initial matching pairs.

[0017] Optionally, in this embodiment of the application, the panoramic image stitching device further includes: a matching pair pre-screening and sorting module, used to pre-screen and sort multiple initial matching pairs according to the confidence level of image key points and the similarity between feature descriptors, to generate sorted candidate matching pairs; an iterative sampling error calculation module, used to perform progressive iterative sampling on the sorted candidate matching pairs using the PROSAC algorithm to calculate the reprojection error of the final matching pair; and a final matching pair elimination module, used to eliminate matching pairs from the final matching pairs whose reprojection error is greater than a preset error threshold.

[0018] Optionally, in this embodiment of the application, the image frequency band fusion rendering module includes: a fused image acquisition submodule, used to perform multi-frequency band fusion on multiple transformed images to obtain a fused image; a synthesized tone mapping submodule, used to perform dynamic range synthesis and tone mapping on the fused image to obtain a tone-mapped image; and a panoramic image acquisition submodule, used to perform color consistency correction on the tone-mapped image to obtain a panoramic image.

[0019] Optionally, in this embodiment, the image fusion acquisition submodule includes: a pyramid construction unit, used to construct a Laplacian pyramid and a weight pyramid corresponding to each of the multiple transformed images; and a multi-band fusion unit, used to perform multi-band weighted fusion of the transformed image based on the Laplacian pyramid and the weight pyramid.

[0020] Optionally, in this embodiment, the image frequency band fusion rendering module further includes: an image region detection submodule, used to detect sky regions and non-sky regions from the fused image; an image region fusion submodule, used to fuse the sky region using a first fusion method to obtain the fused sky region, and to fuse the non-sky region using a second fusion method, wherein the fusion weight of the first fusion method is reduced relative to the fusion weight of the second fusion method; and a sky region repair submodule, used to repair the fused sky region.

[0021] Optionally, in this embodiment, the sky region repair submodule includes: a region area judgment unit, used to determine whether the area value of the fused sky region is less than a preset area threshold; and a sky region repair unit, used to directly repair the fused sky region using OpenCV's inpaint function if the area value of the fused sky region is less than the preset area threshold, and to call a pre-trained SkyGAN generative adversarial network model to repair the fused sky region if the area value of the fused sky region is greater than or equal to the preset area threshold.

[0022] This application also provides an electronic device, including a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform the methods described above.

[0023] This application also provides a computer-readable storage medium storing a computer program, which is executed by a processor to perform the methods described above.

[0024] This application also provides a computer program product, including: a computer program or computer instructions, which are executed by a processor to perform the method described above. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 The illustration shows a flowchart of the panoramic image stitching method provided in an embodiment of this application; Figure 2 The diagram shown illustrates the processing procedure for the matching results provided in an embodiment of this application. Figure 3 The diagram shown is a structural schematic of the panoramic image stitching device provided in an embodiment of this application. Figure 4 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in the embodiments of this application are for illustrative and descriptive purposes only and are not intended to limit the protection scope of the embodiments of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in the embodiments of this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of the embodiments of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0028] Furthermore, the described embodiments are merely a part of the embodiments of this application, and not all of them. The components of the embodiments of this application described and illustrated herein can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of this application, but merely to illustrate selected embodiments of this application.

[0029] It is understood that the terms "first" and "second" in the embodiments of this application are used to distinguish similar objects. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different. In the description of the embodiments of this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. The term "multiple" refers to two or more (including two), and similarly, "multiple groups" refers to two or more groups (including two groups).

[0030] It should be noted that the panoramic image stitching method provided in this application embodiment can be executed by an electronic device. Here, electronic device refers to a device terminal or server with the function of executing computer programs. Device terminals include, for example, smartphones, personal computers, tablets, personal digital assistants, or mobile internet devices. Servers refer to devices that provide computing services through a network. Servers include, for example, x86 servers and non-x86 servers. Non-x86 servers include, for example, mainframes, minicomputers, and UNIX servers.

[0031] Most mainstream panoramic image stitching methods currently rely on manually designed feature extraction algorithms, such as Scale-invariant Feature Transform (SIFT) and Speeded UpRobust Features (SURF). These algorithms define the detection rules and description methods for keypoints through mathematical models. While they perform well under ideal conditions, their performance is highly dependent on the texture richness and illumination consistency of the image. When encountering regions with drastic lighting changes, large viewpoint differences, or areas where the image content itself lacks obvious edges and corners (such as large areas of sky, white walls, water surfaces, and other low-texture areas), these algorithms struggle to consistently detect a sufficient number of keypoints, resulting in sparse and poorly repeatable distribution of extracted feature points. Since subsequent geometric correction (such as homography matrix estimation) and image fusion heavily depend on high-quality, uniformly distributed matching point pairs, the sparsity and low precision of the initial matching directly lead to large model estimation biases and inaccurate alignment, thus affecting the quality of the final fused image.

[0032] After capturing multiple overlapping images containing large areas of sky using a camera, the sky region is typically uniform in color and lacks obvious texture and structural information. Traditional SIFT or SURF algorithms can hardly detect effective keypoints in such areas, resulting in very few or even zero matching points. During image alignment, due to the lack of sufficient constraints, the stitching algorithm can only rely on matching points in textured areas such as image edges or the ground to infer the overall transformation relationship, which easily introduces geometric distortion. In the fusion stage, even when using techniques such as weighted fusion, the lack of reliable matching criteria for the sky region means that there may be subtle differences in the color and brightness of the sky in different images. These differences cannot be smoothly transitioned at the stitching seams, ultimately forming obvious seams or breaks. At the same time, over-reliance on blurring or feathering to cover up seams will lose image details, leading to "ghosting" or blurring phenomena, failing to balance detail preservation and seamless fusion.

[0033] The core bottleneck of existing panoramic stitching methods lies in their insufficient adaptability to complex scenes such as low texture and varying lighting conditions during feature extraction. This deficiency manifests first in the sparsity and instability of keypoint detection, leading to decreased matching accuracy. This decline in matching quality directly causes error accumulation in geometric correction and seam problems in image fusion. Especially in common but feature-scarce scenes such as skies and white walls, traditional methods struggle to establish reliable correspondences, preventing truly seamless fusion. Therefore, due to fundamental flaws in the feature extraction stage, existing technologies cannot guarantee geometric consistency and visual continuity throughout the stitching process. This results in panoramic images with noticeable seams, missing content, or blurred details, indicating poor fusion quality from current panoramic image stitching methods.

[0034] To improve the above issues, please see Figure 1 The illustrated flowchart shows a panoramic image stitching method provided in this application embodiment. The main idea of ​​this panoramic image stitching method is to maintain stable feature matching performance when processing images with different lighting, viewing angles, and scale changes by combining metadata preprocessing and the SuperPoint neural network model, as well as performing fusion rendering on the transformed image. This significantly improves the robustness and accuracy of image keypoint extraction, thereby enhancing the quality of panoramic image stitching. The implementation of the above-described panoramic image stitching method may include: Step S110: Obtain multiple images and metadata for each image, and preprocess the multiple images according to the metadata to obtain preprocessed multiple images with partially overlapping fields of view among the multiple images.

[0035] Multiple images refer to multiple photos or images taken from different perspectives or positions with partially overlapping views, such as the original frames of a 360-degree panoramic photo taken with a smartphone, or ground images taken in succession by a drone.

[0036] Image metadata refers to additional information embedded in an image file that describes the image's attributes, such as EXIF ​​metadata. The EXIF ​​metadata mentioned above includes camera parameters such as lens focal length, GPS geographic coordinates, shooting timestamp, aperture, and ISO.

[0037] Step S120: Extract scale- and rotation-robust image keypoints and their feature descriptors from the preprocessed multiple images using the SuperPoint neural network model.

[0038] The SuperPoint neural network model is a deep learning model specifically designed for extracting keypoints and their feature descriptors from images. It can be a keypoint detector based on a convolutional neural network, and is an end-to-end model for outputting keypoint locations and descriptors. It is understood that by combining the SuperPoint neural network model to extract scale- and rotation-robust image keypoints and feature descriptors, the accuracy and stability of feature point matching can be significantly improved. Especially when images have significant viewpoint changes or lighting differences, it can still maintain a high matching success rate, thus solving the problem of declining matching performance of traditional feature extraction methods in complex scenes.

[0039] Step S130: Perform approximate nearest neighbor search matching on the key points of the image based on the feature descriptors of the key points to obtain multiple initial matching pairs.

[0040] Image keypoints are points in an image that have significant features and can be used for matching, such as the intersection of object edges or the center point of a textured region.

[0041] A feature descriptor is a mathematical feature vector that describes the image region surrounding a keypoint. It is robust to scale and rotation, such as the 256-dimensional SuperPoint descriptor. Specifically, feature descriptors of the image region surrounding a keypoint can be calculated using feature extraction algorithms.

[0042] Approximate nearest neighbor search matching is an efficient feature matching method that quickly finds the most similar features within an acceptable error range.

[0043] The initial matching pair is a pair of corresponding key points in two images that are likely to be correct, found through feature matching. For example, key point 1 in image A matches key point 5 in image B, and key point 3 in image A matches key point 8 in image B.

[0044] Step S140: Eliminate matching pairs with spatial consistency less than a threshold from multiple initial matching pairs using the geometric constraints of the random sampling consensus algorithm to obtain the final matching pairs.

[0045] The Random Sample And Consensus (RANSAC) algorithm is a robust algorithm that estimates mathematical model parameters through iterative random sampling. It is a geometric constraint algorithm used to eliminate mismatches. The geometric constraints of the RANSAC algorithm are restrictions on matching point pairs based on spatial geometric relationships, such as fundamental matrix geometric constraints or homography matrix constraints.

[0046] The final matching pairs are the correctly matched keypoint pairs retained after geometric verification, such as those retained after RANSAC filtering. It is understandable that by using the Random Sampling Consensus Algorithm (RANSAC) to perform geometric constraint filtering on the initial matching pairs, mismatches caused by noise, occlusion, or duplicate textures are effectively eliminated, improving the spatial consistency of the matching pairs. This ensures the accuracy of subsequent image alignment and stitching, avoiding obvious misalignment or ghosting phenomena in the stitched image.

[0047] Step S150: Based on the final matching pair, the camera transformation parameters corresponding to multiple images are globally bundled and adjusted using a nonlinear optimization algorithm to obtain alignment parameters. The multiple images are then transformed according to the alignment parameters to obtain multiple transformed images.

[0048] Nonlinear optimization algorithms are optimization methods used to solve nonlinear least squares problems, such as the Levenberg-Marquardt algorithm, the Gauss-Newton method, or the conjugate gradient method. In essence, by globally binding and adjusting camera transformation parameters using nonlinear optimization algorithms, the alignment parameters between multiple images are optimized, resulting in higher geometric consistency of the stitched panoramic image globally. This reduces the impact of accumulated errors, thereby improving the overall quality and visual coherence of the panoramic image.

[0049] Alignment parameters are a set of parameters that describe the geometric transformation relationships between images, such as rotation matrices, translation vectors, camera intrinsic and extrinsic parameters, etc.

[0050] Step S160: Perform multi-band fusion rendering on multiple transformed images to obtain a panoramic image.

[0051] Multi-band fusion rendering is an image processing technique that weights and fuses different frequency bands (such as high and low frequencies). This technique can be used to fuse colors in the low-frequency band and details in the high-frequency band, thereby reducing seams and lighting differences. In essence, by using multi-band fusion rendering to fuse transformed images, it can smoothly handle lighting and color differences at image stitching boundaries, avoiding obvious seams or unnatural transitions, thus generating more natural and seamless panoramic images.

[0052] Panoramic images are wide-view or 360-degree images obtained by stitching together multiple images, such as wide-angle landscape photos or aerial maps.

[0053] In the implementation of the above scheme, by combining metadata preprocessing and the SuperPoint neural network model to extract image key points and feature descriptors with scale and rotation robustness, the robustness and accuracy of image key point extraction can be significantly improved. Furthermore, by performing fusion rendering on the transformed image, stable feature matching performance can still be maintained when processing images with different lighting, viewing angles and scales, effectively eliminating brightness differences and color discontinuities in the stitching area, smoothly transitioning the edges and textures of different images, and finally generating a seamless panoramic image with good visual effects, thereby improving the quality of panoramic image stitching.

[0054] As an optional implementation of step S110 above, for example, the input processing module in the panoramic image stitching method can be used to acquire multiple images and metadata for each image, and preprocess the multiple images based on the metadata. This input processing module is the core front-end component of the panoramic image stitching method, integrating three major technical dimensions: multi-format compatibility, deep metadata parsing, and preprocessing parameter generation. Functional integration can be achieved through a layered architecture design. After acquiring multiple images, the libtiff library can be used to parse TIFF and RAW format images from the multiple images. It also supports parsing JPEG format images. The TIFFReadRGBAImage tool can be used to perform lossless image compression and pixel decoding on the multiple images. Then, the ExifTool toolchain can be used to extract EXIF ​​metadata in formats such as JPEG / RAW / TIFF from the multiple images. The aforementioned EXIF ​​metadata includes camera parameters such as lens focal length, GPS geographic coordinates, shooting timestamp, aperture, and ISO, which are used to provide the initial projection matrix calculation basis for subsequent geometric correction.

[0055] Optionally, a conditional branching strategy can be used to perform multimodal data adaptation on the aforementioned multiple images. For example, for fisheye lens images taken by a drone, OpenCV's `initUndistortRectifyMap` can be called to perform nonlinear distortion correction on multiple images. For medical images, a smoothing mask can be generated using the medical image BET algorithm, and then used to eliminate black borders in multiple images. For unevenly exposed image sequences, the `ExposureTime` parameter can be extracted from multiple images using `exiftool`. These parameters can trigger the CLAHE algorithm for local histogram equalization, effectively eliminating brightness discontinuities at the stitching seams.

[0056] Optionally, the aforementioned multiple images can be preprocessed. For example, the CLAHE algorithm based on OpenCV can be used to optimize the dynamic range of the multiple images, thereby improving both local contrast enhancement and noise suppression. Specifically, the multiple images are first converted to the LAB color space. Then, the luminance channel (L channel) is divided into 8×8 blocks, and bilinear interpolation is used to eliminate artifacts at the block boundaries. A contrast limiting mechanism can be used; when the local histogram grayscale level statistics exceed a threshold, the excess pixels are uniformly redistributed to each grayscale level, thereby effectively suppressing noise amplification.

[0057] As an optional implementation of step S120 above, the SuperPoint neural network model may include: a shared encoder, a detection branch, and a description branch; the implementation of extracting scale- and rotation-robust image keypoints and feature descriptors of image keypoints from preprocessed multiple images using the SuperPoint neural network model may include: Step S121: For each of the preprocessed images, use a shared encoder to extract features from the image to obtain image features.

[0058] An example implementation of step S121 above involves using a VGG-style encoder as a shared encoder for each of the preprocessed images to extract features and obtain image features. The VGG-style encoder can be constructed using a multi-layer convolutional network, with its output channels set to 128 dimensions and its output size set to 8 (height × width × 128 channels), thereby compressing the input image into a low-resolution, high-dimensional feature map. In this implementation, using a shared encoder to extract features from multiple images uniformly significantly reduces redundant computational resource consumption while ensuring consistency in features processed by different branches, thus improving the overall model's efficiency and accuracy. This design avoids the redundant operation of performing multiple independent feature extractions on the same image in traditional methods, achieving efficient utilization of computational resources.

[0059] Step S122: Use the detection branch to detect key points in the image features and obtain key points in the image.

[0060] It is understood that the implementation of step S122 above is relatively complex. Therefore, the implementation of step S122 will be described in detail below. Specifically, the detection branch described above can employ a detection Sequential network consisting of a first convolutional layer (Conv2d), a ReLU activation function, a second convolutional layer (Conv2d), and a Softmax layer to achieve keypoint detection.

[0061] Step S123: Use the description branch to extract the description map of the image features to obtain a dense description map.

[0062] The implementation of step S123 above can be exemplified by using a detection sequential network consisting of a third convolutional layer (Conv2d), a ReLU activation function, a fourth convolutional layer (Conv2d), and a normalized description vector (L2Norm) as the description branch to extract description maps from image features, obtaining a dense description map, which can be represented as desc_tensor. In the implementation of the above scheme, the separation of the detection and description branches enables the model to process keypoint detection and feature description tasks in parallel, improving processing speed and allowing each branch to focus on its respective optimization objective. This clearly defined architecture makes keypoint detection more accurate and feature description more robust, thus maintaining high performance even in complex scenes.

[0063] Step S124: Sample the dense descriptive map based on the key points of the image to obtain the feature descriptors of the key points of the image.

[0064] The implementation of step S124 above is as follows: The `sample_descriptors` function is used to perform bilinear interpolation sampling on the image keypoints and the dense descriptor map (desc_tensor) to obtain the feature descriptor of the image keypoint. It can be understood that the `sample_descriptors` function extracts the descriptor vector corresponding to the detected image keypoint coordinates from the dense descriptor map, maps the keypoint coordinates to a low-resolution descriptor map grid, and finally uses the N×256 matrix output after accurate sampling descriptor processing via bilinear interpolation as the feature descriptor of the image keypoint. This approach effectively improves the complex local computation of traditional hand-crafted descriptors (such as SIFT), directly utilizing deep learning features to enhance the generation efficiency and robustness of feature descriptors.

[0065] As an optional implementation of step S122 above, the implementation of using detection branches to detect image key points of image features may include: Step S122a: Use the detection branch to detect key points in the image features and obtain the detection feature map.

[0066] For example, step S122a can be implemented by using the SIFT, SURF, or Oriented Fast and Rotated BRIEF (ORB) algorithms in the detection branch to detect key points in the image features and obtain a detection feature map. Optionally, during the use of the detection branch, the probability distribution of each spatial location as a key point can be output through a convolutional layer, and a "dustbin" channel can be introduced to filter out invalid regions in the image features.

[0067] Step S122b: Convert the detection feature map into a heatmap, and perform non-maximum suppression on the heatmap according to a preset suppression threshold to obtain image key points.

[0068] The implementation of step S122b above includes, for example, converting the detection feature map into a heatmap using the Softmax layer in the detection branch, and performing Fast Non-Maximum Suppression (Fast-NMS) on the heatmap according to a preset suppression threshold to obtain image keypoints. In the implementation of the above scheme, by converting the detection feature map into a heatmap, the probability distribution of keypoints in each region of the image can be visually displayed, thereby more accurately locating the keypoints and improving the accuracy and visualization effect of keypoint detection. Furthermore, by performing non-maximum suppression on the heatmap according to the preset suppression threshold, low-probability noise points can be effectively filtered out, while high-probability keypoints are retained, thereby reducing the false detection rate and improving the robustness of keypoint detection.

[0069] As an optional implementation of step S130 above, the multiple images mentioned above may include: a first image and a second image; the implementation of performing approximate nearest neighbor search matching on the image key points based on the feature descriptors of the image key points may include: Step S131: Construct the index parameters and search parameters of the approximate nearest neighbor fast library FLANN based on the KD tree.

[0070] An example implementation of step S131 above is as follows: The Fast Library for Approximate Nearest Neighbors (FLANN) and a KD-tree are used to construct the index parameters and search parameters of FLANN. This process can be represented in code as FLANN+KNN matching: `auto index_params = cv::makePtr` <cv::flann::kdtreeindexparams>(5) and auto search_params = cv::makePtr <cv::flann::searchparams>(50). Here, `index_params` represents the index parameters for constructing FLANN based on a KD-tree (K-Dimensional Tree), and `search_params` represents the search parameters for FLANN. In the implementation of the above scheme, by constructing the index and search parameters of FLANN based on a KD-tree, the feature descriptors of image keypoints can be efficiently organized and managed, thereby significantly improving the speed and efficiency of approximate nearest neighbor search, making the keypoint matching process in large-scale image datasets faster and more efficient.

[0071] Step S132: Construct a FLANN matcher using index parameters and search parameters, and use the FLANN matcher and the K-nearest neighbor algorithm to match the feature descriptors of the first image and the feature descriptors of the second image to obtain multiple matching results.

[0072] An example implementation of step S132 above is as follows: A FLANN matcher is constructed based on the index parameters and search parameters, which can be represented by the code `cv::FlannBasedMatcher matcher(index_params, search_params)`. Here, `index_params` represents the index parameters for constructing the FLANN based on a K-Dimensional Tree, and `search_params` represents the search parameters for the FLANN. The FLANN matcher and the K-Nearest Neighbor (KNN) algorithm are used to perform approximate nearest neighbor search matching on the feature descriptors of the first and second images based on similarity, thereby obtaining multiple matching results. These multiple matching results can be represented as `knn_matches = matcher.knnMatch(desc1, desc2, 2)`, where `desc1` represents the feature descriptor of the first image, `desc2` represents the feature descriptor of the second image, and the matching ratio is the parameter used for testing. In the implementation of the above scheme, by using the FLANN matcher and the K-nearest neighbor algorithm to match the feature descriptors of the first and second images, the number of mismatches can be effectively reduced, and the accuracy and reliability of matching can be improved, thereby providing higher quality initial matching pairs for subsequent image processing tasks.

[0073] Step S133: Filter the multiple matching results to obtain multiple initial matching pairs.

[0074] An example implementation of step S133 above is as follows: For each K-nearest neighbor matching result among multiple matching results, a ratio test is performed on the K-nearest neighbor matching result. K-nearest neighbor matching results with ratios less than a preset threshold are selected from the multiple matching results, and these K-nearest neighbor matching results with ratios less than the preset threshold are determined as the aforementioned initial matching pairs. In the implementation of the above scheme, by filtering multiple matching results, inaccurate matching pairs can be further eliminated, and high-quality initial matching pairs can be retained, thereby improving the overall accuracy and robustness of image matching.

[0075] Please see Figure 2 The diagram illustrates the processing steps for matching results provided in this embodiment of the application. As an optional implementation of step S140 above, after obtaining the final matching pair, it may further include: Step S141: Based on the confidence level of image key points and the similarity between feature descriptors, pre-screen and sort multiple initial matching pairs to generate sorted candidate matching pairs.

[0076] An example implementation of step S141 above is as follows: For each of the multiple initial matching pairs, a quality score is calculated based on the confidence level of the image keypoints and the similarity between the feature descriptors. The multiple initial matching pairs are then pre-screened based on this quality score to obtain the pre-screened initial matches. Next, the pre-screened initial matches are sorted according to the quality score to obtain the sorted candidate matching pairs. In the implementation of the above scheme, by combining the confidence level of the image keypoints and the similarity between the feature descriptors for pre-screening and sorting, candidate matching pairs with higher matching quality can be preferentially retained, thereby significantly improving the efficiency and accuracy of subsequent matching processing. This dual-screening mechanism effectively reduces the interference of low-quality matching pairs and provides more reliable input data for subsequent processing.

[0077] Understandably, the above method calculates a comprehensive quality score for each matching pair by fusing the confidence score (keypoint.response) of image keypoints (i.e., SuperPoint feature points) and the matching similarity (i.e., 1 - match.distance) between feature descriptors, and sorts them from high to low scores. In subsequent sampling, samples are preferentially drawn from this subset to ensure rapid approximation of the correct model in early iterations.

[0078] Step S142: Use the PROSAC algorithm to perform progressive iterative sampling on the sorted candidate matching pairs to calculate the reprojection error of the final matching pair.

[0079] Understandably, the aforementioned incremental iterative sampling involves initially sampling only from the top 20% of the ranked matches, and then gradually expanding the sampling range as the number of iterations increases, according to the formula current_ratio = initial_ratio * (growth_rate)^iter. This incremental iterative sampling mechanism strikes a balance between exploring high-probability regions and global search, avoiding getting trapped in local optima.

[0080] The implementation of step S142 above is as follows: First, an initial iteration upper limit value is set. This initial iteration upper limit value will be dynamically calculated and reassigned in subsequent calculations. Then, based on this iteration upper limit value, progressive iterative sampling is performed from the sorted candidate matching pairs. Random sampling is prioritized from the high-quality matches ranked higher, thereby sampling multiple pairs of matching points and increasing the probability of finding the correct model. Next, a hypothetical homography matrix is ​​calculated based on the multiple pairs of matching points using mathematical methods, and the reprojection error of all matching point pairs under the current homography matrix is ​​calculated. Points with a reprojection error less than a certain threshold are considered "inliers" (correct matches). If the number of inliers in the current model exceeds the historical best, the reprojection error of the final matching pair is obtained. Optionally, a weighted Levenberg-Marquardt optimization algorithm can also be used to fine-tune the reprojection error of the final matching pair. In the implementation of the above scheme, by using the PROSAC algorithm to perform progressive iterative sampling on the sorted candidate matching pairs, the pre-sorting information can be fully utilized. Starting from high-confidence matching, the algorithm gradually expands and converges to the optimal solution faster than the random sampling method. This ordered sampling method greatly reduces the waste of computing resources and improves matching accuracy and algorithm stability.

[0081] Step S143: Remove matching pairs from the final matching pairs whose reprojection error is greater than a preset error threshold.

[0082] For example, step S143 can be implemented as follows: For each of the multiple final matching pairs, determine whether the reprojection error of the final matching pair is greater than a preset error threshold. If so, remove the final matching pair from the multiple final matches, so that the remaining final matching pairs are all matching pairs with an error less than the preset error threshold. In the implementation of the above scheme, by calculating and removing matching pairs with reprojection errors greater than the threshold, accurate screening of the matching results is achieved, ensuring the geometric consistency of the final matching pairs. This step effectively eliminates mismatches caused by factors such as changes in viewing angle and occlusion, significantly improving the reliability and robustness of the matching results.

[0083] As an optional implementation of step S150 above, for example: after obtaining the final matching pairs, a weighted least squares objective function for the final matching pairs is constructed based on the feature point scale (keypoint.size) and matching quality. After removing matching pairs from the final matching pairs whose reprojection errors are greater than a preset error threshold, the removed matching pairs with reprojection errors greater than the preset error threshold can be obtained, and nonlinear optimization algorithms such as Levenberg-Marquardt can be used to perform nonlinear optimization on these matching pairs. That is, camera parameters are optimized by minimizing the reprojection error. For example, global bundle adjustment is performed on the camera transformation parameters corresponding to multiple images to obtain alignment parameters, and multiple images are transformed according to the alignment parameters to obtain multiple transformed images. It is understood that in the process of optimizing camera parameters by minimizing the reprojection error, a solver can also be configured first, and the configured solver can be used to minimize the reprojection error and optimize the camera parameters.

[0084] As an optional implementation of step S160 above, the above-described implementation of multi-band fusion rendering of multiple transformed images may include: Step S161: Perform multi-band fusion on multiple transformed images to obtain the fused image.

[0085] Understandably, the implementation of step S161 above is relatively complex. Therefore, the implementation of step S161 will be described in detail below. The main method is to use multi-band fusion technology (such as Laplacian pyramid fusion) to decompose the image into detail levels of different scales (such as contours, textures, edges, etc.). Then, weighted fusion is performed at each level according to image weights. Its purpose is to solve the seam problem in image stitching and achieve seamless fusion. In the implementation of the above scheme, multi-band fusion of multiple transformed images can effectively preserve image detail information of different frequency bands, avoiding the loss of detail or blurring caused by single-band processing, thereby significantly improving the overall clarity and detail of the fused image.

[0086] Step S162: Perform dynamic range synthesis and tone mapping on the fused image to obtain a tone-mapped image.

[0087] An example of implementing step S162 above is as follows: The fused image output in S161 is typically a high dynamic range (HDR) image, containing brightness information far exceeding the screen's display capabilities. If the original image is a multi-exposure sequence, the fused image can be dynamically range synthesized into an HDR image. Then, the brightness range of the HDR image is "compressed" to the range that a normal display can show (e.g., 0-255), while preserving highlight and shadow details as much as possible, thereby obtaining a tone-mapped image. In the implementation of the above scheme, global tone mapping of the already fused HDR image ensures a natural and consistent brightness transition throughout the panoramic image. It can adaptively adjust the brightness distribution and color contrast of the image, ensuring balanced representation of both bright and dark areas in high dynamic range scenes, thereby enhancing the visual realism and viewing comfort of the image.

[0088] Step S163: Perform color consistency correction on the tone-mapped image to obtain a panoramic image.

[0089] An example implementation of step S163 above involves performing color consistency correction on the tone-mapped image output in step S162. This can be achieved by adjusting the image's mean and standard deviation to ensure a more uniform color distribution across the entire image, eliminating "color blocks" or "color levels" caused by differences in lighting and white balance during shooting. It is understood that this color correction is a final visual enhancement. Performing color correction before fusion or during the HDR stage may result in distortion due to dynamic range variations. Performing it on an LDR image is more stable, intuitive, and directly optimizes colors perceived by the human eye. In implementing this solution, color consistency correction on the tone-mapped image eliminates color deviations that may occur during multi-image fusion or dynamic range processing, ensuring the overall harmony and naturalness of the final panoramic image's color performance and improving the overall image quality.

[0090] As an optional implementation of step S161 above, the above-described implementation of multi-band fusion of multiple transformed images may include: Step S161a: For each of the multiple transformed images, construct the Laplacian pyramid and weight pyramid corresponding to that transformed image.

[0091] The Laplacian pyramid is a multi-scale image representation method that obtains images of different frequency bands through Gaussian pyramid difference, which can be used to perform multi-band weighted fusion on the transformed image above.

[0092] An example implementation of step S161a above is as follows: For each of the multiple transformed images, a Laplacian pyramid and a weight pyramid corresponding to that transformed image are constructed. The process of constructing the weight pyramid corresponding to the transformed image can be represented in code as: std::vector<st ... <cv::mat>weight_pyramid; buildGaussianPyramid(weights[0], weight_pyramid). The process of constructing the Laplacian pyramid corresponding to the transformed image described above can be represented by the following code: void buildLaplacianPyramid(const cv::Mat&img, std::vector <cv::mat>&pyramid) { cv::Mat current = img.clone(); for (int i = 0; i <levels; ++i) { cv::Mat down, up; cv::pyrDown(current, down); cv::pyrUp(down, up, current.size()); cv::subtract(current, up, pyramid[i]); current = down; } pyramid.push_back(current); / / The final level of the Gaussian pyramid } In implementing the above scheme, by constructing independent Laplacian pyramids and weight pyramids for each transformed image, multi-scale image decomposition can be achieved. This preserves detailed features of different frequency bands during the fusion process, avoiding the detail loss problem caused by single-scale fusion. This multi-scale processing method enables the final fused image to simultaneously maintain high-frequency details and low-frequency structural information, significantly improving the quality of image fusion.

[0093] Step S161b: Perform multi-band weighted fusion of the transformed image based on the Laplacian pyramid and the weight pyramid to obtain the fused image.

[0094] The implementation of step S161b above is as follows: First, the Laplace pyramid and the weighted pyramid are statistically analyzed to obtain the total number of frequency bands and the frequency band weight values ​​obtained from the Laplace pyramid and the weighted pyramid. Then, the total number of frequency bands and the frequency band weight values ​​are weighted and fused to obtain the fusion weight. The above multi-band weighted fusion process can be represented by the following code: cv::Mat blended; for (int l = 0; l <levels; ++l) { cv::Mat sum = cv::Mat::zeros(laplacian_pyramid[l].size(), CV_32F); cv::Mat sum_weights = cv::Mat::zeros(weight_pyramid[l].size(), CV_32F); for (size_t i = 0; i <images.size(); ++i) { cv::Mat img_lap, weight; buildPyramidLevel(images[i], l, img_lap); buildPyramidLevel(weights[i], l, weight); cv::accumulate(img_lap.mul(weight), sum); cv::accumulate(weight, sum_weights); } cv::divide(sum, sum_weights, blended_pyramid[l]); In the implementation of the above scheme, by employing a weighted pyramid for multi-band weighted fusion, the fusion weights can be dynamically adjusted according to the importance of different frequency bands, making important features more prominent in the fusion result. This adaptive weighting strategy overcomes the limitations of traditional fixed-weight fusion methods, better adapting to the feature distribution of different image content, thus producing a more natural and visually perceptual fusion effect. Furthermore, by combining the Laplacian pyramid with the weighted pyramid, a synergistic effect of frequency band separation and weighted fusion is achieved. The Laplacian pyramid provides accurate frequency band decomposition, while the weighted pyramid achieves fine-grained frequency band control. The combination of the two enables the fusion process to achieve optimal balance at each frequency band level, thereby producing a fusion effect that surpasses simple linear superposition.

[0095] Optionally, a fusion weight is obtained by weighting and fusing the total number of frequency bands and their weight values. In the above implementation process, by weighting and fusing the total number of frequency bands and their weight values, the differences in importance of different frequency bands in image reconstruction can be more accurately reflected. This allows for a more reasonable allocation of the contribution of each frequency band during the fusion process, improving the detail preservation and overall quality of the fused image. In multi-band fusion, different frequency bands (such as low-frequency contours and high-frequency textures) contribute differently to the final image quality. By weighting and calculating the total number of frequency bands and the weight value of each frequency band, a comprehensive fusion weight can be obtained, reflecting the importance of information in each frequency band. Using this fusion weight, targeted weighted combinations of transformed images from different sources in each frequency band during the image reconstruction stage can more reasonably preserve clear details and suppress blurred areas, effectively reducing seams and artifacts, ultimately generating a fused image with natural transitions, rich details, and higher visual quality.

[0096] Optionally, after obtaining the fusion weights, the transformed image can be fused and reconstructed using these weights to obtain the fused image. In the implementation of the above scheme, by using fusion weights to reconstruct the transformed image, the weights of each frequency band can be dynamically adjusted. This allows for the preservation of high-frequency details while suppressing noise or redundant information, achieving a more optimized inter-band balance and improving image clarity and information integrity. Furthermore, through the dynamic calculation and adjustment of the fusion weights, the algorithm can adapt to different image features and frequency band distributions, enhancing its adaptability and robustness, and ensuring stable fusion results even in complex scenes.

[0097] Traditional multi-band fusion engines often suffer from seam breaks or content loss in the fused sky due to the sparsity of low-texture features and limitations in shooting angle when processing the sky region. To improve this issue, as an optional implementation of step S161 above, after obtaining the fused image, the following may be included: Step S161c: Detect the sky region and non-sky region from the fused image.

[0098] An example implementation of step S161c above is as follows: Sky and non-sky regions are dynamically detected from the fused image using Lab color space thresholding analysis and superpixel segmentation. This allows for targeted reduction of the weight of damaged areas to suppress error propagation during the pyramid fusion stage. The detection of sky and non-sky regions from the fused image can be represented in code as follows: vector <cv::mat>detectSky(const vector <cv::mat>&imgs) { vector <cv::mat>masks for (const auto&img : imgs) { cv::Mat lab, mask; cv::cvtColor(img, lab, cv::COLOR_BGR2Lab); / / Color threshold + SLIC superpixel segmentation cv::inRange(lab, Scalar(190,115,100), Scalar(255,145,155), mask); slicSuperpixelRefine(mask); / / Boundary optimization masks.push_back(mask); } Return masks; Step S161d: The sky region is fused using the first fusion method to obtain the fused sky region, and the non-sky region is fused using the second fusion method. The fusion weight of the first fusion method is reduced relative to the fusion weight of the second fusion method.

[0099] It is understandable that the implementation of step S161d here is similar to the implementation of steps S161a to S161b above. The difference is that in the implementation of step S161d, the blending weight for blending the sky area is reduced compared to the blending weight for blending the non-sky area. Therefore, it will not be elaborated here. If there is anything unclear, please refer to the implementation of steps S161a to S161b above. In the implementation of the above scheme, by using the first blending method to blend the sky area and reducing the blending weight compared to the second blending method for the non-sky area, the over-blending phenomenon in the sky area can be effectively reduced, avoiding unnatural color transitions or loss of detail in the sky. This differentiated weight setting allows the sky area to remain natural and smooth while the non-sky area can still obtain a sufficient blending effect.

[0100] Step S161e: Repair the merged sky area.

[0101] In the implementation process of the above scheme, by detecting the sky region and the non-sky region from the fused image and processing them respectively using different fusion methods, the characteristics of different regions can be optimized, thereby improving the fusion quality of the overall image. This partition processing method avoids the local distortion problem caused by using uniform fusion parameters for the entire image in traditional methods, so that the sky and non-sky regions can both obtain the best visual effect. Further, by repairing the fused sky region, the defects or noise introduced in the fusion process can be further eliminated, and the purity and realism of the sky region can be improved. This step compensates for the possible deficiencies in the fusion process, making the final image more visually coherent and natural.

[0102] As an optional implementation of the above step S161e, the implementation of repairing the fused sky region can include: Step S161f: determining whether the area value of the fused sky region is less than a preset area threshold.

[0103] The implementation of step S161f, for example, determines whether the area value of the fused sky region is less than a preset area threshold through a conditional statement cv::countNonZero(mask) < img.total() * 0.2. Wherein, cv::countNonZero(mask) represents the area value of the fused sky region, img.total() represents the total area of the original image, and img.total() * 0.2 represents the preset area threshold. The value 0.2 here is a pre-set value, and other values (such as 0.3 or 0.4, etc.) can also be set. In the implementation process of the above scheme, by setting the area threshold as the judgment standard for method selection, the scheme realizes the automatic balance between repair precision and calculation cost. Small area regions usually have low requirements for details and are suitable for fast repair; while large area regions require more detailed processing, and using a generative adversarial network can better maintain the naturalness of texture and structure. This hierarchical processing method significantly improves the visual quality of the repair result.

[0104] Step S161g: if the area value of the fused sky region is less than the preset area threshold, directly repairing the fused sky region using the inpaint function of OpenCV.

[0105] Step S161h: if the area value of the fused sky region is greater than or equal to the preset area threshold, calling a pre-trained SkyGAN generative adversarial network model to repair the fused sky region.

[0106] The SkyGAN generative adversarial network model is a generative adversarial network (GAN) model specially used for sky region repair, which can generate realistic sky images.

[0107] The implementation of the above steps S161g to S161h is, for example: if the area value of the fused sky region is less than the preset area threshold (i.e., cv::countNonZero(mask) < img.total()*0.2), the fused sky region is directly repaired using the OpenCV inpaint function, such as cv::inpaint(img, mask, img, 15, cv::INPAINT_TELEA). If the area value of the fused sky region is greater than or equal to the preset area threshold, a pre-trained SkyGAN generative adversarial network model is called to repair the fused sky region, such as auto sky = SkyGAN::generate(img,mask) ; sky.copyTo(img, mask). In the implementation of the above scheme, by dynamically selecting different repair methods according to the size of the sky region area, the optimal allocation of computing resources is realized. For small area regions, a lightweight OpenCV inpaint function is used to reduce the computational overhead; and for large area regions, a more complex SkyGAN model is used to ensure the repair quality, thereby improving the overall processing efficiency while ensuring the effect. Further, by combining the traditional image processing algorithm (OpenCV inpaint) with the deep learning model (SkyGAN), a hybrid repair strategy is formed. This combination not only takes advantage of the efficiency of traditional algorithms in handling simple tasks, but also utilizes the powerful generation capability of deep learning in complex scenarios, thereby realizing the most suitable repair scheme for sky regions of different sizes.

[0108] See Figure 3 The panoramic image stitching device provided by the embodiment of the application is shown in a structural schematic diagram; the embodiment of the application provides a panoramic image stitching device 200, comprising: An image data acquisition processing module 210 is configured to acquire a plurality of images and metadata of each image, and pre-process the plurality of images according to the metadata to obtain pre-processed images, wherein the plurality of images have partially overlapped fields of view.

[0109] A key point descriptor extraction module 220 is configured to extract image key points and feature descriptors of the image key points with scale and rotation robustness from the pre-processed images by using a SuperPoint neural network model.

[0110] An image nearest neighbor search matching module 230 is configured to perform approximate nearest neighbor search matching on the image key points according to the feature descriptors of the image key points to obtain a plurality of initial matching pairs.

[0111] The initial matching data elimination module 240 is configured to eliminate matching pairs with spatial consistency less than a threshold from the plurality of initial matching pairs by using geometric constraints of a random sample consensus algorithm to obtain final matching pairs.

[0112] The image alignment adjustment transformation module 250 is configured to globally bundle and adjust camera transformation parameters corresponding to the plurality of images by using a nonlinear optimization algorithm based on the final matching pairs to obtain alignment parameters, and transform the plurality of images according to the alignment parameters to obtain a plurality of transformed images.

[0113] The image band fusion rendering module 260 is configured to perform multi-band fusion rendering on the plurality of transformed images to obtain a panoramic image.

[0114] In the implementation process of the above scheme, by combining the SuperPoint neural network model to extract key point descriptors with scale and rotation robustness, the limitations of traditional feature extraction methods in complex scenes can be effectively overcome, and the accuracy and stability of feature matching can be significantly improved, thereby providing more reliable basic data for subsequent image stitching. In addition, by using approximate nearest neighbor search matching combined with geometric constraints of the random sample consensus algorithm, efficient screening of initial matching pairs is realized, which not only ensures matching efficiency but also ensures matching quality, effectively reducing the influence of false matching pairs on the final stitching result. Further, by using a nonlinear optimization algorithm to globally bundle and adjust camera transformation parameters, the geometric relationship between the plurality of images can be systematically optimized, and the image alignment accuracy can be significantly improved, thereby obtaining a more natural and seamless panoramic stitching effect. Finally, by using multi-band fusion rendering technology to process the transformed images, the brightness difference and color discontinuity at the stitching boundary can be effectively eliminated, a smoother and more natural image transition is realized, and the overall visual effect of the panoramic image is improved.

[0115] As an optional implementation of the above device, the SuperPoint neural network model comprises a shared encoder, a detection branch, and a description branch; and the key point descriptor extraction module comprises: The image feature extraction submodule is configured to perform feature extraction on each of the plurality of preprocessed images using the shared encoder to obtain image features.

[0116] The image key point detection submodule is configured to perform image key point detection on the image features using the detection branch to obtain image key points.

[0117] The image descriptor extraction module is configured to perform description map extraction on the image features using the description branch to obtain a dense description map.

[0118] The feature descriptor sampling module is configured to sample the dense description map according to the image key points to obtain feature descriptors of the image key points.

[0119] As an optional implementation of the above device, the image key point detection submodule comprises: The detection feature map obtaining unit is configured to perform image key point detection on the image features using the detection branch to obtain a detection feature map.

[0120] The image key point obtaining unit is configured to convert the detection feature map into a heat map and perform non-maximum suppression on the heat map according to a preset suppression threshold to obtain the image key points.

[0121] As an optional implementation of the above device, the plurality of images comprise: a first image and a second image; and the image neighbor search matching module comprises: The matching result obtaining submodule is configured to: construct index parameters and search parameters of a fast library for approximate nearest neighbors (FLANN) based on the KD tree; construct a matcher of the FLANN through the index parameters and the search parameters, and perform matching on the feature descriptors of the first image and the feature descriptors of the second image through the matcher of the FLANN and a K-nearest neighbor algorithm to obtain a plurality of matching results.

[0122] The initial matching pair obtaining submodule is configured to filter the plurality of matching results to obtain a plurality of initial matching pairs.

[0123] As an optional implementation of the above device, the panoramic image stitching device further comprises: The matching pair pre-filtering and sorting module is configured to pre-filter and sort the plurality of initial matching pairs according to the confidence of the image key points and the similarity between the feature descriptors to generate sorted candidate matching pairs.

[0124] The iterative sampling error calculation module is configured to calculate the re-projection error of the final matching pair by using a progressive Proportional Resampling (PROSAC) algorithm to progressively and iteratively sample the sorted candidate matching pairs.

[0125] The final matching pair elimination module is configured to eliminate, from the final matching pair, a matching pair whose re-projection error is greater than a preset error threshold.

[0126] As an optional implementation of the above device, the image band fusion rendering module comprises: The fusion image obtaining submodule is configured to perform multi-band fusion on the plurality of transformed images to obtain a fused image.

[0127] The synthetic tone mapping submodule is configured to perform dynamic range synthesis and tone mapping on the fused image to obtain a tone-mapped image.

[0128] The panoramic image obtaining submodule is configured to perform color consistency correction on the tone-mapped image to obtain a panoramic image.

[0129] As an optional implementation of the above-mentioned device, the image fusion acquisition submodule includes: The pyramid building unit is used to construct the Laplacian pyramid and weight pyramid corresponding to each of the multiple transformed images.

[0130] The multi-band fusion unit is used to perform multi-band weighted fusion of the transformed image based on the Laplacian pyramid and the weight pyramid.

[0131] As an optional implementation of the above-mentioned device, the image frequency band fusion rendering module further includes: The image region detection submodule is used to detect sky regions and non-sky regions from the fused image.

[0132] The image region fusion submodule is used to fuse the sky region using a first fusion method to obtain the fused sky region, and to fuse the non-sky region using a second fusion method. The fusion weight of the first fusion method is reduced relative to the fusion weight of the second fusion method.

[0133] The Sky Region Repair Submodule is used to repair the merged sky region.

[0134] As an optional implementation of the above-mentioned device, the sky area repair submodule includes: The area determination unit is used to determine whether the area value of the merged sky area is less than a preset area threshold.

[0135] The sky region repair unit is used to directly repair the fused sky region using OpenCV's inpaint function if the area value of the fused sky region is less than a preset area threshold, and to call the pre-trained SkyGAN generative adversarial network model to repair the fused sky region if the area value of the fused sky region is greater than or equal to the preset area threshold.

[0136] It should be understood that this device corresponds to the panoramic image stitching method embodiment described above and is capable of performing the various steps involved in the above method embodiment. The specific functions of this device can be found in the description above, and detailed descriptions are appropriately omitted here. The device includes at least one software functional module that can be stored in memory or embedded in the device's operating system (OS) in the form of software or firmware.

[0137] Please see Figure 4 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 300 provided in this application includes a processor 310 and a memory 320. The memory 320 stores machine-readable instructions executable by the processor 310. When the machine-readable instructions are executed by the processor 310, the method described above is performed.

[0138] This application embodiment also provides a computer-readable storage medium 330, on which a computer program is stored. This computer program is executed by a processor 310 to perform the methods described above. The computer-readable storage medium 330 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0139] This application also provides a computer program product, including: a computer program or computer instructions, which are executed by a processor to perform the method described above.

[0140] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0141] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, as provided in the embodiments of this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending primarily on the functions involved.

[0142] Furthermore, the functional modules of each embodiment in this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," "some examples," etc., means that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0143] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.< / cv::mat> < / cv::mat> < / cv::mat> < / cv::mat> < / cv::mat> < / cv::flann::searchparams> < / cv::flann::kdtreeindexparams>

Claims

1. A panoramic image stitching method, characterized in that, include: Multiple images and metadata of each image are acquired, and the multiple images are preprocessed according to the metadata to obtain preprocessed multiple images, wherein the multiple images have partially overlapping fields of view. The SuperPoint neural network model is used to extract scale- and rotation-robust image keypoints and feature descriptors of the image keypoints from the preprocessed multiple images. Based on the feature descriptors of the key points in the image, an approximate nearest neighbor search is performed on the key points to obtain multiple initial matching pairs. By using the geometric constraints of the random sampling consensus algorithm, matching pairs with spatial consistency less than a threshold are removed from the multiple initial matching pairs to obtain the final matching pairs; Based on the final matching pair, the camera transformation parameters corresponding to the multiple images are globally bundled and adjusted using a nonlinear optimization algorithm to obtain alignment parameters. The multiple images are then transformed according to the alignment parameters to obtain multiple transformed images. Multi-band fusion rendering is performed on the multiple transformed images to obtain a panoramic image.

2. The method according to claim 1, characterized in that, The SuperPoint neural network model includes: a shared encoder, a detection branch, and a description branch; the extraction of scale- and rotation-robust image keypoints and feature descriptors of the image keypoints from the preprocessed multiple images using the SuperPoint neural network model includes: For each of the preprocessed images, the shared encoder is used to extract features from the image to obtain image features; The image key points are obtained by using the detection branch to detect image key points in the image features. The image features are extracted using the description branch to obtain a dense description map; The dense descriptive map is sampled based on the image key points to obtain the feature descriptors of the image key points.

3. The method according to claim 2, characterized in that, The step of using the detection branch to detect key points in the image features includes: The detection branch is used to detect key points in the image features to obtain a detection feature map. The detected feature map is converted into a heatmap, and non-maximum suppression is applied to the heatmap according to a preset suppression threshold to obtain the key points of the image.

4. The method according to claim 1, characterized in that, The multiple images include: a first image and a second image; the step of performing approximate nearest neighbor search matching on the key points of the images based on the feature descriptors of the key points includes: The index parameters and search parameters of the approximate nearest neighbor fast library FLANN are constructed based on KD-trees; The FLANN matcher is constructed using the index parameters and the search parameters, and the feature descriptors of the first image and the second image are matched using the FLANN matcher and the K-nearest neighbor algorithm to obtain multiple matching results. The multiple matching results are filtered to obtain the multiple initial matching pairs.

5. The method according to claim 1, characterized in that, After obtaining the final matching pair, the process also includes: Based on the confidence level of the image key points and the similarity between the feature descriptors, the multiple initial matching pairs are pre-screened and sorted to generate sorted candidate matching pairs; The PROSAC algorithm is used to perform progressive iterative sampling on the sorted candidate matching pairs in order to calculate the reprojection error of the final matching pair. Remove matching pairs from the final matching pairs whose reprojection error is greater than a preset error threshold.

6. The method according to claim 1, characterized in that, The multi-band fusion rendering of the multiple transformed images includes: Multi-band fusion is performed on the multiple transformed images to obtain the fused image; The fused image is subjected to dynamic range synthesis and tone mapping to obtain a tone-mapped image. The panorama image is obtained by performing color consistency correction on the tone-mapped image.

7. The method according to claim 6, characterized in that, The multi-band fusion of the multiple transformed images includes: For each of the multiple transformed images, construct the Laplacian pyramid and weight pyramid corresponding to that transformed image; The transformed image is then subjected to multi-band weighted fusion based on the Laplacian pyramid and the weight pyramid.

8. The method according to claim 6, characterized in that, After obtaining the fused image, the process also includes: Detect sky regions and non-sky regions from the fused image; The sky region is fused using a first fusion method to obtain a fused sky region, and the non-sky region is fused using a second fusion method. The fusion weight of the first fusion method is reduced relative to the fusion weight of the second fusion method. Determine whether the area value of the merged sky region is less than a preset area threshold; If so, the OpenCV inpaint function is used to directly repair the fused sky region; otherwise, the pre-trained SkyGAN generative adversarial network model is called to repair the fused sky region.

9. A panoramic image stitching device, characterized in that, include: The image data acquisition and processing module is used to acquire multiple images and metadata of each image, and preprocess the multiple images according to the metadata to obtain preprocessed multiple images, wherein the multiple images have partially overlapping fields of view. The key point descriptor extraction module is used to extract scale- and rotation-robust image key points and feature descriptors of the image key points from the preprocessed multiple images using the SuperPoint neural network model. The image nearest neighbor search and matching module is used to perform approximate nearest neighbor search and matching on the image key points according to the feature descriptors of the image key points to obtain multiple initial matching pairs; The initial matching data elimination module is used to eliminate matching pairs with spatial consistency less than a threshold from the multiple initial matching pairs by using the geometric constraints of the random sampling consensus algorithm, so as to obtain the final matching pairs; The image alignment adjustment and transformation module is used to perform global bundle adjustment of the camera transformation parameters corresponding to the multiple images based on the final matching pair using a nonlinear optimization algorithm to obtain alignment parameters, and transform the multiple images according to the alignment parameters to obtain multiple transformed images. The image frequency band fusion rendering module is used to perform multi-frequency band fusion rendering on the multiple transformed images to obtain a panoramic image.

10. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, the machine-readable instructions being executed by the processor to perform the method of any one of claims 1 to 8.

Images