Multi-image panoramic stitching method based on depth matching, camera and storage medium
Through deep learning end-to-end matching and multi-image global consistency screening, combined with camera parameter optimization, the robustness and computational efficiency issues of traditional panoramic stitching methods in complex scenes are solved, and efficient and seamless panoramic image generation is achieved.
Patent Information
- Application Number
- CN202510693601.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-27
- Publication Date
- 2025-09-16
AI Technical Summary
Traditional panoramic stitching methods are not robust enough when dealing with low-texture areas, lighting changes or complex scenes, and have high computational complexity. In addition, existing deep learning methods lack global optimization and efficient stitching processes for multiple image sequences, making it difficult to meet real-time requirements.
An end-to-end matching method based on deep learning is adopted, combined with multi-image global consistency screening and camera parameter optimization. Feature matching and image fusion are performed through a pre-trained ONNX model. Dynamic connectivity graph analysis and confidence threshold optimization are used to achieve efficient image stitching.
It significantly improves matching accuracy and computational efficiency, ensures global consistency of stitching results, and is suitable for ultra-high-resolution image sequences to generate seamless, wide-field panoramic images.
Smart Images
Figure CN120655503A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision image processing, and specifically relates to a multi-image panoramic stitching method based on depth matching, a camera, and a storage medium. Background Art
[0002] Panoramic image stitching technology is widely used in photography, virtual reality, remote sensing imaging and other fields. Traditional panoramic stitching methods usually rely on feature point detection and matching (such as SIFT, ORB) and geometric transformation estimation (such as homography transformation or bundle adjustment). However, when dealing with low-texture areas (such as starry sky images), illumination changes or complex scenes, traditional methods lack robustness in feature point matching, which can easily lead to stitching failures or obvious seams. In addition, traditional methods rely on manually designed feature extractors, which are difficult to adapt to diverse image content and have high computational complexity.
[0003] In recent years, deep learning technology has made significant progress in the field of image matching. End-to-end matching methods based on convolutional neural networks (CNNs) or Transformers can extract more robust features and achieve higher matching accuracy in complex scenarios. However, existing deep learning matching methods mostly focus on matching single pairs of images and lack the integration of global optimization and efficient stitching processes for multi-image sequences. In addition, existing methods require high computational resources when processing ultra-high-resolution images or large-scale image sequences, making it difficult to meet real-time requirements.
[0004] Therefore, there is an urgent need for a method that combines deep learning matching, multi-image global optimization and efficient stitching process to achieve robust, accurate and efficient panoramic image generation. Summary of the Invention
[0005] In order to address the shortcomings of the existing technology, the present invention provides a multi-image panoramic stitching method based on depth matching, a camera and a storage medium. By innovatively integrating deep learning end-to-end matching, multi-image global consistency screening, camera parameter optimization and efficient image fusion, the problems of insufficient robustness, obvious seams and low computational efficiency of traditional methods in complex scenes are solved.
[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0007] The multi-image panoramic stitching method based on depth matching includes the following steps:
[0008] S1. Image preprocessing and multi-scale scaling: First, multiple image sequences are input. After the multiple image sequences are input, the input multiple images are adaptively scaled according to the customized working resolution, seam resolution, and synthesis resolution to generate a multi-scale image set;
[0009] S2. Deep Learning End-to-End Matching: Use a pre-trained ONNX model to perform end-to-end feature matching on a multi-scale image collection, generating direct matching information between image pairs.
[0010] S3, global consistency screening: by constructing a connectivity graph for image matching, the largest connected subgraph belonging to the same panorama is screened based on the confidence threshold;
[0011] S4, Camera parameter estimation and optimization: homography estimation and bundle adjustment optimization based on depth matching results;
[0012] S5. Exposure compensation and image fusion: Exposure consistency correction is performed on the deformed image through block gain compensation, and then multi-band fusion or feathering fusion is used to generate the final panoramic image.
[0013] Preferably, the multi-scale of the multi-scale image set in S1 includes a working scale, a seam scale, and a synthesis scale, wherein the working scale formula is:
[0014] work scale =min(1.0,sqrt(work megapix ×1e6 / img area ));
[0015] The seam size formula is:
[0016] seam scale =min(1.0,sqrt(seam megapix ×1e6 / img area ));
[0017] The composite scale formula is:
[0018] compose scale =min(1.0,sqrt(compose megapix ×1e6 / img area )).
[0019] Preferably, the deep learning end-to-end matching in S2 includes the following sub-steps:
[0020] S2.1. Feature Extraction: Extract dense feature descriptors from multi-scale image collections using a deep convolutional network.
[0021] S2.2. Matching calculation: Compute matching point pairs based on feature similarity and filter out false matches through geometric consistency verification.
[0022] S2.3. Confidence evaluation: Each pair of matches is assigned a confidence score, which generates direct matching information between the image pairs.
[0023] Preferably, the global consistency screening in step S3 includes the following sub-steps:
[0024] S3.1. Construct a connectivity graph: use images as nodes, matching relationships as edges, and edge weights as matching confidences.
[0025] S3.2. Filter subgraphs: Use depth-first search or breadth-first search to retain the largest connected subgraphs with confidence scores above a threshold.
[0026] S3.3. Update the image set: only retain the images belonging to the largest connected subgraph and their matching relationships, that is, filter out the largest connected subgraph belonging to the same panorama based on the confidence threshold.
[0027] Preferably, in the camera parameter estimation and optimization step S4, the initial camera parameters of each image are calculated using a homography transformation estimator, and then the camera parameters are globally optimized using a bundle adjustment method.
[0028] The present invention also discloses a camera, comprising a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the multi-image panoramic stitching method based on depth matching are implemented.
[0029] The present invention also discloses a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned multi-image panoramic stitching method based on depth matching are implemented.
[0030] By adopting the above technical solution, the present invention has the following beneficial effects:
[0031] (1) This paper deeply integrates the deep learning end-to-end matching method with the multi-image panoramic stitching process, and proposes an efficient feature matching framework based on the ONNX model. The ONNX model supports GPU acceleration, which can significantly improve matching accuracy and computational efficiency.
[0032] (2) The present invention designs a multi-image global consistency screening algorithm, which automatically eliminates inconsistent image pairs through dynamic connectivity graph analysis and confidence threshold optimization to ensure the global consistency of the splicing results;
[0033] (3) This paper proposes an adaptive multi-scale image processing and stitching strategy. By dynamically adjusting the working resolution, seam resolution, and synthesis resolution, it balances computational efficiency and stitching quality. This strategy is applicable to ultra-high-resolution image sequences and addresses the problems of traditional methods such as insufficient robustness, obvious seams, and low computational efficiency in complex scenes.
[0034] In summary, the present invention has the advantages of significantly improving matching accuracy and computational efficiency, ensuring global consistency of stitching results, balancing computational efficiency and stitching quality, and being suitable for ultra-high-resolution image sequences. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a schematic flow diagram of the present invention;
[0036] Figure 2 This is an example of the panoramic output of the present invention. DETAILED DESCRIPTION
[0037] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0038] The components of the embodiments of the present invention generally described and shown in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention.
[0039] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.
[0040] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0041] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.
[0042] Example 1
[0043] In this embodiment, a multi-image panoramic stitching method, camera and storage medium based on depth matching are proposed. Its depth matching algorithm has higher adaptability to complex scenes such as low texture and lighting changes, and multi-scale processing and GPU acceleration significantly reduce the calculation time, which is suitable for real-time or near real-time applications. More specifically, the global consistency screening and optimization algorithm ensures seamless stitching and global geometric consistency, supports multiple deformation, seam detection and fusion strategies, and adapts to the requirements of different scenarios.
[0044] like Figure 1 As shown, in one embodiment of the present invention, the multi-image panoramic stitching method based on depth matching of the present invention includes the following steps:
[0045] S1. Image preprocessing and multi-scale scaling: First, input multiple image sequences. After inputting multiple image sequences, according to the custom (that is, user-defined) working resolution (work megapix ), seam resolution (seam megapix ) and composite resolution (compose megapix ), adaptively scale the input multiple images to generate a multi-scale image set. Specifically, the multi-scale of the multi-scale image set includes working scale, seam scale and synthesis scale, where the working scale formula is:
[0046] work scale =min(1.0,sqrt(work megapix ×1e6 / img area ));
[0047] The seam size formula is:
[0048] seam scale =min(1.0,sqrt(seam megapix ×1e6 / img area ));
[0049] The composite scale formula is:
[0050] compose scale =min(1.0,sqrt(compose megapix ×1e6 / img area ));
[0051] Through the above multi-scale processing, the computational complexity of subsequent feature extraction and matching is reduced, while retaining sufficient high-frequency details for seam detection and image fusion;
[0052] S2. End-to-end matching of deep learning: Use pre-trained ONNX models (support CPU / GPU inference) to perform end-to-end feature matching on multi-scale image sets and generate direct matching information (DirectMatchesInfo) between image pairs. Pre-trained ONNX models refer to models that have been trained on large-scale datasets and stored in ONNX format. ONNX model (Open Neural Network Exchange) is an open standard format for representing deep learning models. It allows different deep learning frameworks (such as PyTorch, TensorFlow, Keras, etc.) to export models to a unified ONNX format, so that these models can be exchanged and shared between different frameworks or platforms, and can be loaded and run by various inference engines or tools that support ONNX. ONNX models contain the structural information of the model (such as the number of layers and node connections of the neural network) and the parameters of the model (weights and biases, etc.). Pre-training refers to training the model on a large-scale general dataset before applying it to specific tasks. These large-scale datasets usually contain rich image, text, speech and other data. By training on these datasets, the model The model can learn some common features and patterns. For example, in the image field, the pre-trained model may have learned features such as edges, textures, and basic shapes of objects in the image. The advantage of pre-training is that it can utilize the rich information contained in large-scale data, reduce the amount of data and computing resources required to train the model on a specific task, and improve the training efficiency and performance of the model. The pre-trained ONNX model combines the versatility of the ONNX format and the advantages of the pre-trained model, making it easy to deploy and fine-tune in different scenarios and tasks to meet the needs of various practical applications. For example, in computer vision tasks, the pre-trained ONNX image classification model can be used as the basis, and then the model can be fine-tuned according to the specific image classification task. This can greatly reduce the training time and data volume while obtaining better classification performance.
[0053] The deep learning end-to-end matching in S2 includes the following sub-steps:
[0054] S2.1. Feature Extraction: Extract dense feature descriptors from multi-scale image collections using a deep convolutional network.
[0055] S2.2. Matching calculation: Compute matching point pairs based on feature similarity and filter out false matches through geometric consistency verification (e.g., RANSAC).
[0056] S2.3. Confidence Assessment: Each matching pair is assigned a confidence score, generating direct matching information between the image pairs for subsequent global optimization. This innovative approach seamlessly integrates ONNX Runtime with OpenCV, supporting CUDA acceleration, significantly improving matching speed and making it suitable for large-scale image sequences.
[0057] S3, global consistency screening: by building a connectivity graph for image matching, based on the confidence threshold screening (conf thresh ) Select the largest connected subgraph belonging to the same panorama;
[0058] The global consistency screening in step S3 includes the following sub-steps:
[0059] S3.1. Construct a connectivity graph: use images as nodes, matching relationships as edges, and edge weights as matching confidences.
[0060] S3.2. Filter subgraphs: Use depth-first search (DFS) or breadth-first search (BFS) to retain the largest connected subgraph with a confidence score higher than a threshold.
[0061] S3.3. Update the image set: Only retain images belonging to the maximum connected subgraph and their matching relationships. In other words, filter out the maximum connected subgraph belonging to the same panorama based on the confidence threshold.
[0062] This step (S3) effectively eliminates inconsistent image pairs (e.g., images of different scenes), improving the robustness of the stitching;
[0063] S4, Camera parameter estimation and optimization: homography estimation and bundle adjustment optimization based on depth matching results;
[0064] In the step S4, the camera parameter estimation and optimization uses a homography transformation estimator to calculate the initial camera parameters (intrinsic parameter matrix K and rotation matrix R) of each image, and then globally optimizes the camera parameters through the bundle adjustment method (BundleAdjusterReproj), supports user-defined optimization masks, and flexibly adjusts the degree of freedom of optimization;
[0065] S5. Exposure Compensation and Image Fusion: GainBlocks is used to correct the exposure consistency of the deformed image. MultiBandBlender or FeatherBlender is then used to generate the final panoramic image. Fusion parameters (such as the number of bands and feathering sharpness) are adaptively adjusted according to the image size.
[0066] The present invention also discloses a camera, comprising a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the multi-image panoramic stitching method based on depth matching are implemented.
[0067] The present invention also discloses a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned multi-image panoramic stitching method based on depth matching are implemented.
[0068] Please refer to the specific Figure 2 , which is an example of the final panoramic image output by the present invention. The present invention solves the problems of insufficient robustness, obvious seams and low computational efficiency of traditional methods in complex scenes by innovatively integrating deep learning end-to-end matching, multi-image global consistency screening, camera parameter optimization and efficient image fusion. The present invention deeply integrates the deep learning end-to-end matching method with the multi-image panoramic stitching process for the first time, and proposes an efficient feature matching framework based on the ONNX model, which supports GPU acceleration and significantly improves matching accuracy and computational efficiency. The present invention designs a multi-image global consistency screening algorithm, which automatically eliminates inconsistent image pairs through dynamic connectivity graph analysis and confidence threshold optimization to ensure the global consistency of the stitching results. The present invention proposes an adaptive multi-scale image processing and stitching strategy, which balances computational efficiency and stitching quality through dynamic adjustment of working resolution, seam resolution and synthetic resolution, and is suitable for ultra-high-resolution image sequences. The present invention is suitable for the automatic stitching of high-resolution image sequences (such as star photography) to generate seamless, wide-field panoramic images.
[0069] This embodiment does not impose any formal restrictions on the shape, material, structure, etc. of the present invention. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are within the scope of protection of the technical solution of the present invention.
Claims
1. A multi-image panoramic stitching method based on depth matching, characterized in that: The following steps are involved: S1. Image preprocessing and multi-scale scaling: First, multiple image sequences are input. After the multiple image sequences are input, the input multiple images are adaptively scaled according to the customized working resolution, seam resolution, and synthesis resolution to generate a multi-scale image set; S2. Deep Learning End-to-End Matching: Use a pre-trained ONNX model to perform end-to-end feature matching on a multi-scale image collection, generating direct matching information between image pairs. S3, global consistency screening: by constructing a connectivity graph for image matching, the largest connected subgraph belonging to the same panorama is screened based on the confidence threshold; S4, Camera parameter estimation and optimization: homography estimation and bundle adjustment optimization based on depth matching results; S5. Exposure compensation and image fusion: Exposure consistency correction is performed on the deformed image through block gain compensation, and then multi-band fusion or feathering fusion is used to generate the final panoramic image.
2. The multi-image panoramic stitching method based on depth matching according to claim 1, characterized in that: The multi-scale of the multi-scale image set in S1 includes working scale, seam scale and synthesis scale, wherein the working scale formula is: work scale =min(1.0,sqrt(work megapix ×1e6 / img area )); The seam size formula is: seam scale =min(1.0,sqrt(seam megapix ×1e6 / img area )); The composite scale formula is: compose scale =min(1.0,sqrt(compose megapix ×1e6 / img area ))。 3. The multi-image panoramic stitching method based on depth matching according to claim 1, characterized in that: The deep learning end-to-end matching in S2 includes the following sub-steps: S2.
1. Feature Extraction: Extract dense feature descriptors from multi-scale image collections using a deep convolutional network. S2.
2. Matching calculation: Compute matching point pairs based on feature similarity and filter out false matches through geometric consistency verification. S2.
3. Confidence evaluation: Each pair of matches is assigned a confidence score, which generates direct matching information between the image pairs.
4. The multi-image panoramic stitching method based on depth matching according to claim 1, characterized in that: The global consistency screening in step S3 includes the following sub-steps: S3.
1. Construct a connectivity graph: use images as nodes, matching relationships as edges, and edge weights as matching confidences. S3.
2. Filter subgraphs: Use depth-first search or breadth-first search to retain the largest connected subgraphs with confidence scores above a threshold. S3.
3. Update the image set: only retain the images belonging to the largest connected subgraph and their matching relationships, that is, filter out the largest connected subgraph belonging to the same panorama based on the confidence threshold.
5. The multi-image panoramic stitching method based on depth matching according to claim 1, characterized in that: In the camera parameter estimation and optimization step S4, the homography transformation estimator is used to calculate the initial camera parameters of each image, and then the camera parameters are globally optimized by the bundle adjustment method.
6. Camera, characterized by: The method comprises a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the multi-image panoramic stitching method based on depth matching according to any one of claims 1 to 5 are implemented.
7. A storage medium having a computer program stored thereon, characterized in that When the computer program is executed by a processor, the steps of the multi-image panoramic stitching method based on depth matching according to any one of claims 1 to 5 are implemented.