Image panorama stitching method and system

Abstract

This invention belongs to the field of image stitching technology, specifically a method and system for panoramic image stitching. The system simultaneously receives multiple parallel images in a sequence, such as infrared or visible light images; performs necessary preprocessing operations on each acquired image; effectively handles occasional frame drops and errors in the upstream data transmission to ensure stable system operation; performs image registration on the preprocessed multi-band images to obtain the homography transformation relationship between adjacent images; optimizes the homography transformation relationship by minimizing reprojection error; and based on the optimized homography transformation relationship, fuses adjacent images to eliminate obvious stitching seams, ultimately obtaining an excellent multi-band panoramic image without artifacts, and enabling fine-grained local backtracking. This method can achieve stable parallel panoramic stitching of multi-band image sequences.

Description

Technical Field

[0001] This invention belongs to the field of image stitching technology, specifically relating to a method and system for panoramic image stitching. Background Technology

[0002] Image stitching technology is widely used in border surveillance, air-to-ground collaborative reconnaissance, low-altitude UAV patrols, battlefield situational awareness, and remote sensing mapping. Its core objective is to stitch multiple images together into a panoramic image with a wider field of view and more continuous content, thereby expanding visual coverage and improving information representation. Especially in multi-band imaging tasks involving both visible and infrared images, panoramic image stitching not only enables large-scale surveillance but also provides a stable spatial foundation for subsequent applications such as target recognition, intelligent analysis, and mission backtracking. Therefore, constructing high-precision, highly robust multi-band image stitching systems has become a key research and engineering focus in recent years.

[0003] Existing image stitching systems primarily focus on image registration and fusion, often assuming stable and consistent image sources. However, in real-world applications, multi-band image acquisition systems commonly suffer from frame rate discrepancies, transmission asynchrony, misaligned frames, frame drops, and even missing entire rows of image data, leading to stitching link interruptions, image jumps, or severe distortion. Furthermore, due to significant differences in resolution, noise levels, and brightness distribution between infrared and visible light images, traditional stitching algorithms struggle to achieve robust and reliable registration when dealing with different images, frequently resulting in registration failures, image misalignment, and stitching artifacts.

[0004] On the other hand, in dynamic scenes with a wide field of view, the perspective differences between images may be significant due to the constantly changing shooting angle. This can even lead to sudden drops in image quality, missing targets, or partial occlusion. The traditional "single feature + one-time registration" strategy lacks the adaptability to handle complex situations. Furthermore, existing stitching systems are typically designed as linear processes, lacking a closed-loop perception mechanism for image quality and stitching status. If registration failure or fusion anomalies occur during stitching, it is difficult to detect and repair them in a timely manner, severely impacting system stability and the usability of the stitched images.

[0005] Furthermore, after stitching is completed, users often need to perform high-resolution backtracking and fine registration on any region of the image for magnified viewing of key targets or subsequent intelligent recognition. However, most existing stitching systems do not save the mapping information between the image and the stitching relationship, making it impossible to achieve accurate traceability "from panoramic image to original image," resulting in low information utilization and difficulty in supporting the high requirements of detail and backtracking capabilities in actual tasks.

[0006] In addition, existing stitching systems generally lack a quantitative evaluation and feedback mechanism for the quality of stitched images, cannot perform real-time self-optimization based on the image fusion effect, and are difficult to achieve unified measurement and comparison of stitching quality, which limits the practicality and controllability of the system in automated monitoring and engineering deployment.

[0007] Therefore, existing multi-band image sequence stitching methods still suffer from problems such as poor stitching robustness, large errors, and susceptibility to failure. Improving the system's adaptability in complex environments, enhancing its ability to perceive and correct image errors, and achieving improved stitching accuracy and stability while maintaining stitching efficiency are of significant research importance for constructing a high-quality panoramic stitching system suitable for multiple application scenarios. Summary of the Invention

[0008] The purpose of this invention is to provide an image panoramic stitching method and system, which improves the accuracy and stability of stitching and ensures the continuous and stable operation of the system by sensing and responding to errors such as misaligned frames, missing frames, and missing lines.

[0009] To achieve the above objectives, according to a first aspect of the present invention, an image panoramic stitching method is provided, comprising the following steps: S1, sequentially acquire images of the target area to obtain several rows of images. Each row of images consists of several frames of images. Preprocess each frame of images. S2, sequentially determine whether each row of the preprocessed image has an intra-row sporadic error or an inter-row persistent error. If a row of the image does not have either of these conditions, proceed to step S3. If a row of the image has an intra-row sporadic error, proceed to step S4. If a row of the image has an inter-row persistent error, proceed to step S5. S3. Based on prior information, obtain the adjacency map of the preprocessed image, and perform intra-row image registration for adjacent images belonging to the same row to obtain the position transformation relationship between adjacent images in the row. Then, perform intra-row image fusion to eliminate the stitching seams of the intra-row images and obtain a complete panoramic image of a row. S4. When an occasional frame misalignment error occurs in a row of images, the misaligned frame image is discarded and the corresponding position is filled with black pixels; when an occasional frame drop error occurs in a row of images, the corresponding position is filled with black pixels; then the images are registered and fused in the row to obtain a complete panoramic image of a row. S5. When a persistent inter-line error occurs, the missing line position is left blank according to the prior information, and the entire line is filled with black pixels to obtain a panoramic view of a complete line filled with black pixels. S6. Perform interline image registration on all the inline panoramic images obtained in steps S3 to S5 to obtain the positional transformation relationship between adjacent images in the interline. S7. Calculate the reprojection error of the image that has been registered in step S6, solve the optimization problem to minimize the reprojection error between images, and optimize the registration parameters of each frame image. S8. Perform interline image fusion on the image optimized in step S7 to eliminate the interline image stitching seams and obtain a complete panoramic image.

[0010] This invention can improve the anti-interference ability, stability and accuracy of panoramic stitching methods in multi-band image sequence scenarios, reduce stitching errors or crashes, and ensure efficient stitching of any number of images of various types in various complex environments.

[0011] Furthermore, in step S1, the preprocessing includes: cropping the effective region of the image, reducing the resolution of the cropped effective image, smoothing and denoising the image after reduction in resolution, eliminating vignetting in the image after smoothing and denoising, and finally correcting distortion.

[0012] To unify the image sizes of different band resolutions, invalid black areas generally exist around low-resolution images. These invalid black areas are eliminated during preprocessing. The rate of reduction is used to reduce the computational load of subsequent stitching. Gaussian blur is used to remove Gaussian blur noise from the image. Vignetting is achieved by overlaying a reverse template of the camera's empty scene imaging on the existing image to compensate for uneven imaging at the image edges. Distortion correction uses pre-labeled camera parameters to correct radial and tangential distortion in the obtained image. Finally, a preprocessed image is obtained that is more suitable for subsequent stitching.

[0013] Furthermore, the system should not completely trust upstream data, as there may be sporadic or persistent errors. Errors may occur at any location, including the beginning, end, within, and during line breaks of a line. In terms of quantity, any number of misaligned or missing frames may occur, and boundary conditions need to be fully considered. When sporadic errors occur, different handling methods are used for different situations. In step S4, for a valid frame in a certain row of images where the previous frame has a misaligned frame (which can be detected using some verification methods during the data decoding stage) or a missing frame, the registration coordinates of this valid frame are estimated using prior information. For a frame error or missing frame that occurs after the first valid frame in a row of images, the X-coordinate of the registration information of this frame error or missing frame remains the same as the previous frame next to it, and the Y-coordinate is filled with the relative position offset between the previous frame and the frame before that plus the coordinate of the previous frame; where X-direction is the parallel direction of the row, and Y-direction is the acquisition direction of several frames in a row of images. For a frame error or missing frame that occurs before the first valid frame in a row of images, the registration information of this frame error or missing frame is filled with the registration information of the corresponding position in the previous row of images plus the registration information of the first valid frame in that row. For a valid frame in a certain row of images where the previous frame has not been misaligned or lost, this valid frame is directly registered in the normal inline image.

[0014] Furthermore, when the system cannot accept a valid complete line of data, it automatically stops waiting. When the system resumes normal reception, it resumes normal operation. However, there may be line drops between different lines of panoramic images obtained through normal reception and use in S5, resulting in no overlap between two lines. Therefore, in step S6, when line drops occur, all intra-line panoramic images obtained in steps S3 to S5 are logically rotated by 90 degrees, and each line is treated as a single frame image. When line drops occur, it is equivalent to frame drops. Then, the frame drop processing method in step S4 is used to process the line drops, thus realizing the line drop processing of inter-line images.

[0015] Furthermore, the intra-line and inter-line image registration includes pixel-level matching and feature-level matching methods. Different pixel-level matching methods have different time consumption and effects, and different feature-level matching methods also have different time consumption and effects. For stitched images and scenes of different qualities, different methods should be selected for image registration: when the images are related by translation transformation and have high real-time requirements, such as in online stitching scenarios, different pixel-level matching methods should be selected first; when the images are related by affine transformation or perspective transformation, have high image quality, rich texture details, and relatively low real-time requirements, such as in online stitching scenarios, traditional feature-level matching image registration methods should be selected.

[0016] When there is a translation transformation relationship between images, the template matching method is used to perform image registration based on pixel-level information, and the position offset coordinates in the X and Y directions of two adjacent images or two rows of images are calculated. When the relationship between images is an affine transformation or perspective transformation, the feature matching method is used to perform image registration based on feature-level information, and the homography transformation matrix H between two adjacent images or two rows of images is calculated.

[0017] When images are related by affine or perspective transformations, have low image quality, and have limited texture details, an image registration method based on deep neural network inference feature matching can be chosen. In this case, time consumption is not a concern, such as in offline stitching scenarios. If time consumption is a concern, it is necessary to consider whether the hardware platform supports deep learning acceleration. If devices such as FPGAs, ASICs, and GPUs can accelerate deep learning methods, then online stitching scenarios can also be used.

[0018] Furthermore, when using pixel-level matching for image registration, the template and search area are selected based on the fluctuation range of the overlapping region or based on salient region extraction. When using template matching for image registration, the registration effect is highly dependent on the selected template and search area: for images with high quality, relatively rich texture details, and high real-time requirements, the strategy of selecting the template and search area based on the fluctuation range of the overlapping region is adopted; while for images with relatively average quality and relatively low real-time requirements, the strategy of selecting the template and search area based on salient region extraction is adopted.

[0019] The selection of template and search area based on the fluctuation range of the overlapping region includes: In real-world scenarios, during image registration, a certain degree of overlap exists between images, and this overlap has a certain fluctuation range. For intra-line registration, the size and position of the template region and search region are determined according to the overlap ratio between intra-line images. Specifically: the size of the search region is set to a certain proportion of the reference image size, which is greater than or equal to the estimated overlap ratio; the size of the template region is set to match the search region but smaller than the search region to improve matching efficiency and positioning accuracy; the positions of the template and search region are set within the potential overlapping area between adjacent images to enhance the robustness of feature matching.

[0020] For interline registration, the aspect ratio and specific position of the search area and template area are dynamically adjusted during the stitching process of a line of images, based on the overlapping relationship between the interline images. The specific position can be an area in the image that overlaps with the previous or next line of images, but is not limited to this.

[0021] The selection of template and search area based on salient region extraction includes: when selecting the template area and search area for image registration, firstly, the salient regions of the image are obtained, then the intersection of the salient regions and the overlapping regions of the images is determined, the template area is set at the intersection of the current image's salient regions and the overlapping regions of the images, and the search area is set on the side where the previous frame image and the current image meet. At the same time, the size and position of this area can be flexibly adjusted according to factors such as the relative displacement trend between images, the distribution of salient regions, and computing resources to obtain better matching results.

[0022] Furthermore, in step S7, when using template matching for image registration, the optimization problem is solved iteratively to minimize the global image offset error; the specific optimization steps include: assuming the final image is obtained... With images Coordinate offset parameters between Then, by solving the optimization problem, the global image offset error can be minimized.

[0023] The optimization problem can now be defined as:

[0024] in Representing an image At a certain coordinate, Representing an image The corresponding coordinates.

[0025] When using the feature matching method for image registration, the affine transformation model is first used to fit the transformation relationship between images and solve the least squares problem. Then, the parameters of this affine transformation model are used as the starting point for optimizing the parameters of the perspective transformation model. The nonlinear optimization problem is solved iteratively, so that the overall reprojection error of all images is minimized under their own homography transformation matrix H.

[0026] The optimization problem under the affine transformation model can be defined as:

[0027] The optimization starting point of the affine transformation model is... The affine transformation parameters of the i-th image relative to the reference image are: , and Represent The first and second row vectors, the set of feature matching point pairs between image i and image j are , To match the number of point pairs, This represents the coordinates of the k-th matching point in the i-th image and the j-th image. Then, it represents the coordinates of the k-th matching point in the j-th image and the i-th image.

[0028] The optimization problem under the perspective transformation model can be defined as:

[0029] The optimization starting point for the perspective transformation model is the solution of the affine transformation model, if... Let be the perspective transformation model parameters for the i-th image after the k-th iteration. That is, before the iteration begins, the perspective transformation parameters are initialized to the solution under the aforementioned affine transformation model. For the i-th and j-th images, the size of the set of matching feature points is... Where the homogeneous coordinates of the k-th pair of matching points in the i-th image are The homogeneous coordinates in the j-th image are .

[0030] Furthermore, after obtaining the positional transformation relationship between images through registration, the images are fused to eliminate the seams between images. Specifically, whether it is intra-line fusion or inter-line fusion, the source image is first decomposed into high and low frequency coefficients using a transformation algorithm. Then, different fusion strategies are applied to different coefficients to complete the fusion in a hierarchical and directional manner. Finally, the image fusion is achieved through inverse transformation.

[0031] When performing image fusion, if the registration error is large, the fused image will have some obvious artifacts. In order to eliminate the artifact phenomenon after image fusion, the best seam line is quickly searched, and the above fusion is only performed within a certain width near the seam line. The area outside the width can be directly taken from the pixel value of a certain image, thereby effectively controlling the artifact problem and improving the quality of the fused image.

[0032] Furthermore, the images acquired in step S1 include one or more of multi-band infrared images and visible light images; when the acquired images include multiple types, the images are received in parallel multiple channels, and each channel of images is processed according to the process from step S2 to S8 to obtain independent panoramic images.

[0033] In the software, task abstraction is established. The task of stitching multiple images can be viewed as the sum of numerous subtasks that process stitching single images. These tasks are uniformly managed by the control module, and the operation of each task is independent of the others. Different inputs run in different tasks. This allows for easy expansion in the future to process more inputs of the same or different bands simultaneously, improving the scalability of the system. Each task stores information related to its own operation, and each task has its own worker thread. The worker threads in each task cooperate based on the producer-consumer pattern and communicate using thread-safe buffers. Therefore, each task also maintains its own related buffers.

[0034] In order to transmit images under limited network bandwidth, the image data needs to be encoded before transmission and then decoded after the system receives the image, so as to reduce network bandwidth pressure.

[0035] Furthermore, obtaining a complete panoramic image also includes a retrospective viewing step, specifically including: using a method based on image feature fine registration and local fine stitching to obtain a local retrospective high-resolution image; For the complete panoramic image obtained in step S8, when a rectangular region needs to be viewed locally, the region of this rectangle in the original single-frame image is determined according to the calculated registration relationship. The image is then traced back to the original single-frame image without degradation, cropped according to the region, and then processed using the methods from steps S4 to S8 to obtain a locally finely stitched high-resolution retrospective image. Since the number of images corresponding to the retrospective region is relatively small, feature registration, which provides better results, is generally used in the image registration stage.

[0036] According to a second aspect of the present invention, an image panoramic stitching system is provided, comprising: processor; A memory having a computer-executable program stored thereon, which, when executed by the processor, causes the processor to perform the image panoramic stitching method as described in any of the preceding claims.

[0037] In summary, compared with the prior art, the above-described technical solutions conceived by this invention mainly possess the following technical advantages: (1) This invention proposes a systematic perception and response strategy for occasional errors (such as frame errors and frame drops) and persistent errors (such as line drops), supporting error detection, position filling, reasonable estimation of registration information and safe system switching, effectively avoiding splicing crashes and abnormal restarts caused by errors, and ensuring the continuous and stable operation of the system.

[0038] (2) By introducing an image preprocessing module, the present invention effectively improves the quality and consistency of images and enhances the accuracy and stability of subsequent stitching.

[0039] (3) The present invention realizes efficient reception and parallel processing of multi-band image sequences under limited bandwidth, ensuring that the image data of different bands do not interfere with each other and are real-time, and improving the overall throughput and robustness of the system.

[0040] (4) This invention combines the hierarchical image registration and fusion mechanism between rows and between rows. In terms of matching method, it flexibly selects pixel-level matching, feature-level matching or deep learning matching strategy according to the actual scene. It also guides the selection of template area and search area through prior information, adapts to splicing tasks with different quality and requirements, and takes into account both real-time performance and accuracy.

[0041] (5) The present invention adopts an iterative optimization strategy to minimize the global error of the positional relationship between images, which significantly improves the consistency and geometric accuracy of image stitching. In particular, for nonlinear transformation scenarios, it takes into account the joint optimization of affine and perspective models, which improves the tightness of stitching.

[0042] (6) This invention utilizes image fusion and a dynamic programming stitching strategy to effectively eliminate stitching seams and fusion artifacts between images, thereby improving the visual quality and spatial continuity of the final multi-band panoramic image.

[0043] (7) The present invention introduces a local high-resolution back-tracking stitching mechanism, which enables high-precision restoration and fine stitching of any region on the basis of large-scale low-resolution stitching, to meet the back-tracking needs of key region details, and to expand the applicability and practical value of the system. Attached Figure Description

[0044] Figure 1 A flowchart of a target capture method in a video sequence scene provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the input and output of the splicing system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the multi-task abstraction provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the input and output of inline image registration and fusion provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the input and output of interline image registration and fusion provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of occasional error handling provided in an embodiment of the present invention; Figure 7 A schematic diagram illustrating the handling of persistent errors using an analogy to handle occasional errors, provided in an embodiment of the present invention. Figure 8 A schematic diagram illustrating the strategy for selecting templates and search areas based on the fluctuation range of overlapping regions for inline registration (left) and interline registration (right) provided in embodiments of the present invention; Figure 9 The salient region generation effect diagram based on Spectral Residual provided in the embodiments of the present invention; Figure 10 The diagram shows the effect of generating the optimal suture line based on the dynamic programming method provided in this embodiment of the invention. Figure 11 Image fusion results provided by embodiments of the present invention, with and without the optimal suture line search algorithm (left and right); Figure 12 This is an image registration effect diagram without minimum reprojection error constraint provided in an embodiment of the present invention; Figure 13 This is an image registration effect diagram with minimum reprojection error constraint provided in an embodiment of the present invention; Figure 14 This is a local fine-grained backtracking effect diagram provided in an embodiment of the present invention; Figure 15 The flowchart of the image panoramic stitching method provided by the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0046] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0047] like Figure 15 This invention provides a method for panoramic image stitching, comprising the following steps: S1, sequentially acquire images of the target area to obtain several rows of images. Each row of images consists of several frames of images. Preprocess each frame of images. S2, sequentially determine whether each row of the preprocessed image has an intra-row sporadic error or an inter-row persistent error. If a row of the image does not have either of these conditions, proceed to step S3. If a row of the image has an intra-row sporadic error, proceed to step S4. If a row of the image has an inter-row persistent error, proceed to step S5. S3. Based on prior information, obtain the adjacency map of the preprocessed image, and perform intra-row image registration for adjacent images belonging to the same row to obtain the position transformation relationship between adjacent images in the row. Then, perform intra-row image fusion to eliminate the stitching seams of the intra-row images and obtain a complete panoramic image of a row. S4. When an occasional frame misalignment error occurs in a row of images, the misaligned frame image is discarded and the corresponding position is filled with black pixels; when an occasional frame drop error occurs in a row of images, the corresponding position is filled with black pixels; then the images are registered and fused in the row to obtain a complete panoramic image of a row. S5. When a persistent inter-line error occurs, the missing line position is left blank according to the prior information, and the entire line is filled with black pixels to obtain a panoramic view of a complete line filled with black pixels. S6. Perform interline image registration on all the inline panoramic images obtained in steps S3 to S5 to obtain the positional transformation relationship between adjacent images in the interline. S7. Calculate the reprojection error of the image that has been registered in step S6, solve the optimization problem to minimize the reprojection error between images, and optimize the registration parameters of each frame image. S8. Perform interline image fusion on the image optimized in step S7 to eliminate the interline image stitching seams and obtain a complete panoramic image.

[0048] The prior information includes: the model and structural parameters of the image acquisition device (e.g., focal length, aperture, shutter speed, etc.), acquisition environment information (e.g., lighting conditions, scene type, etc.), acquisition parameter information (e.g., exposure parameters, overlap rate, acquisition order, etc.), preprocessing information (e.g., distortion correction parameters, etc.), and metadata information (e.g., shooting date, image selection angle, etc.).

[0049] Specifically, obtaining a complete panoramic image also includes a retrospective viewing step, which specifically includes: using a method based on image feature-based fine registration and local fine stitching to obtain a local retrospective high-resolution image; For the complete panoramic image obtained in step S8, when a rectangular region needs to be viewed locally, the region of this rectangle in the original single-frame image is determined according to the calculated registration relationship. The image is then traced back to the original single-frame image without degradation, cropped according to the region, and then processed using the methods from steps S4 to S8 to obtain a locally finely stitched high-resolution retrospective image. Since the number of images corresponding to the retrospective region is relatively small, feature registration, which provides better results, is generally used in the image registration stage.

[0050] The present invention will be further described in detail below through specific embodiments.

[0051] Example 1 A panoramic stitching method for multi-band image sequence scenes. (See also...) Figure 1 , combined Figures 2-14 The panoramic stitching method for multi-band image sequence scenes in this embodiment is described in detail. The specific method includes operations S1-S9.

[0052] Operation S1 allows the system to simultaneously receive multiple multi-channel, multi-band image sequences, such as infrared and visible light images, without interference, and initiate multiple stitching tasks in parallel. The specific steps are as follows: To abstract and segment different tasks, and to ensure that different path image stitching tasks run independently, each task stores information related to its own operation. For example... Figure 3As shown, the task includes contextual information, such as the number of lines already stitched in the current task and whether a line break is needed; it has its own worker threads, including task allocation threads, intra-line registration threads, intra-line fusion threads, inter-line registration threads, inter-line fusion threads, local backtracking threads, and input / output threads; the task also has configuration and mode information, which allows users to configure different system parameters for different types of input, such as image save location and image registration method selection strategy; the worker threads in the task cooperate based on the producer-consumer pattern and communicate using thread-safe buffers, so the task also maintains its own related buffers.

[0053] Input data is obtained in two ways: through local storage media and through network bandwidth communication. With local input, tasks can isolate inputs by reading different folders, preventing inputs from different tasks from mixing. With network input, multiple image inputs are monitored by different ports. Considering the limited network bandwidth (e.g., gigabit bandwidth is generally insufficient to meet the minimum requirement of two 30-frame, 2560×2048 8-bit grayscale image inputs), the data needs to be compressed before network transmission and then decompressed.

[0054] Operation S2 performs preprocessing on the acquired image data from different paths to improve image quality and facilitate subsequent stitching tasks.

[0055] Preferably, operation S2 specifically includes the following sub-operations S20-S24.

[0056] In sub-operation S20, the effective area of ​​the image is first cropped. For example, for image acquisition sensors, although the resolution of the output images from visible light television and infrared thermal imaging sensors is the same, the size of the effective area is different. Both images have a resolution of 1920×1080, but the thermal imaging output shows black borders around the image, with an effective area resolution of only 1280×1024. To prevent these black borders from affecting the stitching result, the system crops the image according to a pre-defined correspondence between image categories and effective areas. However, in actual systems, the method for determining the effective area depends on the specific sensor output.

[0057] In sub-operation S21, the effective image obtained from cropping is downscaled to accelerate subsequent image registration and image fusion, thereby improving the system's stitching efficiency. Generally, when using the NCC method for template matching, the total stitching time varies depending on the image resolution, from intra-line single-frame image registration and fusion to inter-line image registration and fusion. Taking a 2560×2048 image as an example, a 4x resolution downscaling results in a total time of 6647ms, a 16x downscaling results in 1822ms, a 50x downscaling results in 599ms, and a 104x downscaling results in 112ms. Furthermore, as the downscaling rate increases, image quality decreases, and image registration accuracy also decreases. In this example, the system typically uses a downscaling rate of around 50x, ensuring real-time stitching while maintaining a certain level of accuracy. For other image sizes, the appropriate downscaling rate should be selected based on the specific circumstances.

[0058] In sub-operation S22, the down-resolution image is smoothed using a 3×3 Gaussian kernel to reduce the impact of Gaussian noise on registration.

[0059] In sub-operation S23, a certain vignetting removal template is superimposed on the Gaussian-smoothed reduced-rate image to eliminate the vignetting effect caused by the camera's optical lens aperture blocking some light, resulting in a bright center and dark edges in the image, thus making the image brightness more uniform and natural. This vignetting removal template is prepared by shooting in a constant-light indoor environment facing an interference-free background and then performing brightness inversion.

[0060] In sub-operation S24, distortion correction is performed on the reduced-rate image after vignetting removal. This corrects radial distortion caused by errors in the camera's optical lens manufacturing process, resulting in different curvatures between light rays far from and near the lens center, as well as tangential distortion caused by the lens not being perfectly parallel to the image plane during lens installation. In normal stitching tasks, for ordinary cameras, the "Zhang Zhengyou calibration method + Brown model" is used to correct radial and tangential distortions. For a very small number of lenses with particularly large fields of view, a fisheye model is used for correction.

[0061] In operation S3, for the pre-processed downscaled image, whether it is intra-line registration or inter-line registration, there is a possibility that due to various environmental interferences, the upstream data may have occasional errors such as frame loss or frame errors, or continuous errors such as frame loss or frame errors in one or several lines. Therefore, the system cannot absolutely trust the upstream data and needs to have emergency handling plans for these different error situations to ensure that the entire stitching can proceed normally without crashing.

[0062] Specifically, for intermittent errors within a line, regardless of whether the frame is determined to be an error frame by CRC check during image reception decoding or the frame is directly lost and not received, this situation may occur at the beginning, end, within the line, or during line breaks. In terms of quantity, any number of frames may be lost, or even all odd-numbered frames may be lost. Therefore, it is necessary to fully consider the boundary conditions and make the system as adaptable as possible to frame errors and frame losses in various positions. Special processing should be performed on the image and registration information, and black pixels should be used in the image to fill the lost or error frame positions, so that the number of frames within each stripe remains at a certain value.

[0063] First, image coordinates are defined: the acquisition direction of several frames in a row of images is defined as the y-direction, and the direction of several rows side-by-side is defined as the x-direction. When handling the aforementioned intermittent errors within a row, the following approach is used: Figure 6 For example, the following four basic scenarios can cover all occasional error occurrences: (1) Registration information for lost frames after the first valid frame: the y-coordinate is filled with the relative position offset between the previous frame and the frame before that plus the coordinate of one frame; the x-coordinate remains the same as the previous frame. Figure 6 The registration is performed on the image from frame #5; its coordinates (x5, y5) are registered as (x4, y4 + (y4 - y3)).

[0064] (2) The registration information of the lost frames before the first valid frame is filled with the registration information of the corresponding position in the previous row of the image plus the registration information of the first valid frame. Figure 6 The registration of the #1 and #2 frames in the image is performed by first processing the lost frames adjacent to the first valid frame. The coordinates (x2, y2) of the #2 frame image are registered as (x3 + (last_x2 - last_x3), y3 + (last_y2 - last_y3)), and the coordinates (x1, y1) of the #1 frame image are registered as (x2 + (last_x1 - last_x2), y2 + (last_y1 - last_y2)). In particular, if the line is the first line, the calculation is performed as follows (3). Similarly, the lost frames adjacent to the first valid frame are processed first. The coordinates (x2, y2) of the image of frame #2 are (x3, y3-1×(height×(1-overleap))) after registration, and the coordinates (x1, y1) of the image of frame #1 are ((x2, y2-1×(height×(1-overleap))) after registration. Height represents the image height in the current Y direction, and overleap represents the overlap rate between images.

[0065] (3) For valid frames where the previous frame was lost, the registration coordinates are estimated using prior information, thus avoiding the problem of incorrect results obtained by still performing registration when frames are lost. Figure 6The registration of frames #3 and #6 in the image; wherein, in a certain row of images, for the first valid frame (corresponding to) where a frame was dropped in the previous frame. Figure 6 In step #3), the first valid frame is used as the reference, with its x-coordinate defined as 0 and its y-coordinate as 0 + 2 × (height × (1 - overleap)), where height represents the current image height in the Y direction and overleap represents the overlap rate between images; when a frame was dropped in the previous frame and the valid frame is not the first valid frame in the row (corresponding to... Figure 6 If the coordinates of the sixth frame in the previous frame are (x6, y6), then the registered coordinates are (x4, y4 + 2 × (height × (1 - overleap))), which is estimated based on the coordinates of the nearest previous valid frame.

[0066] (4) Valid frames from the previous frame that did not experience frame loss can be directly registered normally (e.g., template matching or feature matching). Figure 6 The situation in frame #4.

[0067] The above processing can maintain the system's fault tolerance even when sporadic errors such as frame errors and frame drops occur, thus enhancing the system's robustness.

[0068] For persistent errors where one or several lines are experiencing frame drops or misalignments, according to Figure 4 , Figure 5 First, intra-line registration and fusion are performed to obtain one line of stripes. Then, inter-line registration and fusion are performed for stitching. As long as at least one frame exists within a line, the aforementioned mechanism for handling occasional intra-line errors can ensure that the line of stripes can be output and maintain a stable number of frames within the line, thus not affecting subsequent inter-line stitching. However, if persistent errors cause errors or loss in the entire line or several lines of the sequence, then the two lines of stripes adjacent to the lost line will not have overlapping areas. Figure 7 Taking the simple loss of one line as an example, all inline panoramic images are logically rotated 90 degrees, and each line is treated as a single frame image. Then, following the processing of frame loss, the interline registration of the first valid line after the line loss occurs is abandoned. Relying on the interline overlap rate information, the coordinates of the fourth line in the registration coordinate system are initialized to obtain the interline registration information when the line loss occurs. Figure 7 In the middle, the entire third row is lost, so it can be handled according to the above situation (1). Its coordinates (x3, y3) are registered as (x2, y2 + (y2 - y1)). The fourth row can be handled according to the above situation (3). Its coordinates (x4, y4) are registered as (x2, y2 + 2 × (height × (1 - overleap))). height represents the height of the nearest row of images in the current Y direction, and overleap represents the inter-row overlap rate.

[0069] Operation S4, after obtaining the preprocessed image through S2, is followed by S3 to determine if there are no sporadic or persistent errors. Figure 4 Then, we perform intra-line image registration to obtain the registration relationship between the images. There are many methods to choose from here: If pixel-level registration methods such as template matching are used for intra-line image registration, the offset coordinates in the X and Y directions of the intra-line image relative to the intra-line reference image are obtained and organized into the form of a homography transformation matrix H:

[0070] When feature matching is used for intra-line image registration, the homography transformation matrix H of the intra-line image relative to the intra-line reference image is obtained:

[0071] When using template matching for inline image registration, it is necessary to select a template area and a search area. Two methods can be selected based on image quality and system real-time requirements: (1) When the image quality is high, the texture details are relatively rich, and the real-time requirements are high, a strategy of selecting the template and search area based on the fluctuation range of the overlapping area is adopted, such as Figure 8 Taking the prior information inline (single frame) overlap rate as a%±b% as an example, the search region height is taken as a%+b% of the image height, the search region width is taken as the image width, and the search region is located on the side of the previous frame image closer to the current frame image. The template region height is taken as a%-b% of the image height, the template region width is taken as 50% of the image width, and the template region is located in the middle part of the side of the current frame image closer to the previous frame image. (2) When the image quality is relatively average and the real-time requirements are relatively low, a strategy of selecting templates and search areas based on salient regions is adopted, such as... Figure 9 The specific steps are as follows: obtain the image spectrum, calculate the spectral mean model, subtract the values ​​to obtain the saliency spectrum, return the saliency spectrum to the image domain, threshold the saliency image to obtain the saliency region, and simultaneously obtain the intersection of the saliency region and the image overlap region based on the prior information of the intra-row (single frame) overlap rate. Use the intersection of the saliency region and the overlap region of the current frame image as the template region, and the intersection of the saliency region and the overlap region of the previous frame image as the search region.

[0072] Taking inline image registration using template matching as an example, there are several methods to choose from: (1) Interpolation-based matching method: 1.1 Image matching degree is calculated based on the sum of absolute differences (SAD), which is suitable for scenarios where the image grayscale changes are small and speed is a high priority. 1.2 Image matching degree is calculated based on the sum of squared differences (SSD), which is better than SAD in terms of noise resistance.

[0073] (2) Relevance-based matching methods: 2.1 Image matching degree is calculated based on normalized cross-correlation (NCC), which has strong robustness to changes in brightness and contrast; 2.2 Image matching degree is calculated based on zero-mean cross-correlation (ZNCC), a simplified version of NCC.

[0074] When using feature matching for intra-line image registration, there are also several methods to choose from: (1) Feature extraction and matching methods based on traditional manual design: 1.1 Based on SIFT feature extraction + KD tree descriptor matching + RANSAC homography matrix estimation, it has scale, rotation and brightness invariance and robust matching; 1.2 Based on SURF feature extraction + KD tree descriptor matching + RANSAC homography matrix estimation, SIFT is accelerated, achieving a trade-off between accuracy and speed; 1.3 Based on ORB feature extraction + KD tree descriptor matching + RANSAC homography matrix estimation, it is extremely fast, but has low accuracy; 1.4 Other similar manually designed feature extraction and feature matching methods are not listed here; they should be used as needed based on specific circumstances. (2) Feature extraction and matching methods based on deep learning: 2.1 SuperPoint feature extraction + SuperGlue matching + homography matrix estimation: The SuperPoint network realizes key point detection and descriptor extraction based on a fully convolutional structure. SuperGlue further introduces a graph matching attention mechanism to improve matching accuracy. It has high overall robustness and is suitable for structured scenes and weak texture regions. 2.2 D2-Net Feature Extraction + Nearest Neighbor Matching + RANSAC Estimation: A Dense feature network that integrates detection and description, extracts dense feature points end-to-end, with good matching accuracy, but high computational cost; 2.3 Novel deep feature extraction networks such as R2D2, DISK, and KeyNet, combined with corresponding matching strategies, improve repeatability and discriminability while maintaining a certain level of real-time performance, and adapt to complex deformations and occlusions; 2.4 Lightweight matching networks such as LightGlue and TopicFM: While maintaining matching accuracy, they significantly reduce inference latency, making them suitable for embedded devices or resource-constrained scenarios; 2.5 Keypoint-free direct matching methods such as LoFTR and Patch2Pix: These methods use Transformer or graph neural networks to directly predict dense correspondences, skipping the keypoint extraction process. They are suitable for tasks with extremely high requirements for geometric relationships. 2.6 Other similar deep learning feature extraction and feature matching methods are not listed here; they should be used as needed.

[0075] Operation S5, based on the registration relationship of the in-row images obtained in S4, i.e., the geometric transformation relationship between the images, projects all images in a row onto the reference plane according to this transformation relationship, such as... Figure 10 As shown, a dynamic programming approach is used to quickly search for the optimal seam line. Then, within a finite width around the seam line, transform domain methods, such as pyramid transform fusion or wavelet transform fusion, are applied to perform intra-row image fusion. Outside the seam line, pixel values ​​from a specific frame are directly used, thus eliminating intra-row image seams and obtaining a high-resolution, uniformly bright, and clearly visible panoramic image of a complete row of images. Figure 11 The left side shows the image fusion effect achieved by using only the transform domain method, while the right side shows the image fusion effect achieved by using the seam line plus transform domain method. The effect on the right side is better than that on the left side.

[0076] Specifically, pyramid-based image fusion is a multi-scale image processing method. Its core idea is to decompose an image into multiple spatial frequency levels and fuse them at each scale separately, achieving a smooth edge and natural brightness transition, eliminating seams and artifacts. The process involves first constructing a Laplacian pyramid (capturing detailed features at different scales) and a Gaussian pyramid mask (representing the spatial distribution of fusion weights, ensuring smooth transitions at image seams) for each of the two images to be stitched. At each layer, the Laplacian map of the corresponding layer is weighted at the pixel level according to the mask weights, achieving a smooth transition from one image to another at the boundary regions. After fusion, the image is reconstructed layer by layer from the bottom layer upwards to reconstruct a complete fused image. Compared to direct stitching, pyramid fusion can coordinate structural and brightness differences between images at multiple scales, resulting in a fusion result without obvious discontinuities or brightness jumps at the edges, leading to a more natural and coherent visual effect.

[0077] Image fusion based on wavelet transform is a frequency domain fusion technique. Its principle is to decompose an image into frequency sub-bands of different scales and directions using wavelet transform, representing low-frequency information (overall contour, brightness) and high-frequency information (edges, details). Selective fusion processing is then performed on these sub-bands to eliminate seams and artifacts. The specific process is as follows: First, the images to be fused are decomposed using wavelet transform, resulting in a set of low-frequency sub-images and high-frequency sub-images in multiple directions. Then, for each corresponding wavelet sub-band, strategies such as maximum value selection, weighted average, or region energy are used to fuse the information from the two images, retaining the information from the one with stronger edges and clearer textures. After all sub-bands are fused, the fused frequency information is reconstructed into a complete image using inverse wavelet transform. Because wavelet transform has excellent time-frequency localization capabilities, it can effectively separate and analyze texture, details, and contours in an image. It can achieve a natural multi-scale transition of texture and brightness in the stitching boundary region, thus significantly reducing abrupt seams and artifacts, resulting in a smoother and more natural fusion result.

[0078] In practical applications, image fusion based on wavelet transform yields an image with an overall grayscale level closer to the original image and higher clarity than image fusion based on pyramid transform. However, wavelet transform is computationally intensive and time-consuming, and is not as computationally efficient as pyramid transform. Therefore, it is necessary to flexibly select the appropriate method based on the specific task's computational resources and real-time requirements.

[0079] Operation S6, through in-line merging, continuously obtains the spliced ​​stripes of each line. After judgment by S3, if there are no persistent errors, then... Figure 5 Similarly, using the image registration methods described above, such as pixel-level matching or feature-level matching, we perform inter-row registration to obtain the positional registration relationship between adjacent strips. Taking a specific stitching example, let the first frame image acquired by the task be the reference image. Since the position of each frame image in its respective strip is known in step S4, and the positional relationship between strips is known through inter-row registration in step S6, we can calculate the positional transformation relationship of all images in the current strip based on the initial reference image after coordinate transformation.

[0080] When using pixel-level matching for interline registration, there are still two strategies to choose from for selecting the template area and the search area: (1) When the image quality is high, the texture details are relatively rich, and the real-time requirements are high, a strategy of selecting the template and search area based on the fluctuation range of the overlapping area is adopted, such as Figure 8If the line overlap rate is c%±d%, then the search area width is c%+d% of the entire line image width, the height is the height of the entire line image, and the search area is located on the side of the previous line image closer to the current line image. The template area width is c%-d% of the entire line image width, and the template area is located in the middle part of the current line image on the side closer to the previous line image, and the height is 50% of the entire line image.

[0081] (2) When the image quality is relatively average and the real-time requirements are relatively low, the aforementioned strategy of selecting templates and search areas based on significant regions is adopted.

[0082] When using feature matching-based methods for interline registration, two strategies are still included: traditional hand-designed feature extraction and matching methods, and deep learning-based feature extraction and matching methods. The specific strategy to choose depends on the task requirements.

[0083] In operation S7, after obtaining the registration relationships of all images in all received complete rows through inter-row registration in S6, there may be significant cumulative errors in the image registration relationships due to mismatches in some areas. At this point, an iterative method is needed to fix the registration relationships of the other rows of images that have already been stitched together, and only optimize the registration relationships between images within the current row to minimize the final global reprojection error. The comparison between before and after optimization is shown below. Figure 12 , Figure 13 As shown.

[0084] Specifically, when template matching is used in the image registration process, let the final image be... With images Coordinate offset parameters between Then, the Levenberg-Marquardt optimization algorithm is used to iteratively solve the problem, so as to minimize the global image offset error.

[0085] The optimization problem can now be defined as:

[0086] in Representing an image Top left corner coordinates Representing an image Top left corner coordinates; When feature matching is used in the image registration process, the transformation matrix H between images has 8 degrees of freedom. First, an affine transformation model is used to fit the transformation relationship between images and solve the least squares problem. Then, the parameters of this affine transformation model are used as the starting point for optimizing the parameters of the perspective transformation model. The Levenberg-Marquardt optimization algorithm is used to iteratively solve the nonlinear optimization problem, so that the overall reprojection error of all images is minimized under their own homography transformation parameters H.

[0087] The optimization problem under the affine transformation model can be defined as:

[0088] The optimization starting point of the affine transformation model is... The affine transformation parameters of the i-th image relative to the reference image are: , and Represent The first and second row vectors, the set of feature matching point pairs between image i and image j are , To match the number of point pairs, This represents the coordinates of the k-th matching point in the i-th image and the j-th image. Then, it represents the coordinates of the k-th matching point in the j-th image and the i-th image.

[0089] The optimization problem under the perspective transformation model can be defined as:

[0090] The optimization starting point for the perspective transformation model is the solution of the affine transformation model, if... Let be the perspective transformation model parameters for the i-th image after the k-th iteration. That is, before the iteration begins, the perspective transformation parameters are initialized to the solution under the aforementioned affine transformation model. For the i-th and j-th images, the size of the set of matching feature points is... Where the homogeneous coordinates of the k-th pair of matching points in the i-th image are The homogeneous coordinates in the j-th image are .

[0091] Operation S8 involves optimizing and updating the registration relationship parameters between images, followed by inter-row image fusion. Specifically, on an infinitely created canvas, after correcting the registration relationship between images within the current row by minimizing the reprojection error constraint, each image in the current row has an optimized absolute position relative to the reference image. Using this absolute position, all images in the current row are projected onto the canvas, and the optimal stitching line between all images in the current row and between all images in the current row and all images in the previous row is quickly searched using dynamic programming. Then, within a finite width around the stitching line, transform domain methods, such as the aforementioned pyramid-based transform fusion and wavelet-based transform fusion, are used to perform intra-row and inter-row image fusion. Outside the stitching line, the pixel values ​​of a certain frame image are directly used to eliminate image seams. By continuously repeating the above operations S3 to S8 on the read images, a complete panoramic image of all received images with high resolution, uniform brightness, and clear visual quality is finally obtained.

[0092] After stitching all images to obtain a panoramic image using operation S9, some local details are not very clear because the stitching process was based on the images that were downscaled during the image preprocessing stage. If there is a need to review specific areas, then fine stitching of the original images that were not downscaled is required. The specific steps are as follows: By performing intra-line and inter-line image registration through S4~S8, the final fused panoramic image can trace back to the original single-frame image based on the calculated registration relationship for each coordinate position. When viewing a rectangular region (x, y, w, h), x represents the horizontal coordinate offset of the top-left corner of the rectangular region relative to the top-left corner of the panoramic image, y represents the vertical coordinate offset of the top-left corner of the rectangular region relative to the top-left corner of the panoramic image, w represents the width of the rectangular region in the horizontal direction, and h represents the width of the rectangular region in the vertical direction. The region of this rectangle in the original single-frame image is obtained. Taking this rectangle as an example, which is obtained by stitching together a part of each of the four images in the original single frame, we trace back to the original four images I1~I4 acquired in step S1. When performing step S2 on them, we do not perform downscaling, but keep everything else the same. Then, we crop the part that belongs to the rectangular region to obtain I1'~I4'. Then, we still use steps S4~S8 to stitch together to obtain a high-resolution backtracking image with fine local stitching. Because the number of images is very small, the processing time is minimal, and high-quality local backtracking stitching is required. Therefore, for steps S4-S8, we prioritize using more effective feature registration and fusion methods. If hardware acceleration is available, we prioritize deep neural network-based methods for feature registration and fusion. If hardware acceleration is unavailable, we prioritize traditional feature-based image registration and fusion algorithms. The final local backtracking result is as follows: Figure 14As shown.

[0093] The panoramic stitching method for multi-band image sequence scenes provided in this invention first decouples the image reception and multi-channel parallel decoding from the task. Then, for each frame, image preprocessing operations such as effective region cropping, Gaussian blurring, vignetting removal, and distortion correction are performed to improve image quality. Before intra-row image registration, it is determined whether there are occasional errors such as frame errors or missing frames; if so, the registration relationship is estimated using prior information. After eliminating these occasional errors, intra-row image registration is performed to obtain the registration relationship between intra-row images. Using these registration relationships, the seam line is located, and intra-row image fusion is performed around the seam line using a transform domain method to obtain a complete row of image strips. These intra-row fused image strips are then used for inter-row registration to obtain the registration relationship between inter-row image strips. Finally, the current... The registration relationship between images within each row is obtained, correspondingly yielding the registration relationship of each frame image within the current row relative to the previous row image. Then, the registration transformation parameters of the previous few rows are frozen, and under the constraint of minimizing the global reprojection error, the registration relationship of all images in the current row is iteratively optimized. Subsequently, all images in the current row are continuously fused with the already stitched images on the canvas. The above process of intra-row registration, intra-row fusion, inter-row registration, iterative optimization, and inter-row fusion is repeated continuously until all received images are stitched together to obtain a complete ultra-high resolution panoramic image. Finally, if there is a need for detailed retrospective viewing of local locations, the original frame image index is deduced based on the retrospective local location, and the original frame image is used to perform intra-row registration, intra-row fusion, inter-row registration, iterative optimization, and inter-row fusion to obtain a local retrospective high-resolution image. This method can solve the following technical problems existing in the prior art: high system structure coupling, poor scalability, lack of modular management mechanism; lack of unified image preprocessing framework; lack of automatic detection and recovery mechanism for problems such as frame errors, frame loss, and line loss; lack of stitching state perception and adaptive control capability in the stitching process; unstable fusion effect, abrupt transition of image boundaries, and easy generation of artifacts; inability to achieve high-resolution backtracking and fine stitching processing for arbitrary areas in panoramic images.

[0094] Example 2 A panoramic stitching system for multi-band image sequence scenes includes: a processor; and a memory storing a computer-executable program, which, when executed by the processor, causes the processor to perform the panoramic stitching method for multi-band image sequence scenes described above. The related technical solutions are the same as in Embodiment 1 and will not be repeated here.

[0095] Example 3 A computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the panoramic stitching method for multi-band image sequence scenes described above. The related technical solutions are the same as in Embodiment 1 and will not be repeated here.

[0096] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for panoramic image stitching, characterized in that, Includes the following steps: S1, sequentially acquire images of the target area to obtain several rows of images. Each row of images consists of several frames of images. Preprocess each frame of images. S2, sequentially determine whether each row of the preprocessed image has an intra-row sporadic error or an inter-row persistent error. If a row of the image does not have either of these conditions, proceed to step S3. If a row of the image has an intra-row sporadic error, proceed to step S4. If a row of the image has an inter-row persistent error, proceed to step S5. S3. Based on prior information, obtain the adjacency map of the preprocessed image, and perform intra-row image registration for adjacent images belonging to the same row to obtain the position transformation relationship between adjacent images in the row. Then, perform intra-row image fusion to eliminate the stitching seams of the intra-row images and obtain a complete panoramic image of a row. S4. When an intermittent frame error occurs in a row of images, the frame error image is discarded and the corresponding position is filled with black pixels. When an intermittent frame drop error occurs in a row of images, black pixels are used to fill the corresponding position; then the images are registered and fused in the row to obtain a complete panoramic image of a row. S5. When a persistent inter-line error occurs, the missing line position is left blank according to the prior information, and the entire line is filled with black pixels to obtain a panoramic view of a complete line filled with black pixels. S6. Perform interline image registration on all the inline panoramic images obtained in steps S3 to S5 to obtain the positional transformation relationship between adjacent images in the interline. S7. Calculate the reprojection error of the image that has been registered in step S6, solve the optimization problem to minimize the reprojection error between images, and optimize the registration parameters of each frame image. S8. Perform interline image fusion on the image optimized in step S7 to eliminate the interline image stitching seams and obtain a complete panoramic image.

2. The image panoramic stitching method according to claim 1, characterized in that, In step S1, the preprocessing includes: cropping the effective region of the image, reducing the resolution of the cropped effective image, smoothing and denoising the image after reduction in resolution, eliminating vignetting in the image after smoothing and denoising, and finally correcting distortion.

3. The image panoramic stitching method according to claim 1, characterized in that, In step S4, for a valid frame in a certain row of images where the previous frame was misframed or lost, the registration coordinates of this valid frame are estimated using prior information. For a frame error or missing frame that occurs after the first valid frame in a row of images, the X-coordinate of the registration information of this frame error or missing frame remains the same as the previous frame next to it, and the Y-coordinate is filled with the relative position offset between the previous frame and the frame before that plus the coordinate of the previous frame; where X-direction is the parallel direction of the row, and Y-direction is the acquisition direction of several frames in a row of images. For a frame error or missing frame that occurs before the first valid frame in a row of images, the registration information of this frame error or missing frame is filled with the registration information of the corresponding position in the previous row of images plus the registration information of the first valid frame in that row. For a valid frame in a certain row of images where the previous frame has not been misaligned or lost, this valid frame is directly registered in the normal inline image.

4. The image panoramic stitching method according to claim 3, characterized in that, In step S6, when a line drop occurs, all the inline panoramic images obtained in steps S3 to S5 are logically rotated by 90 degrees, and each line is treated as a single frame image. Then, the line drop processing method in step S4 is used to process the line drop, thereby realizing the line drop processing of the interline image.

5. The image panoramic stitching method according to claim 4, characterized in that, The inline image registration and interline image registration include pixel-level matching and feature-level matching methods; When there is a translation transformation relationship between images, pixel-level information is used for image registration, and the position offset coordinates in the X and Y directions of two adjacent images or two rows of images are calculated. When the relationship between images is an affine transformation or a perspective transformation, feature-level information is used for image registration, and the homography transformation matrix H between two adjacent images or two rows of images is calculated.

6. The image panoramic stitching method according to claim 5, characterized in that, When using pixel-level matching for image registration, the template and search area are selected based on the fluctuation range of the overlapping area or based on the extraction of salient areas. The selection of template and search area based on the fluctuation range of the overlapping area includes: for intra-line registration, determining the size and position of the template area and search area according to the overlap ratio between intra-line images, wherein: the size of the search area is set to a certain ratio range of the reference image size, which is greater than or equal to the estimated overlap ratio; the size of the template area is set to match the search area but smaller than the search area; the positions of the template and search area are set within the potential overlapping area between adjacent images; for inter-line registration, also based on the inter-line image overlap relationship, dynamically adjusting the aspect ratio and specific position of the search area and template area during the stitching process of a row of images; The selection of template and search area based on salient region extraction includes: when selecting template area and search area for image registration, firstly, the salient region of the image is obtained, then the intersection of the salient region and the overlapping region of the image is determined, the template area is set at the intersection of the current image salient region and the overlapping region of the image, and the search area is set on the side where the previous frame image and the current image meet.

7. The image panoramic stitching method according to claim 5, characterized in that, In step S7, when using pixel-level matching for image registration, the optimization problem is solved iteratively to minimize the global image offset error. When using the feature matching method for image registration, the first step is to use an affine transformation model to fit the transformation relationship between images and solve the least squares problem. Then, the parameters of this affine transformation model are used as the starting point for optimizing the parameters of the perspective transformation model. The nonlinear optimization problem is solved iteratively, so that the overall reprojection error of all images is minimized under their own homography transformation matrix H.

8. The image panoramic stitching method according to claim 1, characterized in that, The images acquired in step S1 include, but are not limited to, one or more of multi-band infrared images and visible light images; when the acquired images include multiple types, the images are received in parallel multiple channels, and each channel of images is processed according to the process of steps S2 to S8 to obtain independent panoramic images.

9. The image panoramic stitching method according to any one of claims 1-8, characterized in that, Obtaining a complete panoramic image also includes a retrospective viewing step, specifically: using a method based on image feature fine registration and local fine stitching to obtain a local retrospective high-resolution image; For the complete panoramic image obtained in step S8, when it is necessary to view a rectangular area locally, the region of this rectangle in the original single-frame image is determined according to the calculated registration relationship. The image is then traced back to the original single-frame image without degradation, cropped according to the region, and then the method of steps S4 to S8 is used to obtain a high-resolution backtracking image with local fine stitching.

10. An image panoramic stitching system, characterized in that, include: processor; A memory having a computer-executable program stored thereon, which, when executed by the processor, causes the processor to perform the image panoramic stitching method as described in any one of claims 1-9.

Images