A multi-path image real-time splicing and repairing method based on a CR-Fill model

Abstract

This invention discloses a real-time multi-path image stitching and restoration method based on the CR-Fill model, belonging to the fields of computer vision and artificial intelligence. It includes image stitching, semantic cropping, and image restoration. In terms of core restoration performance, this invention benefits from the strong constraint of the generator features imposed by the Context Reconstruction Loss (CR Loss) during the training phase. The images generated by this method achieve excellent levels in both semantic rationality and visual fidelity. By completely removing the attention mechanism (CA layer) in the forward inference path and adopting the CR Loss training mechanism, the computational complexity of the system when processing high-resolution images is significantly reduced. The semantic cropping step automatically adapts to the varying boundary conditions generated by different stitching results through advanced image processing algorithms, ensuring that the image patches input to the restoration module are of uniform size and effective content, thus reducing error accumulation from the source.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to an image stitching and restoration method based on generative adversarial networks. More specifically, it is an integrated method for panoramic stitching, semantic cropping, and generative restoration of images acquired in real time from multiple cameras. It can be applied to technical fields requiring seamless visualization and reconstruction of large-scale scenes, such as autonomous driving environmental perception, security monitoring, virtual reality background generation, and geographic information image processing. Background Technology

[0002] With the rapid development of computer vision and artificial intelligence technologies, image processing technology plays a crucial role in many fields such as autonomous driving, security monitoring, and virtual reality. Among them, image stitching and image inpainting, as two key subfields, aim to expand the field of view of an image and restore missing information, and have always been a research hotspot and challenge.

[0003] Traditional methods primarily rely on feature point detection and matching algorithms, such as Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), and Oriented FAST and Rotated BRIEF (ORB). These methods extract feature points from images, perform feature matching and image registration, and finally generate a panoramic background image through a fusion algorithm. However, when dealing with scenes with large parallax, drastic lighting changes, or a lack of texture features, these methods often suffer from matching errors, obvious seams, and ghosting, resulting in poor stitching quality and weak robustness. Although deep learning-based stitching methods have emerged and improved stitching accuracy to some extent, they generally suffer from large model parameter counts, slow inference speeds, and high deployment difficulty on edge devices, and still cannot simultaneously meet the requirements of robust stitching in large parallax scenes and the high real-time performance of industrial scenarios.

[0004] In image inpainting, technological evolution has progressed from early diffusion-based and example-based methods to the current mainstream deep learning-based methods. The introduction of Generative Adversarial Networks (GANs), in particular, has significantly improved the realism and semantic plausibility of inpainted images. To further enhance inpainting quality, many advanced methods have introduced attention mechanisms, especially Contextual Attention (CA). CA explicitly "borrows" features from known regions to fill in missing parts by calculating the similarity between feature blocks in unknown and known regions. While these methods (such as DeepFillv2) have achieved significant progress in performance, they suffer from two inherent drawbacks: First, the CA layer requires calculating pairwise similarities between all blocks in the global feature map, resulting in extremely high computational complexity and significant overhead, severely impacting the model's inference speed and making it difficult to apply to scenarios with strict real-time requirements. Second, due to the lack of direct supervision signals regarding feature similarity, the CA mechanism may select inappropriate reference feature blocks, leading to noticeable artifacts, texture inconsistencies, and other distortions in the generated results.

[0005] Furthermore, current technology typically treats image stitching, cropping, and restoration as independent tasks, using separate models for processing. This "separate approach" has significant drawbacks: on the one hand, errors can propagate and accumulate between steps; for example, seams and color differences generated during the stitching stage can directly affect the quality of subsequent restorations. On the other hand, multiple independent models introduce additional overhead and integration complexity, making efficient end-to-end processing impossible.

[0006] In summary, existing technologies cannot simultaneously solve the challenges of robustness in image stitching and real-time performance in image inpainting. They also lack an integrated collaborative solution that deeply fuses image stitching, semantic cropping, and image inpainting, and there are no relevant technical solutions for applying the CR-Fill model to real-time image stitching and inpainting from multiple cameras. Consequently, they fail to meet the core requirements of industrial scenarios for high-precision, high-real-time performance, and high-robustness image processing. Developing a system that overcomes these limitations, ensuring high-performance indicators (such as structural similarity (SSIM) and peak signal-to-noise ratio (PSNR)) while meeting stringent real-time requirements (such as completion within a limited time), has become a pressing technical problem in this field. This invention aims to effectively address these shortcomings of existing technologies. Summary of the Invention

[0007] In the current field of image processing technology, especially in applications requiring rapid fusion and completion of multi-view images, several core technical bottlenecks urgently need to be overcome. Firstly, at the image stitching level, although traditional feature-point based algorithms (such as SIFT, SURF, and ORB) and some deep learning models are widely used, they still easily produce obvious stitching seams, ghosting effects, and global / local structural misalignment problems when handling complex textures, significant lighting differences, and scenes with large parallax and moving objects. These flaws not only disrupt the visual coherence of the image but also provide inherently flawed input for downstream image restoration tasks. Even with high-performance restoration algorithms, it is difficult to generate high-quality results that are semantically coherent and structurally accurate.

[0008] Secondly, while GANs have become the mainstream technology for image inpainting, the CA layer, widely used to improve inpainting quality, incurs extremely high computational overhead. This mechanism requires calculating the global similarity between all feature blocks in the unknown and known regions. Its computational complexity increases quadratically with image resolution, becoming a core bottleneck restricting the model's inference speed and making it unsuitable for scenarios with stringent real-time requirements, such as environmental perception in real-time monitoring or autonomous driving systems. Furthermore, the feature matching process of the CA layer lacks direct supervision and interpretability, easily selecting mismatched reference feature blocks, leading to distortions such as texture misalignment, blurring, and structural artifacts in the generated results—for example, incorrectly filling the ground area with sky texture.

[0009] Finally, existing technologies typically divide image stitching, semantic cropping, and image inpainting into three independent tasks, employing separate models and serial processing flows without end-to-end collaborative design. This leads to the progressive propagation and accumulation of errors across multiple stages. Geometric distortions and invalid black borders generated during the stitching stage directly reduce cropping accuracy, while non-standard cropping results further increase the difficulty of inpainting. Ultimately, the overall system performance, operational efficiency, and scenario robustness fail to meet the requirements of industrial deployment. Therefore, developing an integrated solution that deeply integrates the above stages and fundamentally solves the three major challenges of stitching quality, inpainting efficiency, and system integration has become crucial for driving this technology towards practical application.

[0010] To address the aforementioned technical problems, this invention proposes a real-time multi-channel image stitching and restoration method based on the CR-Fill model, which includes the following steps: Step 1: Image stitching: After preprocessing the synchronous raw images from multiple cameras, feature extraction, feature point matching, and image fusion are performed sequentially to generate a panoramic background image with a reserved regular blank area in the middle. The specific sub-steps are as follows: Step 1.1, Feature Extraction The Speeded Up Robust Features (SURF) algorithm based on the Hessian matrix is ​​adopted as the primary feature point detection method, supplemented by the Oriented FAST and Rotated BRIEF (ORB) algorithm based on the corner points of the Accelerated Segment Test (FAST) to extract feature points and corresponding feature descriptors from the original image.

[0011] Step 1.2, Feature Point Matching First, feature descriptors of different images are initially matched based on the Nearest Neighbor Distance Ratio (NNDR) method. Then, the Random Sample Consensus (RANSAC) algorithm is used to remove mismatched point pairs. Finally, the homography matrix between images is calculated based on the remaining correct matching point pairs after removal, thereby establishing the perspective transformation relationship between images.

[0012] Step 1.3, Image Fusion The calculated homography matrix is ​​used to project each image onto the same coordinate system, and a multi-band blending strategy is employed to eliminate seams and illumination differences at the stitching points. Specifically, the multi-band blending strategy first constructs a Laplacian pyramid of the image, decomposing the image into sub-bands of different frequencies; then, large-scale weighted blending is performed at the low-frequency levels of the pyramid to eliminate illumination differences, while small-scale weighted blending is performed at the high-frequency levels to preserve texture details; finally, the image is reconstructed through inverse Laplacian pyramid transformation to obtain a panoramic background image with a pre-reserved regular blank area in the middle.

[0013] Step 2: Semantic Pruning The panoramic background image obtained in step one is post-processed. Through binarization mask generation and morphological erosion operations, the maximum inscribed rectangle of the effective content area is located, and the panoramic background image with a hole in the center is cropped. Specific sub-steps are as follows: Step 2.1: Augmentation Processing and Binary Mask Generation Using image augmentation methods to transform panoramic backgrounds Figure 4 The black border is expanded by an integer number of pixels to prevent boundary overflow. Then, the image is subjected to binarization thresholding, which sets the valid pixel areas in the panoramic background image to white and the invalid background to black, generating a binary mask.

[0014] Step 2.2: Calculation of the maximum inscribed rectangle Initialize a rectangular region of the same size as the binary mask, and continuously perform morphological erosion operations on the rectangular region. In each morphological erosion operation, calculate the difference between the current rectangular region and the eroded result. When the difference region no longer contains valid pixels (i.e., it cannot continue to shrink inward), stop the morphological erosion operation. The rectangular region that is retained at this time is the maximum inscribed rectangle.

[0015] Step 2.3, Image Cropping Based on the coordinate parameters of the largest inscribed rectangle calculated in step 2.2, a standard-sized hole is cropped from the panoramic background image generated in step 1. This panoramic background image containing the hole serves as an image patch, becoming the input for subsequent image inpainting steps.

[0016] Step 3: Image Restoration First, a generative adversarial network (GAN) model based on Contextual Reconstruction Fill (CR Fill) is constructed to perform semantic inpainting on the image patches output from step two. The CR Fill-based GAN model consists of three modules: a main generator, an auxiliary context reconstruction branch, and a discriminator.

[0017] I. Main Generator The main generator, arranged in the order of image data processing, consists of five core components: a coarse-network gated convolutional encoder, a coarse-network gated transposed convolutional decoder, a fine-network gated convolutional encoder, a dilated gated convolutional module, and a fine-network gated transposed convolutional decoder. The reason for using gated convolutional layers instead of traditional convolutional layers in these encoders and decoders is that gated convolutional layers can apply different convolutional filter responses based on different input features, thereby effectively eliminating artifacts at the restoration boundaries.

[0018] The core components of the coarse-gated convolutional encoder and coarse-gated transposed convolutional decoder include downsampling gated convolutional layers, which are used to extract high-level semantic features of the image. During this encoding process, intermediate feature maps containing semantic information at different scales are generated. The only component of the dilated gated convolutional module is the dilated gated convolutional layer, which is used to expand the receptive field and capture long-range contextual information without reducing the resolution. The core components of the fine-gated convolutional encoder and fine-gated transposed convolutional decoder include upsampling gated convolutional layers, which are used to restore the feature maps to the original resolution.

[0019] II. Auxiliary Context Restructuring Branch To compensate for the potential decrease in semantic consistency caused by removing the attention mechanism, an auxiliary context reconstruction branch is introduced during the training phase, solely for assisting learning. This branch consists of a similarity encoder and an auxiliary decoder.

[0020] III. Discriminator The discriminator is composed of a multi-layer convolutional encoder module with spectral normalization. Its core function is to distinguish between real and complete images and images restored by the main generator. By training the reverse constraint generator adversarially, the realism, texture consistency and semantic rationality of the restored image are continuously improved.

[0021] Image restoration involves model training and deployment. The specific sub-steps are as follows: Step 3.1, Model Training Step 3.1.1: The similarity encoder in the auxiliary context reconstruction branch receives the intermediate layer feature map generated by the coarse-network gated convolutional encoder from the main generator, treating it as a set of feature blocks. For any feature block within a hole... With any feature block outside the hole ,by , Let represent the feature vectors corresponding to the two feature blocks, and calculate their cosine similarity. : in, and Represents a feature block. Represents the eigenvector. Represents the dot product. The norm of a vector.

[0022] For each feature block within the cavity Based on the aforementioned cosine similarity weights, the feature blocks within the holes are reconstructed as follows: : in, Represents the area outside the cavity. Scaling factor The function is used to normalize similarity weights.

[0023] Step 3.1.2: Reconstruct feature blocks within the hole Combined feature map The auxiliary decoder upsamples the auxiliary image by the auxiliary context reconstruction branch. Context reconstruction loss (CR Loss) is defined as the difference between the auxiliary image and the true complete panoramic background image. The sum of L1 loss and adversarial loss between them: in, Indicates the discriminator, The function is used for stable adversarial training.

[0024] By minimizing the aforementioned CR Loss, the master generator is forced to learn feature representations consistent with the semantics outside the holes, enabling the generated features inside the holes to be accurately reconstructed using features outside the holes.

[0025] Step 3.2, Deployment and Use: During the deployment phase, the auxiliary context reconstruction branch and discriminator are completely removed, retaining only the trained main generator. The main generator directly receives the panoramic background image with holes output from step two, and completes feature encoding and reconstruction through a coarse-network gated convolutional encoder, a coarse-network gated transposed convolutional decoder, a fine-network gated convolutional encoder, a hole-gated convolutional module, and a fine-network gated transposed convolutional decoder, outputting a repaired complete image patch. Then, based on the coordinate parameters cropped in step two, the repaired image patch is seamlessly fused back to its corresponding position in the panoramic background image. Due to the removal of the auxiliary context reconstruction branch, the generation process is significantly accelerated.

[0026] Compared with existing technologies, the multi-path image real-time stitching and restoration method based on the CR-Fill model proposed in this invention has significant technical advantages and beneficial effects, specifically reflected in the following three aspects: First, in terms of core restoration performance, thanks to the strong constraint of the generator features by the auxiliary context reconstruction loss (CR Loss) during the training phase, the images generated by this method achieve excellent levels in both semantic rationality and visual fidelity. Specifically, multiple test results on a limited test set containing three complex scenes—woodland, desert, and city—show that the structural similarity (SSIM) index of the restoration result is consistently maintained above 0.96, and the peak signal-to-noise ratio (PSNR) exceeds 29.5dB. This means that the restored image not only has minimal pixel-level errors, but more importantly, its structural, brightness, and contrast information highly matches the real scene. It can achieve coherent and natural completion for complex textured jungles, regularly structured buildings, and textured desert environments, effectively avoiding artifacts such as texture blurring, structural distortion, or semantic inconsistency that occur in common methods.

[0027] Secondly, the technical solution of this invention brings a revolutionary efficiency improvement, completely solving the real-time bottleneck of high-performance image restoration. By completely removing the attention mechanism (CA layer) in the forward inference path and adopting the CR Loss training mechanism, the computational complexity of the system when processing high-resolution images is significantly reduced. Real-world test data shows that from receiving four 1280×720 pixel input images, through panoramic stitching, semantic cropping, to the final restoration, the average processing time of the entire end-to-end process is only 0.67 seconds, far lower than the processing time of several seconds for existing mainstream image restoration models based on the CA mechanism. This efficiency advantage allows this invention to be seamlessly integrated into practical applications with stringent real-time requirements, such as real-time environment modeling in autonomous driving systems or real-time video stream restoration in security monitoring platforms, achieving a crucial leap from "usable" to "easy to use."

[0028] Furthermore, the end-to-end integrated design adopted in this invention endows the system with unprecedented robustness and practicality. The semantic cropping step automatically adapts to the varying boundary conditions generated by different stitching results through advanced image processing algorithms, ensuring that the image blocks input to the repair module are of uniform size and effective content, reducing error accumulation from the source. The entire system, as a tightly coordinated whole, demonstrates strong adaptability to various real-world scenarios. Regardless of fluctuations in lighting conditions, scene textures, or initial stitching quality, the system maintains stable and reliable output performance. Finally, this solution has high engineering practical value. Its model architecture has been carefully optimized, making its requirements for computing hardware more accessible. It has been successfully deployed on conventional computing platforms equipped with NVIDIA GeForce RTX 3060 and above graphics cards and has achieved stable operation, demonstrating excellent commercialization potential and broad industrial application prospects. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the image stitching process provided in an embodiment of the present invention. The diagram illustrates the core process of generating a seamless panoramic background image by taking multiple synchronous original images with overlapping areas, through feature extraction, feature point matching, homography matrix calculation, perspective transformation, and image fusion steps, ultimately reserving a regular blank area in the middle. This diagram is used to illustrate the basic concepts and key steps of the task performed by the image stitching module in step one of the present invention.

[0030] Figure 2This is a conceptual diagram of image inpainting provided in an embodiment of the present invention. Through intuitive comparison, the diagram illustrates the goal of image inpainting technology: to intelligently predict and fill in the missing areas within holes in a panoramic background image containing holes, based on the contextual information and semantic content outside the holes, so that the repaired result appears visually and semantically continuous and realistic. This diagram serves to clarify the core problem that step three of the present invention ultimately aims to solve.

[0031] Figure 3 This diagram illustrates the training principle of the improved CR-Fill model provided in this embodiment of the invention. This diagram is one of the core innovations of this invention, detailing the workflow of the auxiliary context reconstruction branch. It includes structures such as the Similarity Encoder, Auxiliary Encoder, and Auxiliary Decoder, as well as the gradient backpropagation path. This diagram explains how the behavioral knowledge of correctly matched feature blocks is distilled into the main generator through the context reconstruction loss (CR Loss), providing a theoretical basis for completely removing the attention mechanism during the deployment phase.

[0032] Figure 4 This is a diagram of the overall architecture of the generator network provided in this embodiment of the invention. It adopts a coarse-to-fine encoder-decoder structure, with the core components sequentially consisting of a coarse-network gated convolutional encoder, a coarse-network gated transposed convolutional decoder, a fine-network gated convolutional encoder, a gated dilated convolutional module, and a fine-network gated transposed convolutional decoder. Gated convolutional layers are used to process the feature differences between the inside and outside of the holes, effectively eliminating artifacts at the repair boundaries. This diagram specifically illustrates the internal flow and processing of image data within the repair module, representing the key architectural embodiment of the efficient inference achieved by this invention.

[0033] Figure 5 is a schematic diagram of the multi-channel image stitching and semantic cropping effect provided in an embodiment of the present invention. The figure shows the panoramic background image generated after processing the original images acquired simultaneously from four channels in two typical scenarios—urban architecture and high-altitude distant views—through image stitching in step one and semantic cropping in step two of the present invention. This verifies that the method can achieve seamless stitching and precise cropping of multi-view, high-parallax scenes without obvious seams, ghosting, uneven lighting, or invalid black borders.

[0034] Figure 6 is a schematic diagram comparing the image restoration effects provided in the embodiments of the present invention. This figure demonstrates the image restoration effect based on the CR-Fill model in step three of the present invention by comparing three sets of original images, panoramic background images with holes, and restored panoramic background images. It verifies that the method can achieve semantically accurate restoration of the cropped panoramic background images with holes, and has high texture and structure restoration accuracy for natural scenes such as forests and mountains, without artifacts, texture misalignment, or structural distortion.

[0035] Figure 7 shows the measured end-to-end processing time of the method provided in this embodiment of the invention. The figure illustrates the entire processing time from receiving four synchronous input images of 1280×720 pixels, through panoramic stitching, semantic cropping, to final image restoration. The measured average end-to-end processing time is only 0.67 seconds, verifying that this method can meet the real-time processing requirements of industrial scenarios. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0037] The implementation of this invention requires a hardware and software operating environment adapted to the aforementioned deep learning-based image stitching and restoration method. The system's hardware platform should include at least one multi-core CPU, no less than 16GB of memory, and a graphics processing unit (GPU) with computing power no less than an Nvidia GeForce RTX 3060 to support efficient inference of the deep learning model. Four high-definition cameras (with a resolution of at least 1280×720 pixels) must be connected to the computing platform via a USB 3.0 or higher speed interface to ensure high-speed and stable transmission of image data. A stable version of Windows 10 or Linux is recommended as the operating system. The software development environment is based on Python 3.8 or later, and built using the PyTorch 1.7.0 or later deep learning framework. Key dependencies include OpenCV 4.5.0 (for image processing) and NumPy. System operation begins in the data preprocessing stage. The OpenCV VideoCapture interface is initialized, and four camera devices are simultaneously opened to synchronously acquire four real-time video streams at a frame rate of no less than 30fps. Each frame of the image is immediately converted from the BGR color space to the RGB color space and normalized, scaling the pixel value range from [0, 255] to [-1, 1] to adapt to the input requirements of subsequent deep learning models.

[0038] The process then proceeds to image stitching. This module receives four preprocessed images and uses the SURF algorithm based on the Hessian matrix as the primary feature point detection method, supplemented by the ORB algorithm, to extract feature points and corresponding feature descriptors from the four images. Next, the feature descriptors of the four images are initially matched using the NNDR method, and erroneous matching point pairs are removed using the RANSAC algorithm. Based on the remaining correctly matched point pairs, the homography matrix between the images is calculated. Then, the four images are projected onto a common coordinate system using the homography matrix, and a multi-band fusion strategy is employed to eliminate seams and lighting differences at the stitching points, ultimately outputting a panoramic background image with a pre-reserved regular blank area in the middle.

[0039] Next, the process moves to semantic cropping. Image augmentation methods are used to crop the panoramic background. Figure 4 Expand the black border by N pixels (N is an integer, e.g., 20) around the perimeter. Then, perform binarization thresholding on the image, setting the effective pixel area of ​​the panoramic background to white and the invalid background to black, generating a binary mask. Initialize a rectangle minRect of the same size as the binary mask, and then continuously perform morphological erosion on it. After each erosion, calculate the difference region sub between minRect and the eroded result; the loop terminates when sub no longer contains any effective pixels, at which point the size of minRect is the maximum inscribed rectangle.

[0040] Next, the process moves on to image inpainting. First, a generative adversarial network (GAN) model based on CR-Fill is constructed. This model consists of a main generator, an auxiliary context reconstruction branch, and a discriminator. The main generator is structured as follows: Figure 4 The diagram shows five core components: a coarse-gated convolutional encoder, a coarse-gated transposed convolutional decoder, a fine-gated convolutional encoder, a dilated gated convolutional module, and a fine-gated transposed convolutional decoder. The core units of the coarse-gated and coarse-gated transposed convolutional encoders and decoders include downsampling gated convolutional layers, which generate intermediate feature maps containing semantic information at different scales during the encoding process. The dilated gated convolutional module's only component is a dilated gated convolutional layer. The core units of the fine-gated and fine-gated transposed convolutional encoders and decoders include upsampling gated convolutional layers. The auxiliary context reconstruction branch is as follows... Figure 3 As shown, this branch includes a similarity encoder and an auxiliary decoder. The discriminator is composed of a multi-layer convolutional encoder module with spectral normalization. The image inpainting process is divided into two phases: model training and deployment. During the model training phase, the similarity encoder of the auxiliary context reconstruction branch receives intermediate layer feature maps generated by the coarse-network gated convolutional encoder from the main generator, treating them as a set of feature blocks. For any feature block within a hole... With any feature block outside the hole ,by , Let represent the feature vectors corresponding to the two feature blocks, and calculate their cosine similarity. For each feature block within the cavity. We use the calculated cosine similarity Reconstruct it as a weighted sum of all feature blocks outside the hole: ,in Representing the outside of the void, Scaling factor The function is used to normalize similarity weights. Reconstructing feature blocks within holes. Combined feature map The auxiliary decoder upsamples the auxiliary image by the auxiliary context reconstruction branch. CR Loss is defined as the difference between the auxiliary image and the true complete panoramic background image. The sum of L1 loss and adversarial loss between them: ,in, Indicates the discriminator, The function is used to stabilize adversarial training. By minimizing the aforementioned CR Loss, the main generator is forced to learn feature representations consistent with the semantics outside the holes, enabling its generated features inside the holes to be accurately reconstructed using features outside the holes. During deployment, the auxiliary context reconstruction branch and discriminator are completely removed, retaining only the trained main generator. The main generator directly receives the panoramic background image containing holes and completes feature encoding and reconstruction through a coarse-network gated convolutional encoder, a coarse-network gated transposed convolutional decoder, a fine-network gated convolutional encoder, a hole-gated convolutional module, and a fine-network gated transposed convolutional decoder, outputting a repaired 100×100 pixel image patch. Then, based on the previously cropped coordinate parameters, the repaired image patch is seamlessly fused back to its corresponding position in the panoramic background image, generating a visually complete, coherent, and semantically consistent high-resolution final image.

[0041] Table 1: Test Results of Core Performance Indicators of the Repair Model Provided in the Embodiments of the Invention The multi-channel image stitching effect, image restoration effect, end-to-end processing time, and core performance indicators of the restoration model of this method are respectively provided in the appendix of the instruction manual. Figure 5 To be continued Figure 7 Table 1. Appendix Figure 5The method was verified to be able to seamlessly stitch and accurately semantically crop multiple images in various scenes, such as urban buildings and high-altitude distant views, without obvious stitching seams, ghosting, or uneven lighting. Figure 6 verifies that the method, based on the CR-Fill model, can achieve accurate semantic restoration of the cropped panoramic background image with holes, and has high texture and structure restoration accuracy for natural scenes such as woodlands and mountains, without artifacts, texture misalignment, or other distortion problems. Figure 7 The actual test showed that the average processing time of the entire end-to-end process from receiving four input images to finally outputting a complete panoramic image was only 0.67s, which can meet the real-time requirements of industrial scenarios. Table 1 shows the test data of structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) of the restoration model of the present invention in typical scenarios. Under normal scenarios, the SSIM of the restoration results is stably maintained above 0.96 and the PSNR exceeds 29.5dB, which verifies the excellent restoration accuracy and image restoration capability of the present method.

[0042] The entire system works collaboratively in the aforementioned manner, forming a closed loop from four-channel image acquisition to the final panoramic image output. To ensure long-term stable operation, a memory management mechanism is integrated, effectively preventing memory overflow by overwriting the earliest result after setting a processing frame limit (e.g., 50 frames). Furthermore, a boot-up script can be configured to ensure the system automatically starts running after the computing platform boots, meeting the needs of unattended application scenarios. Through the above specific implementation methods, this invention successfully integrates image stitching, semantic cropping, and image restoration, providing an efficient, robust, and practical integrated solution.

Claims

1. A method for real-time stitching and restoration of multi-channel images based on the CR-Fill model, characterized in that, The method includes the following steps: Step 1: Image stitching: After preprocessing the synchronous raw images from multiple cameras, feature extraction, feature point matching, and image fusion are performed sequentially to generate a panoramic background image with a reserved regular blank area in the middle. The specific sub-steps are as follows: Step 2: Semantic Pruning The panoramic background image obtained in step one is post-processed. By generating a binarized mask and performing morphological erosion operations, the maximum inscribed rectangle of the effective content area is located, and the panoramic background image with a hole in the center is cropped out. Step 3: Image Restoration First, a context-based reconstruction and filling (CR Fill) generative adversarial network model is constructed to perform semantic inpainting on the image patches output in step two. The CR Fill-based generative adversarial network model consists of three modules: a main generator, an auxiliary context reconstruction branch, and a discriminator.

2. The method for real-time stitching and restoration of multi-channel images based on the CR-Fill model according to claim 1, characterized in that, Step one includes the following steps: Step 1.1, Feature Extraction The SURF algorithm based on the Hessian matrix is ​​adopted as the primary feature point detection method, supplemented by the directional FAST and rotation BRIEF algorithms based on the FAST corner points of the acceleration segment test, to extract feature points and corresponding feature descriptors from the original image. Step 1.2, Feature Point Matching First, feature descriptors of different images are initially matched based on the nearest neighbor and second nearest neighbor distance ratio (NNDR) method. Then, the random sampling consensus (RANSAC) algorithm is used to remove mismatched point pairs. Finally, the homography matrix between images is calculated based on the remaining correct matching point pairs after removal, thereby establishing the perspective transformation relationship between images. Step 1.3, Image Fusion The calculated homography matrix is ​​used to project each image onto the same coordinate system. A multi-band fusion strategy is then employed to eliminate seams and lighting differences at the stitching points. Specifically, the multi-band fusion strategy first constructs a Laplacian pyramid of the image, decomposing it into sub-bands of different frequencies. Next, a large-scale weighted fusion is performed at the low-frequency level of the pyramid to eliminate lighting differences, while a small-scale weighted fusion is performed at the high-frequency level to preserve texture details. Finally, the image is reconstructed through inverse Laplacian pyramid transformation to obtain a panoramic background image with a pre-reserved regular blank area in the middle.

3. The method for real-time stitching and restoration of multi-channel images based on the CR-Fill model according to claim 1, characterized in that, Step two includes the following steps: Step 2.1: Augmentation Processing and Binary Mask Generation An image augmentation method is used to extend the panoramic background image with an integer number of black borders to prevent boundary overflow; then, a binarization thresholding process is performed on the image to set the effective pixel areas in the panoramic background image to white and the invalid background to black, thus generating a binary mask. Step 2.2: Calculation of the maximum inscribed rectangle Initialize a rectangular region of the same size as the binary mask, and continuously perform morphological erosion operations on the rectangular region; in each morphological erosion operation, calculate the difference between the current rectangular region and the eroded result; when the difference region no longer contains valid pixels, stop the morphological erosion operation, and the remaining rectangular region is the maximum inscribed rectangle. Step 2.3, Image Cropping Based on the coordinate parameters of the largest inscribed rectangle calculated in step 2.2, a hole of standard size is cropped from the panoramic background image generated in step 1; this panoramic background image containing the hole is used as an image block and becomes the input for subsequent image restoration steps.

4. The method for real-time stitching and restoration of multi-channel images based on the CR-Fill model according to claim 1, characterized in that, Step three includes the following steps: I. Main Generator The main generator, in the order of image data processing, includes a coarse-network gated convolutional encoder, a coarse-network gated transposed convolutional decoder, a fine-network gated convolutional encoder, a hole-gated convolutional module, and a fine-network gated transposed convolutional decoder. The core components of the coarse-gated convolutional encoder and coarse-gated transposed convolutional decoder include downsampling gated convolutional layers, which are used to extract high-level semantic features of the image. During this encoding process, intermediate feature maps containing semantic information at different scales are generated. The only component of the dilated gated convolutional module is the dilated gated convolutional layer, which is used to expand the receptive field and capture long-range contextual information without reducing the resolution. The core components of the fine-gated convolutional encoder and fine-gated transposed convolutional decoder include upsampling gated convolutional layers, which are used to restore the feature maps to the original resolution. II. Auxiliary Context Restructuring Branch The components of this branch are a similarity encoder and an auxiliary decoder; III. Discriminator The discriminator is composed of a multi-layer convolutional encoder module with spectral normalization. Its core function is to distinguish between real and complete images and images restored by the main generator. Through adversarial training, the reverse constraint generator is continuously improved to enhance the realism, texture consistency and semantic rationality of the restored images. Image restoration is divided into model training and deployment; the specific sub-steps are as follows: Step 3.1, Model Training Step 3.1.1: The similarity encoder of the auxiliary context reconstruction branch receives the intermediate layer feature map generated by the coarse-network gated convolutional encoder from the main generator, treating it as a set of feature blocks; for any feature block within the hole... With any feature block outside the hole ,by , Let represent the feature vectors corresponding to the two feature blocks, and calculate their cosine similarity. : in, and Represents a feature block. Represents the eigenvector. Represents the dot product. The norm of a vector; For each feature block within the cavity Based on the aforementioned cosine similarity weights, the feature blocks within the holes are reconstructed as follows: : in, Represents the area outside the cavity. Scaling factor The function is used to normalize similarity weights; Step 3.1.2: Reconstruct feature blocks within the hole Combined feature map The auxiliary decoder upsamples the auxiliary image by the auxiliary context reconstruction branch. Context reconstruction loss is defined as the sum of the auxiliary image and the true complete panoramic background image. The sum of L1 loss and adversarial loss between them: in, Indicates the discriminator, The function is used to stabilize adversarial training; By minimizing the aforementioned CR Loss, the master generator is forced to learn feature representations consistent with the semantics outside the holes, enabling the generated features inside the holes to be accurately reconstructed through the features outside the holes. Step 3.2, Deployment and Use: During the deployment phase, the auxiliary context reconstruction branch and discriminator are completely removed, leaving only the trained main generator. The main generator directly receives the panoramic background image with holes output from step two, and completes feature encoding and reconstruction through a coarse-network gated convolutional encoder, a coarse-network gated transposed convolutional decoder, a fine-network gated convolutional encoder, a hole-gated convolutional module, and a fine-network gated transposed convolutional decoder, outputting the repaired complete image patch. Then, based on the coordinate parameters cropped in step two, the repaired image patch is seamlessly fused back to the corresponding position in the panoramic background image. Due to the removal of the auxiliary context reconstruction branch, the generation process is significantly accelerated.

Images