A time domain splicing panoramic camera system and method based on a visual language large model

Abstract

The application discloses a kind of time domain splicing panoramic camera systems and methods based on visual language big model, belong to computer vision and image processing technical field.The application acquires time domain image sequence by multi-lens camera array, utilizes pre-trained visual language big model VLM to carry out depth semantic analysis to multiple frames of images, generates the structured splicing guide signal including overlapping area semantic segmentation result, motion estimation parameter and splicing seam position suggestion;Time domain feature alignment and overlapping area matching are carried out based on splicing guide signal;Through the weighted cost function of fusion semantic difference graph, gradient energy graph and time domain constraint energy, adaptive splicing seam optimization is carried out;Finally, through the triple time domain consistency constraint of time domain weighted median filtering, bidirectional optical flow detection and VLM semantic consistency check, high-quality panoramic image sequence is output.The application solves the problem of poor splicing quality of traditional method under low texture, illumination change and dynamic scene, and the splicing success rate reaches more than 96%, and time domain PSNR fluctuation is reduced to ±0.8dB.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, specifically to a temporal stitching panoramic camera system and method based on a Vision-Language Model (VLM), and particularly to achieving high-quality stitching and fusion of multiple panoramic images in the temporal dimension by utilizing the semantic understanding capabilities of the multimodal VLM. Background Technology

[0002] Panoramic cameras capture images from different perspectives through multiple lenses and stitch these images together to create a panoramic image covering 360 degrees. They are widely used in security monitoring, virtual reality, autonomous driving, and robot navigation. Traditional panoramic stitching methods primarily rely on feature point matching (such as SIFT, SURF, and ORB algorithms) for image alignment and stitching.

[0003] However, traditional methods have the following shortcomings: (1) Feature point matching is prone to failure in low-texture, repetitive texture or drastic lighting changes, resulting in misalignment or failure of splicing; (2) For multi-frame splicing in the temporal domain, traditional methods lack an understanding of scene semantics and are difficult to handle temporal inconsistencies such as dynamic object occlusion and motion blur; (3) The fusion quality at the splicing seam is limited by the manually designed fusion strategy and is difficult to adaptively handle splicing seam problems in different scenes.

[0004] In recent years, deep learning methods have been introduced into the field of image stitching. Patent CN107580175A discloses a single-lens panoramic stitching method that achieves stitching through camera calibration and image transformation; however, this method relies on precise camera calibration parameters and does not involve semantic understanding. US Patent US20220164922A1 discloses an image stitching method and apparatus based on deep learning, utilizing convolutional neural networks to learn image registration parameters, but it still has shortcomings in temporal consistency and semantic understanding. Patent CN116645496B discloses a dynamic surround-view stitching and stabilization method for trailers based on mesh deformation, introducing temporal Gaussian weighted smoothing; however, it only relies on geometric deformation and optical flow tracking, without utilizing semantic understanding to guide stitching seam optimization and dynamic object processing.

[0005] Regarding semantic seam optimization, the published paper "Semantic Aware Stitching for Panorama" (MDPI Sensors, 2024) proposes a semantically aware panorama stitching algorithm that uses semantic segmentation results to guide seam avoidance. However, it only uses conventional semantic segmentation networks (such as DeepLab) and does not employ large-scale visual language models for deep semantic understanding and motion estimation, nor does it involve temporal consistency constraints. The published paper "Deep SeamPrediction for Image Stitching Based on Selection" (arXiv, 2023) proposes using deep learning to predict seam locations, but it uses a pure visual CNN model and does not involve multimodal semantic reasoning.

[0006] Regarding large-scale visual-language models, patent CN119227744B discloses a multimodal large-scale model system that integrates vision and language, combining visual encoders of DINOv2 and SigLIP. However, this system is applied to general vision tasks and does not cover panoramic image stitching applications. Patent CN118865218A discloses an image recognition method based on a large-scale visual-language model, primarily applied to news image classification, and similarly does not involve panoramic stitching technology. The published STER-VLM framework (ICCV2025) explores the application of VLM in spatiotemporal augmentation tasks, but focuses on video understanding rather than image stitching.

[0007] In addition, patent ZL202511860961.4 discloses a multi-camera 360-degree panoramic image stitching method based on depth information guidance, which obtains scene depth information through a depth sensor to assist in stitching. However, this method relies on additional depth sensor hardware and does not involve semantic guidance and temporal consistency optimization of a large visual language model.

[0008] In summary, the existing technologies have the following gaps: (1) There is no systematic solution that applies the semantic understanding and reasoning capabilities of the Visual Language Model (VLM) to the field of panoramic camera stitching; (2) Existing deep learning-based stitching methods do not utilize the structured stitching guidance signals (including semantic segmentation, motion estimation, and stitching seam suggestions) generated by VLM to uniformly guide the entire stitching process; (3) There is a lack of a temporal consistency stitching framework under VLM semantic constraints, which cannot simultaneously solve comprehensive problems such as the difficulty of matching low-texture scenes, interference from dynamic objects, and temporal flickering. Therefore, there is an urgent need for a technical solution that can utilize the semantic understanding capabilities of VLM to improve the temporal stitching quality of panoramic cameras. Summary of the Invention

[0009] The technical problem to be solved by this invention is: how to overcome the problem of poor stitching quality of traditional panoramic stitching methods under conditions such as low texture, lighting changes, and dynamic scenes, and how to achieve high-quality and consistent stitching of multi-frame panoramic images in the temporal dimension.

[0010] To address the aforementioned technical problems, the present invention provides a temporal stitching panoramic camera system and method based on a large visual language model. The core innovation of this invention lies in the first-time introduction of a pre-trained large visual language model (VLM) into the field of panoramic stitching. By utilizing the collaborative work of the visual encoder and language model decoder within the VLM, a structured stitching guidance signal is generated, which uniformly guides the entire process of temporal feature alignment, adaptive stitching seam optimization, and temporal consistency constraints.

[0011] Option 1: A temporal stitching panoramic camera method based on a large visual language model, comprising the following steps: S1. Multi-lens image acquisition: Using a multi-lens camera array consisting of 4 or more fisheye or wide-angle cameras (each camera has a field of view ≥185 degrees), original images are acquired from different perspectives and continuously acquired at a preset frame rate (e.g., 30fps) to form a temporal image sequence. S2. Image preprocessing: Denoising, distortion correction based on the Brown-Conrady distortion model, spherical projection coordinate transformation, histogram matching exposure consistency adjustment and other preprocessing operations are performed on the acquired original image sequence to generate a preprocessed image frame sequence. S3. VLM Semantic Analysis: The preprocessed image frame sequence is input into a pre-trained Visual Language Model (VLM), which includes a visual encoder and a language model decoder. The visual encoder employs a dual-stream architecture fusing ViT-Large and CLIPViT-L / 14: the ViT stream extracts spatial structure features, and the CLIP stream extracts semantic alignment features. The two streams are fused through a cross-attention mechanism to generate a 1024-dimensional visual semantic feature vector. The language model decoder is based on an LLaMA architecture and generates structured splicing guidance signals through dedicated prompts, including: (a) semantic segmentation masks for overlapping regions (classifying pixels into overlapping areas, non-overlapping areas, and moving objects); (b) scene motion estimation parameters (including global motion vector fields and local motion estimates); and (c) splicing seam location suggestions (recommending the optimal path through low-importance regions based on semantic segmentation results). S4. Temporal Feature Alignment and Matching: Based on the stitching guidance signal output by the VLM, temporal feature alignment is performed on adjacent frame images. An inter-frame semantic similarity matrix S(i,j)=cos(f_i,f_j) is constructed using the semantic feature vectors output by the VLM. The dense motion field between frames is estimated using the RAFT optical flow method, and the homography transformation matrix H is estimated using the RANSAC algorithm. Abnormal matching points corresponding to dynamic objects are then removed under the semantic constraints of the VLM. S5. Adaptive stitching seam optimization and fusion: Based on the semantic difference map E_sem (based on VLM semantic segmentation results, high energy values ​​are assigned to semantically important regions), the gradient energy map E_grad, and the temporal constraint energy E_temp, a fusion cost function E(x,y)=α·E_sem+β·E_grad+γ·E_temp (α=0.4,β=0.3,γ=0.3) is constructed. The optimal stitching seam path is solved by dynamic programming, and a 5-layer Laplacian pyramid multi-band fusion strategy is adopted to perform image fusion on both sides of the stitching seam. S6. Temporal Consistency Constraints and Output: Temporal consistency constraints are applied to the fused panoramic image sequence: Temporal weighted median filtering is performed on 5 consecutive frames (weights decrease with time distance: 1.0, 0.8, 0.6, 0.4, 0.2), bidirectional optical flow is used to detect temporally inconsistent regions and adaptive temporal smoothing constraints are applied, VLM semantic consistency check is introduced to detect and correct stitching artifacts caused by dynamic objects, and finally a high-quality panoramic image sequence with temporal consistency is output.

[0012] Option 2: According to the method described in Option 1, the training process of the Visual Language Model (VLM) in step S3 includes: the first stage is to perform image-text comparison learning pre-training, using image-text pairs as training samples to maximize the similarity between the correct description and image features; the second stage is to perform cross-attention fusion training, learning the fusion weights of ViT and CLIP dual-stream features; the third stage is to perform stitching-specific fine-tuning, freezing the visual encoder parameters, training only the stitching guidance signal generation module, and using a training dataset containing multiple sets of panoramic stitching scenes (each set includes multi-view images, semantic segmentation annotations, stitching seam location annotations, and stitching quality scores).

[0013] Option 3: According to the method described in Option 1, the temporal constraint energy E_temp in step S5 is calculated as follows: E_temp(x,y)=Σ_{k=1}^{K}|I_t(x,y)-I_{tk}(H_k(x,y))|, where K is the number of temporal backtracking frames (K=3 in this embodiment), I_t is the current frame image, and H_k is the homography transformation from the k-th frame to the current frame. Temporal continuity is ensured by penalizing the stitching seam path to cross the temporal inconsistency region.

[0014] Option 4: A temporal stitching panoramic camera device based on a large visual language model, comprising: a multi-lens acquisition module for acquiring multi-view original image sequences through a fisheye or wide-angle camera array; an image preprocessing module for denoising, distortion correction, and exposure consistency adjustment of the original images; a VLM semantic analysis module, including a ViT+CLIP dual-stream visual encoder and an LLaMA language model decoder, for extracting visual semantic features and generating structured stitching guidance signals; a temporal stitching fusion module for performing temporal feature alignment, overlapping region matching, stitching seam optimization, and multi-band fusion based on the stitching guidance signals; a temporal consistency optimization module for applying temporal consistency constraints to the stitched panoramic images; and a panoramic output module for outputting a temporally consistent panoramic image sequence.

[0015] Option 5: An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any one of Options 1 to 3.

[0016] Option 6: A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of Options 1 to 3.

[0017] Beneficial effects Compared with the prior art, the technical solution of this invention has the following advantages: (1) This invention introduces VLM semantic understanding into panoramic stitching for the first time: Unlike existing stitching methods based on conventional CNNs or traditional feature matching, this invention utilizes the collaborative work of the ViT+CLIP dual-stream visual encoder and the LLM language model decoder in VLM to achieve a deep semantic understanding of scene content. VLM can not only identify the object categories in the image, but also understand the spatiotemporal relationships of the scene through language reasoning, thereby generating structured stitching guidance signals such as semantic segmentation, motion estimation, and stitching seam suggestions. Even in challenging scenes such as low texture, repetitive texture, or varying lighting, the stitching success rate is increased from approximately 75% in traditional methods to over 96%.

[0018] (2) Structured splicing guidance signal unifies the entire process: Unlike existing technologies (such as the semantic splicing method of MDPI Sensors 2024, which only uses a semantic segmentation network to guide splicing seam avoidance), this invention generates a structured guidance signal containing semantic segmentation mask, motion estimation parameters and splicing seam position suggestions in one go through the language model decoder of VLM, which drives the entire process of feature alignment, splicing seam optimization and temporal constraint in a unified manner, avoiding the information inconsistency problem caused by the independent operation of each module.

[0019] (3) Significantly improved temporal consistency: By combining VLM semantic constraints with temporal filtering and bidirectional optical flow detection, the PSNR fluctuation between frames of the fused panoramic image sequence is reduced from ±3.2dB to ±0.8dB in the traditional method, and the average SSIM value is increased from 0.91 to 0.97, effectively eliminating splicing flicker and jitter.

[0020] (4) Adaptive seam optimization: The adaptive seam optimization algorithm based on VLM semantic difference map can automatically avoid semantically important regions (such as faces, text, moving objects, etc.), and the seam invisibility rate is increased from about 60% in traditional methods to more than 93%.

[0021] (5) End-to-end high-efficiency processing: The VLM inference and stitching process adopts a cascaded pipeline design. With GPU acceleration, the single-frame stitching latency is less than 50ms, and it supports real-time panoramic stitching output of more than 20fps.

[0022] (6) Wide applicability: The method of the present invention does not rely on specific camera calibration parameters or additional depth sensor hardware, and can be adapted to different focal lengths and different numbers of lens combinations, and is applicable to a variety of application scenarios such as security monitoring, VR / AR, and autonomous driving. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall system architecture of the present invention; Figure 2 This is a schematic diagram of the internal structure of the VLM dual-stream visual encoder and LLM decoder of the present invention; Figure 3 This is a schematic diagram of the temporal feature alignment and matching process of the present invention; Figure 4 This is a schematic diagram of the adaptive splicing seam optimization and multi-band fusion process of the present invention; Figure 5 This is a schematic diagram of the temporal consistency constraint optimization process of the present invention; Figure 6 This is a schematic diagram of the VLM model training process of the present invention; Figure 7 This is a schematic diagram comparing the performance of the method of the present invention with that of a traditional method. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0025] Example 1: Temporal stitching panoramic camera method based on a large visual language model like Figure 1 As shown, this embodiment provides a temporal stitching panoramic camera method based on a large visual language model, including the following steps: S1. Multi-lens image acquisition: Raw images are acquired from different perspectives using a multi-lens camera array consisting of four fisheye cameras (each with a field of view ≥185 degrees). The camera array continuously acquires images at a frame rate of 30fps, forming a temporal image sequence. Each camera has a resolution of 4000×3000 pixels, and the acquired raw images are transmitted to an embedded processing unit (such as NVIDIA Jetson AGX Orin) via a MIPI interface.

[0026] S2. Image Preprocessing: The following preprocessing operations are performed on the acquired raw image sequence: (a) Lens distortion correction is performed based on the pre-calibrated camera intrinsic parameters, and the Brown-Conrady distortion model is used to eliminate the radial and tangential distortion of the fisheye lens; (b) Based on the extrinsic parameter calibration results between multiple cameras, the images of each camera are projected onto a unified spherical coordinate system; (c) The exposure consistency of the images of each camera is adjusted by the histogram matching method; (d) Noise reduction is performed using a 3×3 Gaussian filter.

[0027] S3, VLM semantic analysis: such as Figure 2 As shown, the preprocessed multi-frame images are input into the pre-trained Visual Language Model (VLM). In this embodiment, the VLM adopts the following architecture: the visual encoder uses a dual-stream architecture fusing ViT-Large (approximately 300M parameters) and CLIP ViT-L / 14, where the ViT stream is responsible for extracting spatial structure features, and the CLIP stream is responsible for extracting semantic alignment features. The two stream features are fused through a cross-attention mechanism to generate a visual semantic feature vector with a dimension of 1024. The language model decoder is based on the LLaMA-7B architecture. After receiving the visual semantic feature vector, it generates a structured splicing guidance signal guided by a special splicing task prompt. The splicing guidance signal includes: (a) a semantic segmentation mask for overlapping regions, classifying image pixels into three categories: overlapping regions, non-overlapping regions, and moving objects; (b) scene motion estimation parameters, including a global motion vector field and local motion estimation; and (c) a splicing seam location suggestion, recommending a path through a low-importance region for the splicing seam based on the semantic segmentation results.

[0028] S4. Temporal feature alignment and matching: such as Figure 3As shown, temporal feature alignment of adjacent frames is performed based on the stitching guidance signal output by the VLM. Specifically, this includes: (a) calculating the inter-frame semantic similarity matrix S(i,j)=cos(f_i,f_j) using the semantic feature vector output by the VLM; (b) estimating the dense motion field between frames based on the semantic similarity matrix and the RAFT optical flow method; (c) estimating the homography transformation matrix H between adjacent frames from the dense motion field using the RANSAC algorithm; and (d) under the semantic constraints of the VLM, marking the pixel regions corresponding to moving objects as abnormal matching points and removing them to ensure the robustness of homography estimation.

[0029] S5, Adaptive seam optimization and integration: such as Figure 4 As shown, the stitching seam optimization is performed according to the following steps: (a) Based on the VLM semantic segmentation results, calculate the semantic difference map E_sem(x,y) and assign high energy values ​​to semantically important regions; (b) Calculate the image gradient energy map E_grad(x,y)=|∇I_x|+|∇I_y|; (c) Calculate the temporal constraint energy E_temp(x,y)=Σ_{k=1}^{3}|I_t(x,y)-I_{tk}(H_k(x,y))|; (d) Construct the fusion cost function E(x,y)=0.4·E_sem+0.3·E_grad+0.3·E_temp; (e) Solve the minimum cost path from top to bottom on the cost map through dynamic programming as the optimal stitching seam; (f) Adopt a 5-layer Laplacian pyramid multi-band fusion strategy on both sides of the stitching seam, and reconstruct the final stitched image after independent fusion of each frequency band.

[0030] S6. Temporal Consistency Constraints and Output: (e.g.) Figure 5 As shown, temporal consistency constraints are applied to the fused panoramic image sequence: (a) Temporal weighted median filtering is performed on 5 consecutive panoramic images, with weights decreasing in temporal distance (1.0, 0.8, 0.6, 0.4, 0.2); (b) Bidirectional optical flow is used to detect temporal inconsistencies between adjacent frames, and adaptive Gaussian temporal smoothing constraints are applied to the detected inconsistencies; (c) The temporally filtered image is input again into VLM for semantic consistency verification, detecting and correcting stitching artifacts caused by dynamic objects. The final output is a high-quality panoramic image sequence with temporal consistency.

[0031] Example 2: VLM Model Training like Figure 6 As shown, the training process of the Visual Language Model (VLM) used in this invention includes three stages: Phase 1 (Image-Text Contrast Learning Pre-training): CLIP-style contrast learning pre-training is performed using a large-scale image-text pair dataset to maximize the cosine similarity between correct text descriptions and corresponding image features, while minimizing the similarity of mismatched text-image pairs. ViT-Large also undergoes self-supervised pre-training (e.g., MAE) on an image classification dataset to learn spatial structure feature extraction capabilities.

[0032] Phase 2 (Cross-Attention Fusion Training): Based on freezing the pre-trained parameters of the two streams, train the cross-attention fusion layer between the ViT stream and the CLIP stream to learn how to optimally fuse spatial structure features and semantic alignment features to generate a 1024-dimensional unified visual semantic feature vector.

[0033] Phase 3 (Fine-tuning for stitching): Construct a dedicated training dataset containing multiple sets of panoramic stitching scenes. Each set includes multi-view images, corresponding semantic segmentation annotations, stitching seam location annotations, and stitching quality scores. Freeze the visual encoder parameters and train only the stitching guidance signal generation module (i.e., the stitching-specific output head in the LLaMA decoder). Fine-tune the dataset using the stitching quality scores as reward signals.

[0034] Example 3: Performance Comparison Experiment To verify the technical effect of the present invention, comparative experiments were conducted in three typical scenarios: (1) Scenario 1 - Indoor office environment: low texture wall + multiple people moving; (2) Scenario 2 - Outdoor street scene: strong light change + vehicle movement; (3) Scenario 3 - Night scene: low light + light interference.

[0035] The comparison methods include: traditional SIFT feature matching and stitching methods, traditional deep learning stitching methods based on CNNs (such as the method disclosed in US20220164922A1), semantically aware stitching methods (such as the method in MDPI Sensors 2024), and the method of this invention. Evaluation metrics include PSNR, SSIM, stitching success rate, and stitch seam invisibility rate.

[0036]

[0037] Table 1 Comparison of splicing performance of different methods The experimental results show that the method of this invention significantly outperforms traditional methods and existing deep learning methods in all metrics. In particular, regarding splicing success rate and temporal consistency, the method of this invention improves upon the traditional SIFT method by 20.9 percentage points and reduces PSNR fluctuation by 75%, respectively. Compared with semantically aware splicing methods, this invention improves the splicing seam invisibility rate by 12.2 percentage points, demonstrating the significant advantage of VLM semantic reasoning in splicing seam optimization.

[0038] Example 4: The device of the present invention This embodiment provides a temporal stitching panoramic camera device based on a large visual language model, including: (1) a multi-lens acquisition module, consisting of four fisheye cameras with a field of view ≥185 degrees, connected to the processing unit via a MIPI interface; (2) an image preprocessing module, running on an FPGA, implementing distortion correction, spherical projection transformation, exposure adjustment, and noise reduction; (3) a VLM semantic analysis module, running on an embedded GPU (such as NVIDIA Jetson AGX Orin), including a ViT+CLIP dual-stream visual encoder and an LLaMA decoder, generating structured stitching guidance signals; (4) a temporal stitching fusion module, performing semantic feature alignment, stitching seam optimization, and multi-band fusion on the GPU; (5) a temporal consistency optimization module, implementing temporal weighted median filtering, bidirectional optical flow temporal inconsistency detection, and VLM semantic consistency verification; and (6) a panoramic output module, outputting a temporally consistent panoramic image sequence via an HDMI or network interface.

[0039] Example 5: VLM Online Adaptation Mechanism To further enhance the system's robustness in complex and changing environments, this embodiment adds a VLM online adaptation mechanism based on Embodiment 1. For example... Figure 1 As shown in the feedback loop, the stitching quality evaluation module monitors the quality metrics of the output panoramic image in real time (such as stitch seam visibility score and temporal consistency index). When the quality metrics fall below a preset threshold, it automatically triggers online fine-tuning of the VLM decoder: using the stitching result of the current frame and the evaluation feedback as training samples, it updates the parameters of the stitching-specific output head of the LLaMA decoder with a low learning rate. This online adaptation mechanism enables the system to automatically adjust its stitching strategy when facing new scenes (such as previously unseen texture patterns or motion types), continuously maintaining high stitching quality.

[0040] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A temporal stitching panoramic camera method based on a large visual language model, characterized in that, The method includes: S1. Acquire original images from different perspectives using a multi-lens camera array, and continuously acquire them at a preset frame rate to form a temporal image sequence; S2. Preprocess the time-domain image sequence, including denoising, distortion correction and exposure consistency adjustment, to generate a preprocessed image frame sequence; S3. Input the preprocessed image frame sequence into the pre-trained visual language large model (VLM). The VLM includes a visual encoder and a language model decoder. The visual encoder uses a dual-stream architecture that integrates ViT and CLIP to extract visual semantic feature vectors of the image. The language model decoder generates a structured stitching guidance signal based on the visual semantic feature vectors. The stitching guidance signal includes semantic segmentation results of overlapping regions, scene motion estimation parameters, and stitching seam position suggestions. S4. Based on the stitching guidance signal, perform temporal feature alignment on adjacent frame images, and use semantic consistency constraints to perform overlapping region matching and homography deformation estimation. S5. Construct a fusion cost function based on the weighted sum of semantic difference map, gradient energy map and temporal constraint energy, solve the optimal stitching seam path through dynamic programming, and perform image fusion on both sides of the stitching seam using a multi-band fusion strategy. S6. Apply temporal consistency constraints to the fused panoramic image sequence, use optical flow and temporal filtering to suppress flicker and jitter between adjacent frames, and correct stitching artifacts through VLM semantic consistency verification to output a temporally consistent panoramic image sequence.

2. The method according to claim 1, characterized in that, The training process of the Visual Language Model (VLM) described in S3 includes: constructing a training dataset containing multiple sets of panoramic stitching scenes, each set of data including multi-view images, corresponding semantic segmentation annotations, stitching seam location annotations, and stitching quality scores; adopting a contrastive learning strategy, using image-text pairs as training samples, to maximize the similarity between the correct description and image features; in the stitching-specific fine-tuning stage, freezing the visual encoder parameters and training only the stitching guidance signal generation module.

3. The method according to claim 1, characterized in that, The temporal feature alignment described in S4 includes: constructing an inter-frame semantic similarity matrix using the semantic feature vectors output by VLM; estimating the dense motion field between frames based on the semantic similarity matrix and combining it with optical flow; estimating the homography transformation matrix between adjacent frames using the RANSAC algorithm; and removing matching points corresponding to dynamic objects under VLM semantic constraints.

4. The method according to claim 1, characterized in that, The fusion cost function described in S5 is: E(x,y) = α·E_sem(x,y) + β·E_grad(x,y) + γ·E_temp(x,y), where E_sem is the semantic difference energy based on the VLM semantic segmentation result, E_grad is the image gradient energy, E_temp is the temporal constraint energy, and α, β, and γ are adjustable weight parameters.

5. The method according to claim 4, characterized in that, The temporal constraint energy E_temp is calculated as follows: E_temp(x,y) = Σ_{k=1}^{K}|I_t(x,y) - I_{tk}(H_k(x,y))|, where K is the number of temporal backtracking frames, I_t is the current frame image, and H_k is the homography transformation from the k-th frame to the current frame. Temporal continuity is ensured by penalizing the stitching seam path through the temporal inconsistency region.

6. The method according to claim 1, characterized in that, The temporal consistency constraints described in S6 include: performing temporal median filtering on the stitched panoramic image sequence to suppress abnormal pixels in isolated frames; using bidirectional optical flow to detect temporally inconsistent regions and applying adaptive temporal smoothing constraints to these regions; and introducing VLM semantic consistency verification to detect and correct stitching artifacts caused by dynamic objects.

7. The method according to claim 1, characterized in that, It also includes a VLM online adaptation mechanism: real-time monitoring of the stitching quality index of the output panoramic image, and when the quality index is lower than a preset threshold, online fine-tuning of the stitching output head of the VLM decoder is performed using the stitching result of the current frame and evaluation feedback.

8. A temporal stitching panoramic camera device based on a large visual language model, characterized in that, include: The system includes a multi-lens acquisition module for acquiring raw image sequences from multiple perspectives using fisheye or wide-angle camera arrays; an image preprocessing module for denoising, distortion correction, and exposure consistency adjustment of the raw images; a VLM semantic analysis module, comprising a visual encoder and a language model decoder, for extracting visual semantic features and generating structured stitching guidance signals; a temporal stitching and fusion module for performing temporal feature alignment, overlapping region matching, stitching seam optimization, and image fusion based on the stitching guidance signals; and a temporal consistency optimization module for applying temporal consistency constraints to the stitched panoramic image. The panoramic output module is used to output a panoramic image sequence that is consistent in the time domain.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the method described in any one of claims 1-7.

Images