High-speed high-dynamic range color panoramic image reconstruction method and system
By deploying pulse and color cameras side-by-side, and combining adaptive geometric models and image editing techniques, the quality problem of panoramic image reconstruction in high-speed rotating scenes was solved, and high dynamic range color panoramic imaging was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING JIAOTONG UNIV
- Filing Date
- 2026-01-16
- Publication Date
- 2026-06-09
AI Technical Summary
In high-speed rotating scenes, traditional frame-based color cameras struggle to achieve high frame rates and high dynamic range imaging, leading to problems such as misalignment, motion blur, and inconsistent exposure in panoramic images. Existing methods are unable to achieve high-quality color panoramic image reconstruction in high-speed rotating scenes.
High frame rate, high dynamic range pulse cameras and traditional color cameras are deployed side by side. The pulse camera recovers high dynamic range, low blur panoramic structural information of high-speed rotating scenes, while the color camera is introduced to obtain color priors of the real scene. Adaptive geometric models and image editing models are used to generate high dynamic range color panoramic images.
It achieves high dynamic range color panoramic imaging with continuous structure and consistent color, overcoming the limitations of traditional imaging systems in high-speed, high-dynamic scenes, and improving the visual quality and overall effect of panoramic images.
Smart Images
Figure CN122175776A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image reconstruction technology, specifically to a high-speed, high dynamic range color panoramic image reconstruction method and system. Background Technology
[0002] Acquiring high frame rate, high dynamic range (HDR) panoramic images in high-speed rotating scenes presents significant challenges. Traditional frame-based color cameras are limited by the inherent trade-off between shutter speed, frame rate, and dynamic range. Increasing the frame rate usually requires shortening the exposure time, leading to a decrease in signal-to-noise ratio and image quality degradation. Meanwhile, the limited dynamic range makes it difficult to simultaneously preserve highlight and shadow details, resulting in localized overexposure or underexposure in the stitched panoramic image. Furthermore, in high-speed rotating scenes, panoramic images are prone to misalignment, motion blur, and inconsistent exposure during stitching, severely impacting the reconstruction quality and visual effect of the panoramic image.
[0003] In recent years, pulse cameras, with their unique asynchronous imaging mechanism, have provided new insights into visual perception in high-speed, high-dynamic scenes. Pulse cameras can output "events" (single-bit pulse streams) of scene brightness changes at sampling rates exceeding 20,000 Hz, offering advantages such as high frame rate, high dynamic range, and low data redundancy. They effectively capture scene details in high-speed motion, unaffected by motion blur. However, pulse cameras output single-bit pulse sequences, which cannot directly form images. Previous methods typically rely on event accumulation within a time window or complex optimization algorithms to recover images. But in high-speed rotating scenes, events captured by pulse cameras in a very short time come from different viewpoints. Directly accumulating these events over time can easily lead to aliasing of multi-view information, making it difficult to directly extend to existing pulse image reconstruction for panoramic images. Furthermore, pulse cameras typically produce monochrome imaging, and their reconstruction results lack color information, failing to meet the practical application requirements of color panoramic imaging. Previous methods typically involve spatially aligning images acquired by pulse cameras and ordinary color cameras at the feature level, and then fusing the color image information into the pulse reconstruction result to obtain a color image. However, these methods rely on precise matching and complex alignment calculations, which make it difficult to guarantee alignment accuracy and computational complexity in large field of view and high-speed scenes, thus making it difficult to achieve high-quality color panoramic image reconstruction.
[0004] With the widespread adoption of smart devices, digital images, as the most intuitive information carrier, are widely used in fields such as security monitoring, autonomous driving, virtual reality, and robot vision. However, single-view images can only cover limited scene information, making it difficult to meet the needs of overall environmental perception. Therefore, the demand for high-fidelity, wide-field-of-view panoramic images is becoming increasingly urgent. In static or low-speed scenes, high-quality panoramic images can usually be obtained by using rotating cameras or multi-lens arrays combined with image stitching technology. However, in high-speed rotating scenes, traditional frame-based color cameras are limited by shutter speed, frame rate, and dynamic range, making it difficult to achieve high frame rate and high dynamic range imaging. This leads to problems such as misalignment, motion blur, and inconsistent exposure in the reconstructed panoramic images, affecting visual quality. To address these challenges, researchers have begun to explore camera systems and reconstruction algorithms with special imaging mechanisms to optimize images. A typical approach is Technique 1: Panoramas from Photons: This technique uses a high-speed binary frame sequence captured by a single-photon camera to estimate fast scene motion, thereby creating panoramic images under fast motion and extremely low light conditions. Technology 2: Multi-Bracket High Dynamic Range Imaging with Event Cameras: Unlike Technology 1, which only uses single-photon camera images to create grayscale panoramas, this technology combines high-speed event cameras with ordinary color cameras, using motion information captured by the high-speed camera and color images for precise pixel alignment.
[0005] However, Technique 1 uses a uniform, fixed initial spatial transformation matrix and employs an iterative method to gradually estimate the motion trajectory, making it difficult to calculate the precise transformation of the image from different viewpoints. Furthermore, in real-world scenes, images reconstructed from binary frames captured by a single-photon camera lose color information. While Technique 2 uses a high-speed event camera and a color camera to reconstruct images, this method can only reconstruct a single image and cannot reconstruct panoramic images from multiple viewpoints. Additionally, the spatial alignment of color image frames relies on precise matching and has high computational complexity. Summary of the Invention
[0006] The purpose of this invention is to provide a high-speed, high dynamic range (HDR) color panoramic image reconstruction method and system to solve at least one of the technical problems existing in the background art. This invention deploys a high-frame-rate, high-dynamic-range pulse camera and a traditional color camera side-by-side. The pulse camera recovers the high-dynamic, low-blur panoramic structure and detail information in a rapidly rotating scene, while the color camera acquires color priors from the real scene. This achieves structurally continuous and color-consistent panoramic HDR color image reconstruction, thereby overcoming the limitations of traditional imaging systems in high-speed, high-dynamic scenes and realizing high-quality HDR color panoramic imaging.
[0007] To achieve the above objectives, the present invention adopts the following technical solution:
[0008] In a first aspect, the present invention provides a high-speed, high dynamic range color panoramic image reconstruction method, comprising:
[0009] Acquire a series of pulse reconstruction images of the scene to be reconstructed from multiple perspectives based on pulse signals;
[0010] Feature extraction and matching of pulse reconstructed images are performed using a time window-based approach. Specifically, the pulse discharge time difference is introduced to extract feature points from the pulse reconstructed images, enhancing the expression of texture details and improving the stability of feature matching. Vertical distance geometric constraints are introduced as a selection criterion for candidate matching pairs, and multiple images are aligned to a unified coordinate system.
[0011] Multiple images are aligned to a unified coordinate system, and an adaptive geometric model consisting of global transformation and mesh-based local transformation is used to align the images, resulting in distorted images and corresponding masks. All images are then fused to generate a high dynamic range panoramic image. In particular, to address the problem of texture detail loss that easily occurs during the direct fusion of multiple images, a weighted fusion strategy based on texture optimization is proposed to ensure smooth transition while enhancing local texture.
[0012] Based on a single local viewpoint color cue image, an image editing model is used with the SFT strategy, using text cue, visual cue, and input pulse panoramic image as conditions to guide the model to generate a high dynamic range panoramic color image; LoRA is combined to fine-tune the image editing model.
[0013] As a further definition of the first aspect of the present invention, high dynamic range image reconstruction based on pulse signals includes: a pulse camera continuously accumulates photons to generate electrons; when the accumulated charge is below a set threshold, the readout circuit is 0; once the accumulated electrons exceed the threshold, a pulse event is triggered, which is represented as follows:
[0014] ;
[0015] in Indicates the cumulative charge. Using a preset threshold, the time window is used. Reconstructing the image ;in, The sampling interval is... It is the window length. This represents the output state of the pulse camera at time t. Indicates time window The total number of sampling points contained therein Indicates the first time within the time window Each sampling time, This indicates the start time of the time window; since pulse data inevitably contains inherent noise, all valid pulse peaks participate in image reconstruction.
[0016] As a further limitation of the first aspect of the invention, in order to achieve a balance between high dynamic range and local detail, a pulsed discharge time difference is introduced to enhance the ability to retain texture information:
[0017] ;
[0018] in, and They represent time respectively The most recent pulse peak time before and after, This indicates the maximum search range.
[0019] As a further limitation of the first aspect of the invention, a vertical distance geometric constraint is introduced as a screening criterion for candidate matching pairs: Let the initial matching set be... ,in It is the first of the source images One feature point, Is the target image with Matching corresponding points, It represents the total number of matching pairs.
[0020] in , ;
[0021] . Represents the source image. Representing the target image, for each pair of matching points, calculate its displacement in the vertical direction. And calculate their mean. and variance The following criteria are used to filter candidate matches in order to eliminate abnormal erroneous matches with vertical offsets: ;in, This represents the filtered set of matches. This represents the initial set of matches.
[0022] As a further limitation of the first aspect of the present invention, the weighted fusion strategy based on texture optimization includes: calculating the geometric center of the effective region of two adjacent distorted images and constructing a connection direction vector; for pixels in the overlapping region, defining their normalized weights through linear projection and constructing an initial weight mask; performing joint bilateral filtering on the initial weight mask to obtain a final weight mask; and using the weight mask to fuse the images to obtain a high dynamic range panoramic image with smooth transitions in the overlapping region while maintaining texture details.
[0023] As a further limitation of the first aspect of the present invention, it is set that and Given two adjacent distorted images, their corresponding masks are respectively and Calculate the geometric center of the effective region of the two images. and And construct the connection direction vector For pixels in the overlapping region The normalized weights are defined by linear projection. ;in It is a point To the geometric center vector and direction vector dot product, and These are the minimum and maximum values of the projection within the overlapping region, respectively. It is a constant to prevent division by zero, and the initial weight mask is constructed based on it. ;in The overlapping area The non-overlapping regions are defined to ensure the integrity of the non-overlapping regions and the smoothness of the overlapping regions. The initial weighted mask is subjected to joint bilateral filtering to obtain the final weighted mask. The image is then fused using the weighted mask to obtain a high dynamic range panoramic image that has a smooth transition in the overlapping regions while maintaining texture details.
[0024] In a second aspect, the present invention provides a high-speed, high dynamic range color panoramic image reconstruction system, comprising:
[0025] The acquisition module is used to acquire a series of pulse reconstruction images of the scene to be reconstructed from multiple perspectives based on pulse signals;
[0026] The feature matching module is used to extract and match features from pulse reconstructed images based on a time window. Specifically, it introduces the pulse discharge time difference to extract feature points from the pulse reconstructed images, enhances the expression of texture details and improves the stability of feature matching, and introduces vertical distance geometric constraints as a selection criterion for candidate matching pairs to align multiple images to a unified coordinate system.
[0027] The fusion module is used to align multiple images to a unified coordinate system. It employs an adaptive geometric model consisting of global transformation and mesh-based local transformation to align the images, resulting in distorted images and corresponding masks. All images are then fused to generate a high dynamic range panoramic image. To address the issue of texture detail loss during direct fusion of multiple images, a texture-optimized weighted fusion strategy is proposed to ensure smooth transitions while enhancing local textures.
[0028] The generation module is used to generate color cue images based on a single local viewpoint. It uses an image editing model and an SFT strategy, guided by text cue, visual cue, and input pulse panoramic image as conditions, to generate high dynamic range panoramic color images. LoRA is used to fine-tune the image editing model.
[0029] Thirdly, the present invention provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the high-speed, high dynamic range color panoramic image reconstruction method as described in the first aspect.
[0030] Fourthly, the present invention provides a computer device including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the high-speed, high dynamic range color panoramic image reconstruction method as described in the first aspect.
[0031] Fifthly, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the high-speed, high dynamic range color panoramic image reconstruction method as described in the first aspect.
[0032] Terminology Explanation: HDR: High Dynamic Range; DSFT: Differential SpikeFiring Time; LORA: Low-Rank Adaptation; SFT: Supervised Fine-Tuning; Moving-DLT: Moving Direct Linear Transformation.
[0033] The beneficial effects of this invention are as follows: By deploying a high frame rate, high dynamic range pulse camera and a traditional color camera side by side, the pulse camera is used to recover the high dynamic and low blur panoramic structure and detail information in high-speed rotating scenes, while the color camera is introduced to obtain the color prior of the real scene, so as to realize the reconstruction of panoramic high dynamic range color images with continuous structure and consistent color. This overcomes the limitations of traditional imaging systems in high-speed, high-dynamic scenes and realizes high-quality HDR color panoramic imaging.
[0034] The advantages of additional aspects of the invention will be set forth more clearly in the following description or will be learned by practice of the invention. Attached Figure Description
[0035] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0036] Figure 1 This is an overall framework diagram of the high-speed, high dynamic range color panoramic image reconstruction method based on pulse signals according to an embodiment of the present invention.
[0037] Figure 2 This is a flowchart of the high-speed, high dynamic range color panoramic image reconstruction method based on pulse signals according to an embodiment of the present invention. Detailed Implementation
[0038] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0039] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0040] It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as here.
[0041] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.
[0042] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.
[0043] To facilitate understanding of the present invention, the present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.
[0044] Those skilled in the art should understand that the accompanying drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily essential for implementing the present invention.
[0045] This invention proposes a high-speed, high dynamic range (HDR) color panoramic image reconstruction method. Binary frames captured by pulse cameras typically contain significant noise, and images reconstructed based on accumulated pulses over a time window are prone to noise or motion blur. Therefore, this invention introduces Time Difference of Pulse Discharge (DSFT) to effectively enhance texture details and suppress blur. Previous methods for binary frame reconstruction often lead to misalignment or overlap between multi-view images. The proposed DSFT-enhanced vertical constraint matching algorithm is used to more accurately estimate the geometric transformation matrix between images, thereby achieving high-precision alignment. Pulse cameras can only acquire monochrome images, and directly using multiple frames from a color camera for feature-level spatial alignment is computationally complex. This invention uses a single frame captured by a color camera as a visual cue, combined with an image editing model for colorization, efficiently restoring color information. The method of this invention achieves high dynamic range color panoramic imaging effects, outperforming existing solutions in terms of visual quality and structural consistency.
[0046] Example 1
[0047] In this embodiment 1, a high-speed, high dynamic range color panoramic image reconstruction system is first provided, including: an acquisition module for acquiring a series of pulse reconstruction images of the scene to be reconstructed from multiple perspectives based on pulse signals; a feature matching module for performing feature extraction and matching on the pulse reconstruction images using a time window; wherein, pulse discharge time difference is introduced to extract feature points from the pulse reconstruction images, enhancing texture detail expression and improving the stability of feature matching; vertical distance geometric constraints are introduced as a screening criterion for candidate matching pairs; and multiple images are aligned to a unified coordinate system; a fusion module is used to fuse multiple images after they are aligned to a unified coordinate system, employing a global transformation and a feature matching module based on a time window. An adaptive geometric model composed of local transformations of the mesh is used to align the images, resulting in distorted images and corresponding masks. All images are then fused to generate a high dynamic range panoramic image. To address the issue of texture detail loss during direct fusion of multiple images, a weighted fusion strategy based on texture optimization is proposed to ensure smooth transitions while enhancing local textures. The generation module uses a color cue image based on a single local viewpoint. An image editing model employs an SFT strategy, using text cue, visual cue, and an input pulse panoramic image as conditions to guide the model in generating a high dynamic range panoramic color image. LoRA is incorporated to fine-tune the image editing model.
[0048] In this embodiment, a high-speed, high dynamic range (HDR) color panoramic image reconstruction method is implemented using the aforementioned system. This method includes: acquiring a series of pulse reconstruction images of the scene to be reconstructed from multiple perspectives based on pulse signals; performing feature extraction and matching on the pulse reconstruction images using a time window; wherein, a pulse discharge time difference is introduced to extract feature points from the pulse reconstruction images, enhancing texture detail expression and improving the stability of feature matching; a vertical distance geometric constraint is introduced as a screening criterion for candidate matching pairs; and multiple images are aligned to a unified coordinate system; aligning multiple images to a unified coordinate system, an adaptive geometric model composed of global transformation and mesh-based local transformation is used to align the images, obtaining distorted images and corresponding masks; and fusing all images to generate a HDR panoramic image; wherein, to address the problem of texture detail loss during direct fusion of multiple images, a weighted fusion strategy based on texture optimization is proposed to ensure smooth transitions while enhancing local textures; based on a single local perspective color cue image, an image editing model is used, employing an SFT strategy, with text cue, visual cue, and the input pulse panoramic image as conditions to guide the model to generate a HDR panoramic color image; wherein, LoRA is combined to fine-tune the image editing model.
[0049] The high dynamic range image reconstruction based on pulse signals includes: a pulse camera continuously accumulates photons to generate electrons; when the accumulated charge is below a set threshold, the readout circuit is 0; once the accumulated electrons exceed the threshold, a pulse event is triggered, which is represented as follows:
[0050] ;
[0051] in Indicates the cumulative charge. Using a preset threshold, the time window is used. Reconstructing the image ;in, The sampling interval is... It is the window length. This represents the output state of the pulse camera at time t. Indicates time window The total number of sampling points contained therein Indicates the first time within the time window Each sampling time, This indicates the start time of the time window. Since pulse data inevitably contains inherent noise, all valid pulse peaks are included in the image reconstruction.
[0052] To achieve a balance between high dynamic range and local detail, a pulsed discharge time difference is introduced to enhance the preservation of texture information:
[0053] ;
[0054] in, and They represent time respectively The most recent pulse peak time before and after, This indicates the maximum search range.
[0055] Introducing a vertical distance geometric constraint as a selection criterion for candidate matching pairs: Let the initial matching set be... ,in It is the first of the source images One feature point, Is the target image with Matching corresponding points, This represents the total number of matching pairs. , , . Represents the source image. This represents the target image.
[0056] For each pair of matching points, calculate its displacement in the vertical direction. And calculate their mean. and variance The following criteria are used to filter candidate matches in order to eliminate abnormal erroneous matches with vertical offsets: ;in, This represents the filtered set of matches. This represents the initial set of matches.
[0057] The texture-optimized weighted fusion strategy includes: calculating the geometric center of the effective region of two adjacent distorted images and constructing a connection direction vector; defining the normalized weight of pixels in the overlapping region through linear projection to construct an initial weight mask; performing joint bilateral filtering on the initial weight mask to obtain the final weight mask; and using the weight mask to fuse the images to obtain a high dynamic range panoramic image with smooth transitions in the overlapping region while maintaining texture details.
[0058] Specifically, let's take two adjacent images as an example to illustrate the weighted fusion strategy based on texture optimization: Let... and Given two adjacent distorted images, their corresponding masks are respectively and Calculate the geometric center of the effective region of the two images. and And construct the connection direction vector For pixels in the overlapping region The normalized weights are defined by linear projection. ;in It is a point To the geometric center vector and direction vector dot product, and These are the minimum and maximum values of the projection within the overlapping region, respectively. It is a constant to prevent division by zero, and the initial weight mask is constructed based on it. ;in The overlapping area The non-overlapping regions are defined to ensure the integrity of the non-overlapping regions and the smoothness of the overlapping regions. The initial weighted mask is subjected to joint bilateral filtering to obtain the final weighted mask. The image is then fused using the weighted mask to obtain a high dynamic range panoramic image that has a smooth transition in the overlapping regions while maintaining texture details.
[0059] Example 2
[0060] This embodiment proposes a high-speed, high dynamic range (HDR) color panoramic image reconstruction method based on a pulse camera. This method captures diverse scene information in rapidly rotating scenes to achieve HDR panoramic image reconstruction, utilizing the ultra-high sampling rate of 200,000 Hz of the pulse camera for data acquisition. Specifically, the pulse camera is fixedly mounted on a tripod, and real-world scenes are captured by rotating the camera at 180° per second. Eight real-world scenes were selected, resulting in 20 datasets. During acquisition, the pulse camera outputs raw single-bit pulse signals. After decoding, the raw data forms a continuous binary pulse stream with a resolution of 250×400 pixels. Simultaneously, a high-resolution color camera is placed side-by-side with the pulse camera to synchronously capture color frames as color cues for subsequent colorization. The obtained binary pulse stream is then preprocessed to obtain a reconstructed image. This reconstructed image is then input into a panoramic reconstruction algorithm to obtain a HDR panoramic image. Finally, the panoramic image and the color cues captured by the color camera are input into a finely tuned image editing model for inference colorization, thereby obtaining a HDR panoramic color image.
[0061] like Figure 2 As shown, the high-speed, high dynamic range color panoramic image reconstruction method based on a pulse camera described in this embodiment specifically includes three steps: pulse signal preprocessing, high dynamic range panoramic image reconstruction, and coloring based on a fine-tuning image editing model.
[0062] (1) Pulse signal preprocessing: This step utilizes the asynchronous response characteristics of the pulse camera to photon events to accumulate and process the continuously output pulse signals to reconstruct a high dynamic range image.
[0063] High dynamic range image reconstruction based on pulse signals: A pulse camera continuously accumulates photons to generate electrons. When the accumulated charge is below a set threshold, the readout circuit is 0; once the accumulated electrons exceed the threshold, a pulse event is triggered, which is represented as follows:
[0064]
[0065] in Indicates the cumulative charge. Using a preset threshold, the time window is used. Images can be reconstructed. .in The sampling interval is... This refers to the window length. Since pulse data inevitably contains inherent noise, a larger window length is chosen. This allows for the acquisition of more stable high dynamic range images. Based on this, all effective pulse peaks will participate in image reconstruction, providing reliable input for subsequent tasks.
[0066] The trade-off between high dynamic range and sharpness: However, excessively large time windows can lead to motion blur and reduce image sharpness. To achieve a balance between high dynamic range and local detail, Time Difference of Pulse Discharge (DSFT) is introduced to enhance the preservation of texture information, defined as follows:
[0067]
[0068] in, and They represent time respectively The most recent pulse peak time before and after, This indicates the maximum search range.
[0069] (2) High dynamic range panoramic image reconstruction: The panoramic image is reconstructed using a series of multi-view pulse reconstruction images obtained in step (1).
[0070] Vertical Constraint Matching Enhanced by DSFT: Feature extraction and matching are performed using images reconstructed based on time windows. However, due to the potential introduction of motion blur during time integration, the robustness of matching based on these images is limited. Therefore, image reconstruction results based on the DSFT method are further used for feature point extraction, enhancing texture detail representation and improving the stability of feature matching. Considering that the relative motion between adjacent frames is typically small under high frame rates, especially vertical displacement, certain constraints should be imposed. Therefore, a vertical distance geometric constraint is introduced as a selection criterion for candidate matching pairs. Specifically, let the initial matching set be... ,in , For each pair of matching points, calculate its displacement in the vertical direction. And calculate their mean. and variance Then, the following criteria are used to filter candidate matching relationships in order to eliminate abnormal erroneous matches with vertical offset.
[0071]
[0072] High dynamic range panoramic image synthesis: To align multiple images to a unified coordinate system, an adaptive geometric model consisting of global transformation and mesh-based local transformation (Moving-DLT) is used to align the images, resulting in a distorted image. and the corresponding mask Then, all images are fused to generate a high dynamic range panoramic image.
[0073] To address the issue of texture detail loss during direct fusion of multiple images, a weighted fusion strategy based on texture optimization is designed to ensure smooth transitions while enhancing local textures. In this embodiment, the fusion process is illustrated using two adjacent images as an example, as detailed below.
[0074] set up and Given two adjacent distorted images, their corresponding masks are respectively and First, calculate the geometric center of the effective region of both images. and And construct the connection direction vector For pixels in the overlapping region The normalized weights are defined by linear projection. in It is a point To the geometric center vector and direction vector dot product, and These are the minimum and maximum values of the projection within the overlapping region, respectively. It is a constant to prevent division by zero, and the initial weight mask is constructed based on it.
[0075]
[0076] in The overlapping area This ensures the integrity of non-overlapping regions and the smoothness of overlapping regions.
[0077] To further enhance texture consistency, a joint bilateral filter is applied to the initial weighted mask to obtain the final weighted mask:
[0078]
[0079] in and These represent the spatial kernel and the range kernel, respectively. For the initial weight mask The guiding image is obtained after Gaussian filtering. Finally, the images are fused using the aforementioned weighted mask to obtain a high dynamic range panoramic image with smooth transitions in overlapping areas while preserving texture details.
[0080] .
[0081] (3) Coloring based on fine-tuned image editing model: The panoramic image reconstructed by the pulse camera is monochrome. To add color information, a color camera in a hybrid camera system is used to simultaneously capture a single local viewpoint color cue image in the scene acquired by the pulse camera. In order to maintain the edge structure during the color filling process while avoiding fine spatial alignment of the panoramic image, the generation capability of the large language model is utilized, and the Qwen-image-edit-2509 image editing model is used in combination with LoRA for fine-tuning. The SFT strategy is used, with text cue, visual cue, and input pulse panoramic image as conditions to guide the model to generate a high dynamic range panoramic color image. The text cue is "Colors in Figure 1 are adjusted to match the style of the color image in Figure 2."
[0082] In this embodiment, synthetic data is used for model fine-tuning. The synthetic data is constructed as follows: the synthetic dataset contains two data sources: synthetic panoramic images generated from manually collected video sequences (each segment lasting 10-15 seconds); and panoramic images from the internet used to simulate diverse scenes. To more closely resemble the imaging characteristics of a real pulse camera, all synthetic panoramic images are converted into pulse form as model input to enhance the data's ability to fit the actual imaging process. This embodiment constructs 9 video sequences and 1,050 panoramic images from the internet, of which 1,000 images are used as the training set for fine-tuning. For each set of panoramic images, a local region of 256×256 pixels is randomly cropped, and the corresponding color image is used as a color cue input to the model. The complete original color panoramic image serves as a panoramic reference baseline, used to supervise the learning of the model's output results during the fine-tuning process.
[0083] In summary, the method proposed in this embodiment can reconstruct high dynamic range (HDR) color panoramic images in high-speed rotating scenes, effectively capturing panoramic structure and detail information, and reducing the impact of motion blur and exposure on panoramic image quality. A hybrid camera system with a pulse camera and a color camera arranged side-by-side is introduced. This fully leverages the imaging advantages of the pulse camera in terms of high frame rate and high dynamic range, while the color camera acquires color information from the real scene, effectively compensating for the color information loss caused by monochrome imaging with the pulse camera. This embodiment achieves a high dynamic range color panoramic imaging effect with continuous structure and consistent color by introducing a pulse signal-based panoramic reconstruction module and a fine-tuned color shading module, thereby improving the overall quality of the panoramic image.
[0084] Binary frames captured by pulse cameras typically contain significant noise, and images reconstructed based on accumulated pulses over a time window are prone to noise or motion blur. To address this, we introduce Time Difference of Pulse Discharge (DSFT) to effectively enhance texture details and suppress blur. Previous methods for binary frame reconstruction often lead to misalignment or overlap between multi-view images. Our proposed DSFT-enhanced vertical constraint matching algorithm more accurately estimates the geometric transformation matrix between images, achieving high-precision alignment. Since pulse cameras can only acquire monochrome images, directly using multiple frames from a color camera for feature-level spatial alignment is computationally complex. Therefore, we use a single frame captured by a color camera as a visual cue, combining it with an image editing model for colorization to efficiently recover color information. This method achieves high dynamic range color panoramic imaging, outperforming existing solutions in terms of visual quality and structural consistency.
[0085] In practical applications, the image editing model structure can be replaced with other similar structures, such as other neural network models capable of colorization or other large models. The pulse camera can be replaced with other special types of cameras, such as event cameras, commercial high-speed cameras like Phantom, or other specialized equipment capable of acquiring high-speed motion information.
[0086] Example 3
[0087] This embodiment 3 provides a non-transitory computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, they implement the high-speed, high dynamic range color panoramic image reconstruction method described above. The method includes:
[0088] Acquire a series of pulse reconstruction images of the scene to be reconstructed from multiple perspectives based on pulse signals;
[0089] Feature extraction and matching of pulse reconstructed images are performed using a time window-based approach. Specifically, the pulse discharge time difference is introduced to extract feature points from the pulse reconstructed images, enhancing the expression of texture details and improving the stability of feature matching. Vertical distance geometric constraints are introduced as a selection criterion for candidate matching pairs, and multiple images are aligned to a unified coordinate system.
[0090] Multiple images are aligned to a unified coordinate system, and an adaptive geometric model consisting of global transformation and mesh-based local transformation is used to align the images, resulting in distorted images and corresponding masks. All images are then fused to generate a high dynamic range panoramic image. In particular, to address the problem of texture detail loss that easily occurs during the direct fusion of multiple images, a weighted fusion strategy based on texture optimization is proposed to ensure smooth transition while enhancing local texture.
[0091] Based on a single local viewpoint color cue image, an image editing model is used with the SFT strategy, using text cue, visual cue, and input pulse panoramic image as conditions to guide the model to generate a high dynamic range panoramic color image; LoRA is combined to fine-tune the image editing model.
[0092] Example 4
[0093] This embodiment 4 provides a computer device, including a memory and a processor, wherein the processor and the memory communicate with each other, and the memory stores program instructions that can be executed by the processor. The processor calls the program instructions to execute the high-speed, high dynamic range color panoramic image reconstruction method described above, the method including:
[0094] Acquire a series of pulse reconstruction images of the scene to be reconstructed from multiple perspectives based on pulse signals;
[0095] Feature extraction and matching of pulse reconstructed images are performed using a time window-based approach. Specifically, the pulse discharge time difference is introduced to extract feature points from the pulse reconstructed images, enhancing the expression of texture details and improving the stability of feature matching. Vertical distance geometric constraints are introduced as a selection criterion for candidate matching pairs, and multiple images are aligned to a unified coordinate system.
[0096] Multiple images are aligned to a unified coordinate system, and an adaptive geometric model consisting of global transformation and mesh-based local transformation is used to align the images, resulting in distorted images and corresponding masks. All images are then fused to generate a high dynamic range panoramic image. In particular, to address the problem of texture detail loss that easily occurs during the direct fusion of multiple images, a weighted fusion strategy based on texture optimization is proposed to ensure smooth transition while enhancing local texture.
[0097] Based on a single local viewpoint color cue image, an image editing model is used with the SFT strategy, using text cue, visual cue, and input pulse panoramic image as conditions to guide the model to generate a high dynamic range panoramic color image; LoRA is combined to fine-tune the image editing model.
[0098] Example 5
[0099] This embodiment 5 provides an electronic device, including: a processor, a memory, and a computer program; wherein, the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the high-speed, high dynamic range color panoramic image reconstruction method described above, the method including:
[0100] Acquire a series of pulse reconstruction images of the scene to be reconstructed from multiple perspectives based on pulse signals;
[0101] Feature extraction and matching of pulse reconstructed images are performed using a time window-based approach. Specifically, the pulse discharge time difference is introduced to extract feature points from the pulse reconstructed images, enhancing the expression of texture details and improving the stability of feature matching. Vertical distance geometric constraints are introduced as a selection criterion for candidate matching pairs, and multiple images are aligned to a unified coordinate system.
[0102] Multiple images are aligned to a unified coordinate system, and an adaptive geometric model consisting of global transformation and mesh-based local transformation is used to align the images, resulting in distorted images and corresponding masks. All images are then fused to generate a high dynamic range panoramic image. In particular, to address the problem of texture detail loss that easily occurs during the direct fusion of multiple images, a weighted fusion strategy based on texture optimization is proposed to ensure smooth transition while enhancing local texture.
[0103] Based on a single local viewpoint color cue image, an image editing model is used with the SFT strategy, using text cue, visual cue, and input pulse panoramic image as conditions to guide the model to generate a high dynamic range panoramic color image; LoRA is combined to fine-tune the image editing model.
[0104] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0105] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0106] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0107] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0108] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solutions disclosed in the present invention, various modifications or variations that can be made by those skilled in the art without creative effort should be included within the scope of protection of the present invention.
Claims
1. A high-speed high-dynamic range color panoramic image reconstruction method, characterized in that, include: Acquire a series of pulse reconstruction images of the scene to be reconstructed from multiple perspectives based on pulse signals; Feature extraction and matching of pulse reconstructed images are performed using a time window-based approach. Specifically, the pulse discharge time difference is introduced to extract feature points from the pulse reconstructed images, enhancing the expression of texture details and improving the stability of feature matching. Vertical distance geometric constraints are introduced as a selection criterion for candidate matching pairs, and multiple images are aligned to a unified coordinate system. Multiple images are aligned to a unified coordinate system, and an adaptive geometric model consisting of global transformation and mesh-based local transformation is used to align the images, resulting in distorted images and corresponding masks. All images are then fused to generate a high dynamic range panoramic image. In particular, to address the problem of texture detail loss that easily occurs during the direct fusion of multiple images, a weighted fusion strategy based on texture optimization is proposed to ensure smooth transition while enhancing local texture. Based on a single local viewpoint color cue image, an image editing model is used with the SFT strategy, using text cue, visual cue, and input pulse panoramic image as conditions to guide the model to generate a high dynamic range panoramic color image; LoRA is combined to fine-tune the image editing model.
2. The high-speed, high dynamic range color panoramic image reconstruction method according to claim 1, characterized in that, High dynamic range image reconstruction based on pulse signals includes: a pulse camera continuously accumulates photons to generate electrons; when the accumulated charge is below a set threshold, the readout circuit is 0; once the accumulated electrons exceed the threshold, a pulse event is triggered, which is represented as follows: ； in Indicates the cumulative charge. Using a preset threshold, the time window is used. Reconstructing the image ;in, The sampling interval is... It is the window length. This represents the output state of the pulse camera at time t. Indicates time window The total number of sampling points contained therein Indicates the first time within the time window Each sampling time, This indicates the start time of the time window; since pulse data inevitably contains inherent noise, all valid pulse peaks participate in image reconstruction.
3. The high-speed, high dynamic range color panoramic image reconstruction method according to claim 2, characterized in that, To achieve a balance between high dynamic range and local detail, a pulsed discharge time difference is introduced to enhance the preservation of texture information: ； in, and They represent time respectively The most recent pulse peak time before and after, This indicates the maximum search range.
4. The high-speed, high dynamic range color panoramic image reconstruction method according to claim 3, characterized in that, Introducing a vertical distance geometric constraint as a selection criterion for candidate matching pairs: Let the initial matching set be... ,in It is the first of the source images One feature point, Is the target image with Matching corresponding points, It is the total number of matching pairs; where , , ; Represents the source image. Represent the target image; for each pair of matching points, calculate its displacement in the vertical direction. And calculate their mean. and variance The following criteria are used to filter candidate matches in order to eliminate abnormal erroneous matches with vertical offsets: ;in, This represents the filtered set of matches. This represents the initial set of matches.
5. The high-speed, high dynamic range color panoramic image reconstruction method according to claim 1, characterized in that, The texture-optimized weighted fusion strategy includes: calculating the geometric center of the effective region of two adjacent distorted images and constructing a connection direction vector; defining the normalized weight of pixels in the overlapping region through linear projection to construct an initial weight mask; performing joint bilateral filtering on the initial weight mask to obtain the final weight mask; and using the weight mask to fuse the images to obtain a high dynamic range panoramic image with smooth transitions in the overlapping region while maintaining texture details.
6. The high-speed, high dynamic range color panoramic image reconstruction method according to claim 5, characterized in that, set up and Given two adjacent distorted images, their corresponding masks are respectively and Calculate the geometric center of the effective region of the two images. and And construct the connection direction vector For pixels in the overlapping region The normalized weights are defined by linear projection. ;in It is a point To the geometric center vector and direction vector dot product, and These are the minimum and maximum values of the projection within the overlapping region, respectively. It is a constant to prevent division by zero, and the initial weight mask is constructed based on it. ;in The overlapping area The non-overlapping regions are defined to ensure the integrity of the non-overlapping regions and the smoothness of the overlapping regions. The initial weighted mask is subjected to joint bilateral filtering to obtain the final weighted mask. The image is then fused using the weighted mask to obtain a high dynamic range panoramic image that has a smooth transition in the overlapping regions while maintaining texture details.
7. A high-speed, high dynamic range color panoramic image reconstruction system, characterized in that, include: The acquisition module is used to acquire a series of pulse reconstruction images of the scene to be reconstructed from multiple perspectives based on pulse signals; The feature matching module is used to extract and match features from pulse reconstructed images based on a time window. Specifically, it introduces the pulse discharge time difference to extract feature points from the pulse reconstructed images, enhances the expression of texture details and improves the stability of feature matching, and introduces vertical distance geometric constraints as a selection criterion for candidate matching pairs to align multiple images to a unified coordinate system. The fusion module is used to align multiple images to a unified coordinate system. It employs an adaptive geometric model consisting of global transformation and mesh-based local transformation to align the images, resulting in distorted images and corresponding masks. All images are then fused to generate a high dynamic range panoramic image. To address the issue of texture detail loss during direct fusion of multiple images, a texture-optimized weighted fusion strategy is proposed to ensure smooth transitions while enhancing local textures. The generation module is used to generate color cue images based on a single local viewpoint. It uses an image editing model and an SFT strategy, guided by text cue, visual cue, and input pulse panoramic image as conditions, to generate high dynamic range panoramic color images. LoRA is used to fine-tune the image editing model.
8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the high-speed, high dynamic range color panoramic image reconstruction method as described in any one of claims 1-6.
9. A computer device, characterized in that, The system includes a memory and a processor, which communicate with each other. The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the high-speed, high dynamic range color panoramic image reconstruction method as described in any one of claims 1-6.
10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the high-speed, high dynamic range color panoramic image reconstruction method as described in any one of claims 1-6.