An event camera based high dynamic range remote sensing image reconstruction method and system

Abstract

The application discloses a high-dynamic-range remote sensing image reconstruction method and system based on an event camera, adopts a hybrid camera system of a traditional RGB industrial camera and an event camera, aligns multimodal expressions of event features and image features in a feature space, processes the multimodal expressions through an event-based dynamic range enhancement network, obtains a preliminary rough and clean high-dynamic-range image, acquires gradient information from the generated rough high-dynamic-range remote sensing image, acquires short-time event frames from a voxel sequence, strengthens the structure of the preliminary rough high-dynamic-range image through a gradient-enhanced high-dynamic-range reconstruction network, and generates a fine high-dynamic-range remote sensing image. The application effectively solves the problems that it is difficult to recover texture information in a saturated area and noise in a reconstruction process in a current single-exposure high-dynamic-range remote sensing image reconstruction method, and improves the generation quality of the high-dynamic-range remote sensing image.

Description

Technical Field

[0001] This invention relates to the field of high dynamic range imaging and image reconstruction, specifically to a high dynamic range remote sensing image reconstruction method and system based on an event camera. Background Technology

[0002] Remote sensing imagery is a crucial tool for Earth observation, widely used in geological exploration, precision agriculture, urban planning, and disaster monitoring. However, due to complex lighting conditions, real-world remote sensing images often suffer from low dynamic range, resulting in incomplete scene information. High dynamic range (HDR) imaging is an important task in remote sensing image processing. It enhances low dynamic range (LDR) remote sensing images and facilitates downstream tasks such as target detection and image segmentation.

[0003] Frame-based HDR image reconstruction mainly consists of two methods: multi-exposure HDR image reconstruction and single-exposure HDR image reconstruction. The former generates an HDR image by fusing multiple images with different exposures, but due to its reliance on specific software / hardware techniques, it often causes time delays, and the fused image may exhibit artifacts. Traditional single-exposure HDR image reconstruction methods expand the dynamic range by estimating light source density or by performing cross-bilateral filtering to enhance the input low dynamic range image. However, these methods struggle to handle the diverse semantic information in complex scenes.

[0004] With the development of deep learning, some methods have emerged that leverage imaging knowledge, tone mapping, and other prior information, along with efficient network models, to directly generate HDR images from single LDR images. For example, Eilertsen et al. proposed an encoder-decoder network, HDRCNN, in collaboration with skip pass, to predict the raw values ​​of saturated pixels in an image, thereby recovering details in overexposed areas. Marnerides et al. proposed the Expandent method, designing a multi-scale autoencoder architecture designed to learn different levels of detail from LDR images. However, these methods often ignore quantization artifacts and noise. Santos et al. effectively mitigated blur and halo artifacts in HDR images by introducing masking features and perceptual loss, but in some cases, this may lead to inaccurate colors in the reconstructed saturated regions. KUNet proposes a unified network based on the imaging process by integrating LDR-to-HDR imaging knowledge into the UNet architecture.

[0005] Despite significant progress in some methods, reconstructing HDR images from a single image remains a complex and ill-posed problem. This is because information is lost in underexposed and overexposed areas, and texture details in these areas can only be predicted using prior knowledge of the scene.

[0006] Event cameras, based on biomimetic technology, are visual sensors that, unlike traditional RGB cameras which measure the intensity of each pixel, asynchronously detect changes in scene brightness, giving them a unique advantage in capturing high dynamic range (HDR) scenes. Given that event cameras can provide additional scene detail in LDR scenes, existing methods have proposed various strategies to leverage events to guide the LDR-to-HDR mapping. Han et al. proposed a multimodal camera system and learning framework for HDR, taking intensity maps generated by LDR and E2VID as input. However, they were unable to perform end-to-end optimization of the model and heavily relied on E2VID performance. Richard introduced a novel event-to-image feature distillation module that directly transforms event features into the image feature space without relying on intermediate intensity images. However, when applied to remote sensing datasets, these methods often fail to achieve satisfactory generalization results. Furthermore, remote sensing images are often captured at high altitudes, facing the problem of light attenuation.

[0007] Therefore, how to overcome the insufficient information content of single-exposure LDR, and at the same time make full use of the information from event cameras and RGB modal data, is an urgent problem to be solved in the field of high dynamic range remote sensing image reconstruction. Summary of the Invention

[0008] To address the issues of insufficient detail in single-exposure HDR image reconstruction and how to fully utilize event camera and RGB modal data, this invention provides a high dynamic range remote sensing image reconstruction method based on event camera.

[0009] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a high dynamic range remote sensing image reconstruction method based on an event camera, and includes the following steps: Step S1: Transfer the event camera data event stream Compressed voxels Using the event encoder Processing the input voxel sequence Obtain event characteristics Using an image encoder Processing input low dynamic range remote sensing images Obtain image features ,in and All are multi-layer convolutional encoders. Then, a hybrid cross-attention module is used. To fuse and align event features and image features into a multimodal representation.

[0010] Step S2: Enhance multimodal representations using the Event-Based Dynamic Range Reinforcement Network (E-DRE). Processing is performed to obtain a preliminary, coarse, and clean HDR image. .

[0011] Step S3: Generate a coarse HDR image from step S2 Obtaining gradient information From voxel sequences Get short event frames The gradient-enhanced high dynamic range reconstruction network G-HDRR is used to structurally enhance the initially coarse HDR image, generating a refined HDR remote sensing image. .

[0012] Furthermore, the frame compression method in step S1 is as follows: based on the timestamp, the event stream is embedded into a voxel grid of size 5×h×w.

[0013] Furthermore, to better integrate the information from the two modalities, this invention proposes a hybrid cross-attention module to fuse multimodal representations of aligned event features and image features, fully utilizing the information from both modalities and effectively mitigating the domain gap problem. In step S1... The module implementation method is as follows: given LDR image features and event characteristics Two parallel feature attention operations (inner attention and cross attention) are performed to learn key information within each modality branch. For the inner attention operation, a single convolutional block is used to generate image content features. and the characteristics of the event content Then, a dual attention module consisting of channel attention and spatial attention is used for feature processing to generate... and For cross-attention operations, cascaded convolutional blocks are used to... and Generate image attention feature map Event attention feature map Features are refined by performing dot product on the feature map. and It also provides cross-modal weighted adjustments to generate... and Finally, the features generated by inward attention and cross-attention operations are summed, and a splicing operation is performed to obtain the fused features. .

[0014] Furthermore, in step S2, the E-DRE includes a noise reduction module. The implementation method is as follows: design denoising networks of different scales using convolutional blocks and upsampling / downsampling operations, and then fuse their feature outputs to reduce the impact of random noise on the recovery results under different receptive fields.

[0015] Furthermore, step S2 of the E-DRE also includes a dense feature fusion module. The implementation method is as follows: a dense fusion module is used as the main component to ensure sufficient connection between non-adjacent level features, and missing spatial information is corrected during downsampling and upsampling, and finally coarse HDR image reconstruction is performed.

[0016] Furthermore, to highlight detailed texture information, this invention proposes a gradient enhancement method. The module extracts structural features from image gradients and event frames, reduces the impact of noise, effectively enhances texture in low-contrast areas, and improves the realism of the reconstructed HDR remote sensing image. Step S3's G-HDRR includes a gradient enhancement module. The implementation method is as follows: The module generates a coarse HDR image in step S2. gradient information and short event frames The input is a concatenated set of learnable convolutional blocks, which encode structure-related positional information. Scaling and shifting operations modulate the input features to enhance details in low-contrast regions.

[0017] Furthermore, the G-HDRR in step S3 also includes a multi-scale fusion module. The implementation method is as follows: multi-scale feature extraction is achieved by expanding the convolution of dense residual blocks (DRDB) with different receptive fields. Simultaneously, efficient feature fusion is achieved by merging local residual skip connections within DRDB blocks and global residual skip connections between the three DRDB blocks.

[0018] Furthermore, the neural network is trained using both real and synthetic remote sensing data. The specific data generation and training process is as follows: Dataset generation includes the following two methods: A hybrid camera system is mounted on an aircraft to capture remote sensing data of real-world scenes. LDR images and corresponding event stream raw files are saved. By aligning and merging alternating exposure LDR images, a true HDR reference image is obtained. The data is then organized to construct a real-world dataset, which is divided into training and test sets.

[0019] By filtering and processing the existing remote sensing dataset DOTA-v1.0, realistic HDR reference images were obtained. The ESIM simulator was used to simulate and generate corresponding events. A series of extremely exposed LDR images were generated by performing brightness transformations on the original HDR images and introducing random noise. Finally, a synthetic dataset was obtained and divided into training and test sets.

[0020] (2) Training of neural networks a) Input the event stream and LDR images into the network architecture for end-to-end training.

[0021] b) The entire training process includes two loss functions: coarse optimization loss. Optimize loss The following are respectively represented:

[0022] Among them, For true HDR images, This represents the first [unit / item] in the pre-trained VGG-19 network. layer, Indicates the first Weights of layer feature maps , , Indicates the first The shape of the layer feature map Indicates the discriminator, These are hyperparameters used to control the trade-offs among these three factors.

[0023]

[0024] in, This represents the gradient operator for extracting high-frequency spatial information, and AP(·) represents the average pooling function along the channel.

[0025] The final loss function can be expressed as:

[0026] in and Hyperparameters are used to control the trade-offs of each item.

[0027] Based on the same inventive concept, this solution also designs a high dynamic range remote sensing image reconstruction system based on an event camera: This includes a data multimodal representation generation module, which extracts event features from the event stream of event camera data and from low dynamic range remote sensing images. Obtaining image features Multimodal representations are obtained by fusing aligned event features and image features.

[0028] A coarse HDR image generation module processes multimodal representations through an event-based dynamic range enhancement network to obtain preliminary HDR images. The dynamic range enhancement network includes a denoising module and a feature fusion module. High-resolution HDR remote sensing images By using a gradient-enhanced high dynamic range reconstruction network, the initially coarse HDR image is structurally enhanced to generate a refined HDR remote sensing image. The gradient-enhanced high dynamic range reconstruction network includes a gradient enhancement module and a multi-scale fusion module.

[0029] Based on the same inventive concept, this solution also designs an electronic device, including: one or more processors; and a storage device for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement a high dynamic range remote sensing image reconstruction method based on an event camera.

[0030] Based on the same inventive concept, this solution also designs a computer-readable medium storing a computer program that, when executed by a processor, implements a high dynamic range remote sensing image reconstruction method based on an event camera.

[0031] Compared with the prior art, the present invention has the following advantages: 1) The high dynamic range remote sensing image reconstruction method based on event camera provided by the present invention combines traditional RGB camera with event camera, making full use of the high dynamic range and high temporal resolution characteristics of event camera, solving the ill-conditioning problem of single exposure HDR image reconstruction, realizing the restoration of texture details in highly saturated areas, and effectively improving the quality of reconstructed HDR remote sensing images.

[0032] 2) This invention proposes a coarse-to-fine network architecture and training strategy. First, the multimodal features are processed by the event-based dynamic range enhancement network E-DRE to generate a preliminary coarse HDR image. Then, the structure of the preliminary coarse HDR image is strengthened by the gradient-enhanced high dynamic range reconstruction network G-HDRR, which alleviates the light attenuation problem in the imaging process and effectively restores the texture details of the reconstructed HDR remote sensing image.

[0033] 3) The neural network architecture ERM-HDRI proposed in this invention can fully utilize the advantages of traditional RGB cameras and event cameras to generate realistic and clear high dynamic range remote sensing images. The effect is far superior to current HDR remote sensing image reconstruction methods. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the embodiments will be briefly described below.

[0035] Figure 1 This is a general block diagram of the technical solution of the present invention.

[0036] Figure 2 This is a structural diagram of the multimodal feature extraction and fusion and event-based dynamic range enhancement network E-DRE in the neural network architecture ERM-HDRI proposed in this invention.

[0037] Figure 3 This is a diagram of the gradient-enhanced high dynamic range reconstruction network G-HDRR proposed in this invention.

[0038] Figure 4 The hybrid cross-attention proposed in this invention Module structure diagram.

[0039] Figure 5 The gradient enhancement proposed in this invention Module structure diagram. Detailed Implementation Plan The specific implementation of the high dynamic range remote sensing image reconstruction method based on an event camera provided by the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described examples are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0040] Example 1 The high dynamic range (HVR) remote sensing image reconstruction method based on event cameras proposed in this invention mainly consists of three steps: multimodal representation fusion and alignment, event-based dynamic range enhancement, and gradient-enhanced HVR reconstruction. Its neural network architecture includes two encoders. and Processing event and image features separately; a hybrid cross-attention module To fuse and align multimodal representations of event features and image features; a denoising module and a dense feature fusion module It is used for dynamic range enhancement to obtain a preliminary, coarse, clean HDR image. A gradient enhancement module and a multi-scale fusion module This is used to achieve gradient-enhanced high dynamic range reconstruction, resulting in a final, refined HDR remote sensing image. .

[0041] The method of the present invention specifically includes the following steps: Step S1: Transfer the event camera data event stream Compressed voxels Using the event encoder Processing the input voxel sequence Obtain event characteristics Using an image encoder Processing input low dynamic range remote sensing images Obtain image features ,in and All are multi-layer convolutional encoders. Then, a hybrid cross-attention module ( This is used to fuse and align event features and image features into a multimodal representation.

[0042] Step S2: Multimodal representation through Event-Based Dynamic Range Enhancement Network (E-DRE) Processing is performed to obtain a preliminary, coarse, and clean HDR image. .

[0043] Step S3: Generate a coarse HDR image from step S2 Obtaining gradient information From voxel sequences Get short event frames The structure of the initially coarse HDR image is enhanced by using a gradient-enhanced high dynamic range reconstruction network (G-HDRR) to generate a refined HDR remote sensing image. .

[0044] The entire process can be represented as:

[0045] in, This indicates frame compression. This indicates the gradient-enhanced high dynamic range reconstruction network G-HDRR. This represents an event-based dynamic range augmentation network (E-DRE).

[0046] Specifically, the frame compression method in step S1 is as follows.

[0047] Furthermore, the frame compression method in step S1 is as follows: based on the timestamp, the event stream is embedded into a voxel grid of size 5×h×w. In the real dataset, h = 720, w = 1280; in the synthetic dataset, h = 480, w = 640.

[0048] In step S1 The module method is as follows.

[0049] Considering the different imaging mechanisms of traditional RGB cameras and event cameras, and the significant domain gap between optical images and event streams, we choose to align the features of the two modalities in the feature space and perform preliminary fusion. The specific implementation is as follows: given LDR image features... and event characteristics Two parallel feature attention operations (inner attention and cross attention) are performed to learn key information within each modality branch. For the inner attention operation, a single convolutional block is used to generate image content features. and the characteristics of the event content Then, a dual attention module consisting of channel attention and spatial attention is used for feature processing to generate... and For cross-attention operations, cascaded convolutional blocks are used to... and Generate image attention feature map Event attention feature map Features are refined by performing dot product on the feature map. and It also provides cross-modal weighted adjustments to generate... and Finally, the features generated by inward attention and cross-attention operations are summed, and a splicing operation is performed to obtain the fused features. .

[0050] In step S2, the multimodal representation is processed using an event-based dynamic range enhancement network to obtain a preliminary HDR image. Noise denoising can be achieved using existing techniques, such as adaptive filters and deep learning-based DnCNN; feature fusion can be achieved using addition, multiplication, attention mechanisms, etc. The preferred method in this invention is as follows: Noise reduction module The method is as follows.

[0051] Considering the significant noise present in the original LDR image and event stream, which can affect the subsequent HDR reconstruction process, the following approach is adopted. Denoising preprocessing is performed on multimodal representations. Three denoising networks of different scales are designed using convolutional blocks and upsampling / downsampling operations, and then their feature outputs are fused to reduce the impact of random noise on the recovery results under different receptive fields.

[0052] Dense feature fusion module in step S2 The method is as follows: Considering the complexity of the fusion features between the two modalities, we adopt A deeper level of fusion processing is performed. Specifically, a dense fusion module is used as the main component to ensure sufficient connectivity between non-adjacent level features, and missing spatial information is corrected during downsampling and upsampling. Finally, the feature representation in the latent space is decoded to perform coarse HDR image reconstruction.

[0053] Gradient enhancement module in step S3 The method is as follows: Considering the light attenuation problem during the acquisition of remote sensing images and the weak information in low-contrast areas of the event stream, the following methods are used: The module performs gradient enhancement, specifically generating a coarse HDR image in step S2. gradient information and short event frames The network takes cascaded convolutional blocks as input and uses these cascaded learnable convolutional blocks to encode structure-related positional information. It modulates the input features through scaling and shifting operations to enhance details in low-contrast regions. In practice, the network employs three... Module cascading.

[0054] Multi-scale fusion module in step S3 The method is as follows: In order to integrate the detailed information at different scales with the coarse HDR image generated in step S2 To achieve full integration, multi-scale feature extraction is first accomplished by expanding the dense residual blocks (DRDB) with different receptive fields through convolution. Simultaneously, efficient feature fusion is achieved by merging local residual skip connections within DRDB blocks and global residual skip connections between the three DRDB blocks. In implementation, the network employs three... Module cascading.

[0055] Finally, the fused features are decoded to obtain a reconstructed, detailed HDR remote sensing image. .

[0056] The real dataset and the synthetic data are respectively divided into training set and test set.

[0057] The neural network was built and trained using Python, based on the PyTorch framework, and trained using an NVIDIA GeForce RTX 4090 GPU.

[0058] a) Input the event stream and LDR images into the network architecture for end-to-end training.

[0059] b) The entire training process includes two loss functions: coarse optimization loss. Optimize loss The specific steps for constructing the loss function in this invention are as follows: coarse optimization of the loss. Represented as:

[0060] Specifically, the brightness Manhattan distance between the output image and the real HDR image is calculated as a pixel-level loss function. The calculation formula is:

[0061] For the pre-trained VGG-19 network, the perceptual loss is calculated using the reconstructed image and the real HDR image. As a pixel-level loss function, it is used to measure perceptual differences in human vision. The calculation formula is:

[0062] Considering that LDR images suffer from severe color information degradation and contain little color information within events, we employ GAN loss, which is highly effective in image colorization tasks, to recover color information and generate more natural images. This is illustrated below:

[0063] Among them, For true HDR images, This represents the first [unit / item] in the pre-trained VGG-19 network. layer, Indicates the first Weights of layer feature maps, , , Indicates the first The shape of the layer feature map Indicates the discriminator, This is a hyperparameter used to control the trade-offs among these three items, and it is set to 10 in this embodiment.

[0064] Fine-tuning loss It can be represented as:

[0065] Specifically, it adopted In , and In addition to spatial loss, a high-frequency spatial information difference is introduced to calculate the difference between the reconstructed fine HDR image and the original HDR image, which is a key aspect of the structure enhancement process. The calculation formula is as follows:

[0066] in, This represents the gradient operator for extracting high-frequency spatial information, and AP(·) represents the average pooling function along the channel.

[0067] The final loss function can be expressed as:

[0068] in and These are hyperparameters used to control the trade-offs of each term. Specifically, in this implementation, they are set to 1 and 2 respectively.

[0069] This invention provides a high dynamic range remote sensing image reconstruction method based on an event camera. This method inputs GRB images captured by a traditional industrial camera and event streams obtained through the event camera into a trained network. First, multimodal features are obtained through an encoder, and then a cross-attention module is used to reconstruct these features. We fuse and align multimodal representations of event features and image features. Then, we process these multimodal representations using an Event-Based Dynamic Range Enhancement (E-DRE) network to obtain a preliminary coarse HDR image. Finally, we use a Gradient Enhancement High Dynamic Range Reconstruction (G-HDRR) network to structurally enhance the coarse HDR image, generating a refined HDR remote sensing image. Finally, we construct real-world and synthetic datasets to evaluate our network, helping it to better generalize to real-world data.

[0070] Example 2 Based on the same inventive concept, this solution also designs a high dynamic range remote sensing image reconstruction system based on an event camera, the overall architecture of which is shown in the attached figure. Figure 1 ; This includes a data multimodal representation generation module, which extracts event features from the event stream of event camera data and from low dynamic range remote sensing images. Obtaining image features Multimodal representations are obtained by fusing aligned event features and image features.

[0071] A coarse HDR image generation module processes multimodal representations through an event-based dynamic range enhancement network to obtain preliminary HDR images. The dynamic range enhancement network includes a denoising module and a dense feature fusion module. High-resolution HDR remote sensing images By using a gradient-enhanced high dynamic range reconstruction network, the initially coarse HDR image is structurally enhanced to generate a refined HDR remote sensing image. The gradient-enhanced high dynamic range reconstruction network includes a gradient enhancement module and a multi-scale fusion module.

[0072] The neural network in this invention is trained using real remote sensing datasets and synthetic remote sensing datasets. The dataset acquisition process in this embodiment is as follows: 1) Dataset generation a) The hybrid camera system is equipped with two different cameras: an RGB camera, namely the FLIR BFS-U3-32S4, for capturing real LDR images at 10 FPS, and an event camera, namely the Prophesee Gen4.1 EVK2-HD, for collecting concurrent event streams.

[0073] (b) The two cameras are synchronized using trigger signals generated by the RGB camera. Specifically, the RGB camera generates alternating positive and negative trigger signals at the beginning and end of the exposure. The event camera captures the signals from the RGB camera and records accurate timestamps and polarity information. The recorded signal information allows for precise extraction of events within the exposure time of the LDR image.

[0074] c) Spatial calibration is performed between the RGB camera and the event camera to ensure consistent field of view across all cameras. Specifically, this embodiment uses the E2VID method to reconstruct the intensity map of the event. Subsequently, calibration is performed between the frames captured by the RGB camera and the reconstructed intensity map.

[0075] d) A hybrid camera system was mounted on an aircraft to capture images at an altitude of approximately 5000 meters. This resulted in 20,000 alternating exposure LDR images and their corresponding event streams. The images had a resolution of 1280 × 720 and included diverse scenes such as lakes, mountains, buildings, and deserts. By aligning and merging the alternating exposure LDR images, a true HDR reference image was obtained. This data constituted the real-world dataset.

[0076] e) 2340 images were selected from the DOTA-v1.0 remote sensing dataset as raw data. A sliding window method was used to process each image, generating 600 sub-images with random step sizes as true HDR reference images. The ESIM simulator was used to simulate the corresponding events. By performing brightness transformations on the original HDR images and introducing random noise, a series of extremely exposed LDR images were generated. Finally, a synthetic dataset containing 1,404,000 LDR images with a resolution of 640 × 480 was obtained.

[0077] Since the device described in Embodiment 2 of the present invention is for implementing the high dynamic range remote sensing image reconstruction method based on event camera in Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the electronic device based on the method described in Embodiment 1 of the present invention, and therefore will not be described again here.

[0078] Example 3 Based on the same inventive concept, the present invention also provides an electronic device, including one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in Embodiment 1.

[0079] Since the device described in Embodiment 3 of this invention is the electronic device used in implementing the high dynamic range remote sensing image reconstruction method and system based on an event camera according to Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this electronic device based on the method described in Embodiment 1 of this invention, and therefore will not be described again here. All electronic devices used in any method of this invention fall within the scope of protection of this invention.

[0080] Example 4 Based on the same inventive concept, the present invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in Embodiment 1.

[0081] Since the device described in Embodiment 4 of this invention is a computer-readable medium used in implementing the high dynamic range remote sensing image reconstruction method and system based on an event camera according to Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this electronic device based on the method described in Embodiment 1 of this invention, and therefore will not be described again here. All electronic devices used in any method of this invention are within the scope of protection of this invention.

[0082] The specific embodiments described herein are merely illustrative examples of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A high dynamic range remote sensing image reconstruction method based on an event camera, characterized in that, Includes the following steps: Step S1: Extract event features from the event stream of the event camera data, and from the low dynamic range remote sensing images. Obtaining image features Multimodal representation is obtained by fusing alignment event features and image features, specifically: Given LDR image features and event characteristics It performs two parallel processes, inner attention and cross attention, to learn key information within each modality branch. For the inner attention operation, a convolutional block is used to generate image content features. and the characteristics of the event content Then, a dual attention module consisting of channel attention and spatial attention is used for feature processing to generate... and For cross-attention operations, cascaded convolutional blocks are used to... and Generate image attention feature map Event attention feature map Features are refined by performing dot product on the feature map. and It also provides cross-modal weighted adjustments to generate... and Finally, the features generated by inward attention and cross-attention operations are summed, and a splicing operation is performed to obtain the fused features. ; Step S2: Process the multimodal representation using an event-based dynamic range enhancement network to obtain a preliminary HDR image. The dynamic range enhancement network includes a denoising module and a dense feature fusion module. The denoising module takes the multimodal representation as input and performs multi-scale denoising on the multimodal representation. It reduces the impact of random noise on the recovery result under different receptive fields. The dense feature fusion module takes the output features of the denoising module as input to ensure sufficient connection between adjacent level features and corrects missing spatial information during downsampling and upsampling. Finally, it outputs a coarse HDR image. The specific processing procedure of the event-based dynamic range augmentation network is as follows: Denoising networks of different scales are designed using convolutional blocks and upsampling / downsampling operations. Their feature outputs are then fused to reduce the impact of random noise on the restoration results under different receptive fields. Dense fusion modules are used as the main components to ensure sufficient connectivity between non-adjacent level features. Missing spatial information is corrected during downsampling and upsampling. Finally, preliminary HDR image reconstruction is performed. Step S3: The initially coarse HDR image is structurally enhanced using a gradient-enhanced high dynamic range reconstruction network to generate a refined HDR remote sensing image. The gradient-enhanced high dynamic range reconstruction network includes a gradient enhancement module and a multi-scale fusion module. The gradient enhancement module takes the generated coarse HDR image gradient information and the corresponding concatenation of short-time event frames as input, mines structural features from the image gradient and event frames, reduces the influence of noise, effectively enhances the texture of low-contrast areas, improves the realism of the reconstructed HDR remote sensing image, and outputs the enhanced fusion features. The multi-scale fusion module takes the generated coarse HDR image and the output of the gradient enhancement module as input, aiming to achieve multi-scale feature extraction and efficient feature fusion, and finally output a fine HDR image. The gradient enhancement module generates a preliminary HDR image. The gradient information and the concatenation of short-term event frames are used as inputs. Concatenated learnable convolutional blocks are used to encode structure-related positional information. The input features are modulated through scaling and shifting operations to enhance the details of low-contrast regions. The multi-scale fusion module achieves multi-scale feature extraction by expanding convolutions on dense residual blocks (DRDB) with different receptive fields, and achieves efficient feature fusion by merging local residual skip connections within DRDB blocks and global residual skip connections between three DRDB blocks.

2. The high dynamic range remote sensing image reconstruction method based on an event camera according to claim 1, characterized in that: In step S1, the event camera data can be real data or synthetic data. The method for obtaining real data is as follows: The hybrid camera system is mounted on an aircraft to capture remote sensing data of real scenes, save LDR images and corresponding event stream raw files, and obtain real HDR reference images by aligning and merging alternately exposed LDR images. The data is then organized to construct a real dataset, which is divided into training and testing sets. The method for acquiring synthetic data is as follows: By filtering and processing the existing remote sensing dataset DOTA-v1.0, real HDR reference images are obtained. The ESIM simulator is used to simulate and generate corresponding events. By performing brightness transformation on the original HDR images and introducing random noise, a series of extreme exposure LDR images are generated, and finally a synthetic dataset is obtained, which is divided into training set and test set.

3. The high dynamic range remote sensing image reconstruction method based on an event camera according to claim 1, characterized in that: In step S1, the event camera data event stream is... Compressed voxels Using the event encoder Processing the input voxel sequence Obtain event characteristics The frame compression method is as follows: based on the timestamp, the event stream is embedded into a voxel grid of size 5×h×w.

4. The high dynamic range remote sensing image reconstruction method based on an event camera according to claim 1, characterized in that: The network training in steps 2 and 3 is as follows: a) Input the event stream and LDR images into the network architecture for end-to-end training; b) The entire training process includes two loss functions: coarse optimization loss. Optimize loss ; respectively represent the following: Among them, For true HDR images, Indicates the first in the pre-trained network layer, Indicates the first Weights of layer feature maps, , , Indicates the first The shape of the layer feature map Indicates the discriminator, These are hyperparameters used to control the trade-offs among these three factors. ; in, AP(·) represents the gradient operator for extracting high-frequency spatial information, and AP(·) represents the average pooling function along the channel. The final loss function can be expressed as: in and Hyperparameters are used to control the trade-offs of each item.

5. A high dynamic range remote sensing image reconstruction system based on an event camera, characterized in that: This includes a data multimodal representation generation module, which extracts event features from the event stream of event camera data and from low dynamic range remote sensing images. Obtaining image features Multimodal representation is obtained by fusing alignment event features and image features; Specifically: Given LDR image features and event characteristics It performs two parallel processes, inner attention and cross attention, to learn key information within each modality branch. For the inner attention operation, a convolutional block is used to generate image content features. and the characteristics of the event content Then, a dual attention module consisting of channel attention and spatial attention is used for feature processing to generate... and For cross-attention operations, cascaded convolutional blocks are used to... and Generate image attention feature map Event attention feature map Features are refined by performing dot product on the feature map. and It also provides cross-modal weighted adjustments to generate... and Finally, the features generated by inward attention and cross-attention operations are summed, and a splicing operation is performed to obtain the fused features. ; A coarse HDR image generation module processes multimodal representations through an event-based dynamic range enhancement network to obtain preliminary HDR images. The dynamic range enhancement network includes a denoising module and a feature fusion module. The specific processing procedure of the event-based dynamic range augmentation network is as follows: Denoising networks of different scales are designed using convolutional blocks and upsampling / downsampling operations. Their feature outputs are then fused to reduce the impact of random noise on the restoration results under different receptive fields. Dense fusion modules are used as the main components to ensure sufficient connectivity between non-adjacent level features. Missing spatial information is corrected during downsampling and upsampling. Finally, preliminary HDR image reconstruction is performed. High-resolution HDR remote sensing images By using a gradient-enhanced high dynamic range reconstruction network, the initially coarse HDR image is structurally enhanced to generate a refined HDR remote sensing image. The gradient-enhanced high dynamic range reconstruction network includes a gradient enhancement module and a multi-scale fusion module; The gradient enhancement module generates a preliminary HDR image. The gradient information and the concatenation of short-term event frames are used as inputs. Concatenated learnable convolutional blocks are used to encode structure-related positional information. The input features are modulated through scaling and shifting operations to enhance the details of low-contrast regions. The multi-scale fusion module achieves multi-scale feature extraction by expanding convolutions on dense residual blocks (DRDB) with different receptive fields, and achieves efficient feature fusion by merging local residual skip connections within DRDB blocks and global residual skip connections between three DRDB blocks.

6. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-4.

7. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it implements the method of any one of claims 1-4.

Images