An event-driven, full-illumination dynamic scene-oriented texture-enhanced video super-resolution method and system

Abstract

The application provides an event-driven texture-enhanced video super-resolution method and system for a full-illumination dynamic scene, which comprises the following steps: acquiring a low-resolution RGB video sequence and an event stream in space-time synchronization; performing pixel-by-pixel illumination perception on the RGB frame; performing double-layer adaptive coding and space-time joint denoising on the event stream; using a time sequence alignment branch to aggregate the current RGB frame and adjacent RGB frames before and after the current RGB frame to obtain context features; using an event texture enhancement branch to encode a normalized event voxel grid to obtain event texture features; generating an iterative mask according to a pixel-by-pixel illumination intensity map; inputting the iterative mask into a block-by-block iterative texture updater to iteratively update to obtain high-frequency texture features; calculating an adaptive fusion weight according to the pixel-by-pixel illumination intensity map, weighting and pixel-by-pixel fusing the context features and the high-frequency texture features, and performing up-sampling reconstruction on the fused features to obtain a high-resolution video sequence. The application can balance the texture recovery accuracy, bright and dark detail preservation capability and real-time processing performance under extreme illumination conditions.

Description

Technical Field

[0001] This invention relates to the technical field of computer vision, video processing and neuromorphic sensors, and in particular to an event-driven, texture-enhanced video super-resolution method and system for dynamic scenes under full illumination. It is an event-driven texture-enhanced video super-resolution (VSR) method for low-light / strong backlight (0.1 lux-10000 lux) scenes. Background Technology

[0002] With the comprehensive coverage of security monitoring, the widespread adoption of autonomous driving in vehicles, and the intelligent upgrading of industrial inspection, high-definition video has become a core input for intelligent sensing systems. Video super-resolution technology, as a key means of improving video clarity, can restore low-resolution video to high-resolution video, directly determining the accuracy of subsequent tasks such as target detection, recognition, and tracking. High-quality super-resolution can improve target detection accuracy by 20%-30%. Conversely, if the super-resolution effect fails, it will lead to blurry monitoring images, misjudgments by vehicle vision systems, and missed industrial defects. This can result in data invalidation and rework (with a single industrial inspection costing over 5000 yuan), or even traffic accidents and production safety accidents.

[0003] However, low-light / strong backlight environments are typical complex scenarios for video super-resolution technology: On the one hand, in low light, the signal-to-noise ratio of RGB images is extremely low (SNR<10dB), high-frequency texture information is completely submerged by noise, and the grayscale dynamic range is compressed to 30%-50% of the original range; in strong backlight, bright areas are overexposed and dark areas are underexposed, making it impossible to retain details in both bright and dark areas simultaneously. On the other hand, although neuromorphic event sensors (event cameras) have a high dynamic range of over 120dB and microsecond-level temporal resolution, events are sparse and noise increases in low light (noise rate exceeds 20%), while events are too dense and contain a large number of invalid triggers in strong light, which existing methods cannot adaptively handle.

[0004] Current low-light video super-resolution technologies are mainly divided into three categories, all of which have significant limitations:

[0005] Pure RGB low-light super-resolution schemes: These schemes rely on algorithms such as Retinex and LLNet, and depend on image enhancement and super-level coupling processing. However, their effectiveness in low light is limited—the PSNR drops by 3-5 dB in extremely low light (0.1 lux) compared to normal lighting, and they cannot recover high-frequency textures completely obscured by noise; under strong backlighting, the dynamic range is less than 60 dB, and the problems of overexposure in bright areas and underexposure in dark areas cannot be solved.

[0006] General event-assisted super-resolution solutions: These solutions are based on algorithms such as EGVSR, EBVSR, and EvTexture, utilizing events to assist motion estimation and alignment. However, these solutions are not optimized for extreme lighting conditions—sparse events in low light lead to insufficient texture recovery, while event noise in strong light causes severe artifacts. Furthermore, they employ a fixed fusion strategy, failing to dynamically adjust the contribution weights of events and RGB values ​​based on light intensity, resulting in performance degradation of over 40% in extreme scenarios.

[0007] High computing power relies on super-resolution solutions: Algorithms are designed for specific high-computing power platforms without considering the deployment requirements of edge devices. When high-computing power solutions are directly deployed to edge devices, the processing time for a single frame exceeds 50ms (corresponding to a frame rate of < 20 frames / s), which cannot meet real-time requirements.

[0008] Existing publicly available technologies only focus on arbitrary-scale reconstruction in general scenes, without introducing a lighting perception mechanism, making them unable to adapt to extreme low-light / high-light conditions; or they do not address texture enhancement and dynamic range expansion under extreme lighting conditions; or they do not achieve video super-resolution and real-time edge inference; or they do not take into account high-frequency texture restoration and spatiotemporal super-resolution. Existing technologies have not yet achieved deep integration of "light perception - event adaptive encoding - block texture iteration - lighting adaptive fusion": either they only solve a single problem (such as only optimizing low-light enhancement without utilizing the high dynamic range (HDR) characteristics of events), or they lack edge deployment capabilities, failing to simultaneously meet the integrated requirements of "extreme lighting adaptation (0.1 lux-10000 lux), real-time performance (≥30fps@1080p), and high-precision texture restoration (PSNR improvement ≥1.2dB)". Maintenance personnel are forced to face a dilemma: abandoning intelligent systems under extreme lighting conditions, switching to manual inspection (reducing efficiency by 70%), or risking misjudgment by using them. Therefore, developing an event-driven texture enhancement video super-resolution method and device specifically designed for low-light / strong backlight environments has become a key technical challenge that urgently needs to be solved in the field of intelligent sensing. Summary of the Invention

[0009] To address the technical problems of existing technologies such as texture loss, insufficient dynamic range, severe noise, and poor real-time edge deployment in extreme lighting scenarios (0.1 lux to 10000 lux), this invention proposes an event-driven, texture-enhanced video super-resolution method and system for dynamic scenes under all lighting conditions. It is the first integrated solution to simultaneously cover "extreme lighting adaptation + texture enhancement + spatiotemporal super-resolution + real-time edge inference". Specifically designed for extreme lighting operation scenarios such as security monitoring, vehicle-mounted assisted driving, industrial defect detection, and drone aerial photography, it achieves high precision (PSNR improvement of 1.2-2.1 dB), high dynamic range (≥115 dB), and real-time performance (30 fps@1080p) for video super-resolution under extreme lighting conditions through four major technical modules: adaptive encoding of lighting-sensing events, block-by-block progressive texture iteration, lighting-adaptive dual-branch fusion, and multi-constraint joint loss optimization. This provides core technical support for intelligent sensing systems under complex lighting conditions.

[0010] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0011] The technical solution of this application includes: acquiring spatiotemporally synchronized low-resolution RGB video sequences and event streams and performing geometric correction, timestamp alignment, and preprocessing; generating pixel-by-pixel illumination intensity maps based on RGB frames, and performing global hardware threshold configuration, region-by-region software filtering, spatiotemporal noise filtering, adaptive voxel encoding, and normalization on the event stream based on the scene's average illumination intensity and local illumination intensity; extracting temporal context features using a bidirectional temporal alignment branch, and extracting high-frequency texture features using an event texture enhancement branch and a PTO module; dynamically calculating the weights of the two branches based on pixel-by-pixel illumination intensity and completing feature fusion; and finally performing 4x upsampling reconstruction using pixel rearrangement to output a high-resolution, high dynamic range video sequence.

[0012] An event-driven, texture-enhanced video super-resolution method for dynamic scenes with full illumination is proposed, with the following steps:

[0013] S1: Acquire spatiotemporally synchronized low-resolution RGB video sequences and event streams, and preprocess them separately;

[0014] S2: Perform pixel-by-pixel illumination sensing on the preprocessed low-resolution RGB frame to generate a pixel-by-pixel illumination intensity map; perform dual-layer adaptive coding and spatiotemporal joint denoising on the event stream based on the pixel-by-pixel illumination intensity map to generate a normalized event voxel grid.

[0015] S3: The temporal alignment branch of the dual-branch structure is used to perform bidirectional optical flow estimation, feature deformation, and temporal aggregation on the current RGB frame and its preceding and following RGB frames to obtain temporally consistent context features; the event texture enhancement branch of the dual-branch structure is used to perform multi-scale encoding on the normalized event voxel grid to obtain event texture features in different time chambers; a block-by-block iterative mask is generated based on the pixel-by-pixel illumination intensity map; the context features, event texture features, and iterative mask are input together into the block-by-block iterative texture updater of the PTO module for iterative updating to obtain high-frequency texture features;

[0016] S4: Dynamically calculate adaptive fusion weights based on the pixel-by-pixel illumination intensity map, and perform weighted pixel-by-pixel fusion of context features and high-frequency texture features to obtain fused features;

[0017] S5: Upsampling reconstruction is performed using fusion features to output high-resolution, high-dynamic-range video sequences.

[0018] Preferably, the spatiotemporal synchronization includes time synchronization and spatial synchronization, wherein the time synchronization is achieved by shared hardware triggering and a unified clock reference; and the spatial synchronization is achieved by joint geometric calibration.

[0019] The preprocessing includes: performing dark current correction, pixel value normalization, and lightweight noise reduction on RGB frames; sorting the event stream by timestamp, deleting out-of-bounds coordinates, abnormal timestamps, and duplicate events, and unifying event polarity encoding.

[0020] The method for implementing the joint geometric calibration is as follows: the pixel coordinates of the RGB frame and the event coordinates of the event stream are distorted, and the event coordinates are reprojected onto the RGB imaging plane according to the extrinsic transformation matrix from the event camera coordinate system to the RGB camera coordinate system. The event positions with the same spatial resolution as the RGB frame are obtained through bilinear interpolation. Events that exceed the common field of view after reprojection are deleted, and the two data streams are cropped to a common effective area.

[0021] Preferably, the method for generating the pixel-by-pixel illumination intensity map is as follows: convert the preprocessed RGB frame into the YCbCr color space, extract the Y channel as the initial illumination intensity map; use a 5×5 Gaussian filter to smooth the initial illumination intensity map, and normalize the pixel values ​​to [0, 1] to obtain the pixel-by-pixel illumination intensity map.

[0022] The implementation method of dual-layer adaptive coding and spatiotemporal joint denoising includes: 1) global hardware threshold pre-configuration: setting the global trigger threshold of the event camera according to the average illumination intensity of the scene before event stream acquisition; 2) region-by-region software secondary filtering: performing local threshold processing on the acquired event stream according to the pixel-by-pixel illumination intensity map; 3) spatiotemporal noise filtering: using a spatiotemporal window composed of a 3×3 spatial neighborhood and a preset time window to filter isolated noise events, and using an event frequency statistical method to identify and filter hot pixels; 4) event voxel grid coding and normalization: adaptively determining the number of time bins according to the event density, encoding the filtered events into a voxel grid, and performing 98th percentile truncation normalization to obtain a normalized event voxel grid.

[0023] Preferably, the temporal alignment branch uses optical flow estimation to perform sub-pixel-level alignment of the current RGB frame with its preceding and following adjacent RGB frames to obtain alignment features; it uses forward-backward optical flow consistency to generate a validity mask, removes occluded regions and unreliable optical flow regions, and concatenates the alignment features of the preceding and following adjacent RGB frames with the features of the current RGB frame. These features are then aggregated by a context extractor to obtain temporally consistent context features F. kc ;

[0024] The event texture enhancement branch uses the event texture extractor of the PTO module to extract multi-scale event texture features from the normalized event voxel grid. The block-by-block iterative texture updater combines a globally unified maximum number of iterations with block-by-block iterative mask dynamic updates to achieve illumination-adaptive high-frequency texture restoration and obtain high-frequency texture features.

[0025] Preferably, the optical flow estimation employs a pyramid optical flow network or a FlowNet network;

[0026] The context extractor includes a 3×3 convolution and eight cascaded residual blocks arranged sequentially; the 3×3 convolution maps the features of the current RGB frame to the alignment features of the forward and backward adjacent RGB frames to a uniform number of channels.

[0027] Each residual block contains a first convolutional layer, a batch normalization layer, a ReLU activation function, a second convolutional layer, and a residual summation layer connected in sequence. The output of the residual summation second convolutional layer is added element-wise to the input of the residual block.

[0028] The event texture extractor adopts a 5-layer U-Net structure, including a 5-level encoder and a symmetrical 5-level decoder. Each level encoder extracts event textures through 3×3 convolution and non-linear activation, and reduces the spatial size and increases the number of channels through stride convolution or downsampling. Each level decoder upsamples step by step to restore the spatial size, and concatenates the current decoded features with the encoder features of the same scale along the channel dimension, and then fuses them through 3×3 convolution; thus generating event texture features with consistent spatial resolution for different time chambers.

[0029] The block-by-block iterative texture updater includes a block-by-block iterative control module, a ConvGRU module, a residual block group, a residual addition module, and an iteration count determination module connected in sequence; the block-by-block iterative control module generates the iteration mask M for the b-th iteration based on the pixel-by-pixel illumination intensity map. b The ConvGRU module will change the previous hidden state. Contextual features F kc Event texture features of the b-th time chamber Channel splicing is performed, and candidate hidden states are obtained by controlling the amount of historical information retained and the amount of current event information written through update and reset gates. According to the iterative mask M b Obtain the hidden state in the b-th iteration. ,in, This represents the element-wise multiplication operator; a residual block group containing 5 concatenated residual blocks will be hidden. Mapped to the residual texture feature Δ of this round k b The residual addition module updates the texture feature F. k b =F k b-1 +Δ k b Among them, F k b-1 and F k b These represent the texture features before and after the update, respectively. The iteration count determination module checks whether the effective iteration count corresponding to each image patch of the pixel-by-pixel illumination intensity map or the globally unified maximum iteration count has been reached. If not, it returns to the ConvGRU module to continue iterating; if reached, it outputs the final high-frequency texture feature F. k T =F k B B represents the maximum timeframe or iteration round actually executed.

[0030] Preferably, the iteration mask M for generating the b-th iteration is... b The method is as follows: the maximum number of iterations is set globally to 7; the pixel-by-pixel illumination intensity map is divided into 8×8 pixel non-overlapping image blocks, and the average illumination intensity of each image block is calculated to generate iterative masks for different iteration rounds; when the resolution of the illumination map and the feature map are inconsistent, the nearest neighbor method is used to scale the iterative mask to the corresponding feature scale; in each iteration, only the hidden state and event texture features of the ConvGRU module are updated for the image blocks with an iteration mask of 1, and the image blocks with a mask of 0 directly retain the results of the previous iteration;

[0031] The average illumination intensity of the image patch is obtained by the arithmetic mean of the 64 normalized illumination pixel values ​​within the image patch. The generation rules for the iterative mask are as follows: when the average illumination intensity is < 0.2, the iterative mask is 1 in the first to seventh iterations; when the average illumination intensity is 0.2 ≤ average illumination intensity ≤ 0.8, the iterative mask is 1 in the first to fifth iterations and 0 in the sixth to seventh iterations; when the average illumination intensity is > 0.8, the iterative mask is 1 in the first to third iterations and 0 in the fourth to seventh iterations.

[0032] Preferably, the formula for calculating the adaptive fusion weights is: Among them, w A For time-aligned branch weights, w T Enhance the branch weights for event textures, where α is an adaptive coefficient with a value range of [0.5, 1.0], and satisfies w A +w T =1; L is the illumination intensity of the pixel-by-pixel illumination intensity map;

[0033] Align the time-series branch weights w A Timing alignment feature F kc Element-wise multiplication yields the weighted time series feature F. k A =w A ⊙F kc Event texture enhancement branch weight w T With high-frequency texture features F k T Element-wise multiplication yields the weighted texture feature F. k E =w T ⊙F k T The pixel-by-pixel illumination intensity map is scaled to the same spatial size as the feature map and then copied along the channel dimension or mapped via a 1×1 convolution to form the illumination-guided feature F. k L ;Weigh the time series features F k A Weighted texture features F k E and illumination guidance features F k L The sequence is stitched along the channel dimension and passed through three 3×3 convolutional layers in sequence. The first two convolutional layers are followed by a ReLU activation function, and the last convolutional layer outputs the fused feature F. k f .

[0034] Preferably, the upsampling is performed using any one of pixel rearrangement, transposed convolution, nearest neighbor upsampling + convolution, or bicubic upsampling + convolution.

[0035] An end-to-end training optimization is performed on optical flow estimation, context extractor, event texture extractor, block-by-block iterative texture updater, adaptive fusion, and reconstruction using a total loss function that includes low-illumination texture weighting loss and dynamic range constraint loss. The total loss function is as follows: Among them, L ch For Charbonnier loss, L low For low-light texture weighting loss, L dr The loss is a dynamic range constraint loss, where β1 and β2 are both hyperparameters.

[0036] Preferably, the average illuminance of the scene is obtained by an illuminance sensor synchronized with the RGB camera: the most recent illuminance value is read before the start of each RGB frame exposure, and the average of 3 to 5 consecutive illuminance values ​​is taken; when no illuminance sensor is configured, an illuminance mapping table can be established according to the pre-calibrated exposure time, analog gain and Y channel average of the RGB camera, and the spatial average of the Y channel of the current frame can be converted into an estimated illuminance.

[0037] The global trigger threshold represents the logarithmic change in brightness required to trigger an event. When the logarithmic change in brightness is <1 lux, the global trigger threshold is set to 0.05; when the logarithmic change in brightness is 1 to 1000 lux, the global trigger threshold is set to 0.1; and when the logarithmic change in brightness is >1000 lux, the global trigger threshold is set to 0.2.

[0038] The region-by-region software secondary filtering is performed after the event coordinates are reprojected onto the RGB imaging plane. The implementation method is as follows: based on the position (x, y) of the i-th event... i y i Read the illumination intensity L at the corresponding position in the pixel-by-pixel illumination intensity map, and calculate the normalized event confidence magnitude a by combining it with the neighborhood support of the i-th event in the local time surface. i When the light intensity L < 0.2, retain the normalized event confidence amplitude a. i Weak events ≥ 0.03; when light intensity L > 0.8, the normalized event confidence amplitude a is removed. i Low confidence events < 0.15; normalized event confidence magnitude a i When the value is between 0.03 and 0.15, it is linearly interpolated according to the light intensity L;

[0039] The method for filtering isolated noise is as follows: a 3×3 spatial neighborhood and a time window with a fixed transmission delay of ±Δt are established with the event to be judged as the center, where Δt is the time offset of the fixed transmission delay. The number of neighborhood events with the same polarity or continuous time with the event to be judged within the time window is counted. When the number of neighborhood events is less than a preset threshold, the event to be judged is identified as background activity noise and deleted.

[0040] The method for filtering hot pixels is as follows: calculate the number of event triggers for each pixel within the statistical window. When the number of event triggers for a certain pixel is consistently higher than the sum of the average event frequency of the entire image and three times the standard deviation, and the correlation of the neighboring events of the pixel is lower than a preset threshold, add the pixel to the hot pixel mask and block its output in subsequent event packets.

[0041] The event voxel grid encoding method is as follows: normalize the event time interval corresponding to the current RGB frame to [0, B-1], where B is the number of time bins; each event is linearly interpolated and accumulated to the adjacent time bins according to its polarity and normalized time position to form an event voxel grid with size B×H×W, where H and W are the spatial height and spatial width of the event voxel grid, respectively.

[0042] Calculate the 98th percentile q of the non-zero absolute value of the event voxel grid. 98 The voxel values ​​of the event voxel grid are truncated to [-q]. 98 q 98 Divide by percentile q 98 Normalized to [-1, 1];

[0043] The number of time bins, B, is adaptively selected based on the event density per unit time and per unit pixel: event density ≤ 0.05 events / (pixel) . s) retrieves 3 timeframes; 05 events / (pixel . s) < event density ≤ 0.2 events / (pixel) . s) ) Take 5 time bins, event density > 0.2 events / (pixel) . s) or when the movement is faster, take 7 time slots;

[0044] Both the first and second convolutional layers are 3×3 convolutions;

[0045] Initial texture features F k 0 The event texture features output by the event texture extractor are obtained by a 3×3 convolution, and the initial hidden state is obtained. Initialized from contextual features via 3×3 convolution;

[0046] The Charbonnier loss L ch Based on the reconstructed high-resolution frame X HR Compared with the real high-resolution reference frame X GT Calculation of pixel residuals between them;

[0047] The low-light texture weighted loss is: ;

[0048] The dynamic range constraint loss is a weighted sum of the detail preservation error in bright areas and the detail preservation error in dark areas, and:

[0049] ;

[0050] in, This represents the pixel value of the reconstructed high-resolution image at spatial coordinates (i,j); The pixel value of the real high-resolution reference image at spatial coordinates (i,j) as the training supervision label; This represents the normalized illumination intensity value of the pixel-by-pixel illumination map at spatial coordinates (i,j); MSE represents the mean square error. and All are high-resolution images X output by the reconstruction module. HR The pixel region extracted under a specified luminance mask; and For a true high-resolution reference image X GT In the corresponding region under the same brightness mask; λ is the weighting coefficient for the detail preservation error in the bright area;

[0051] The upsampling factor r of the pixel rearrangement method is 4; the fused features F are then processed using a 3×3 convolution. k f The mapping is used to reconstruct features, and the number of channels is set to 3×r. 2 =48; According to the pixel rearrangement rule, each consecutive r in the reconstructed feature of size H×W×48 is... 2 The channels are rearranged into spatial dimensions to obtain a high-resolution residual image of size 4H×4W×3; the base image (4 times the size of the current low-resolution RGB frame obtained by bicubic interpolation) is added pixel by pixel to the high-resolution residual image, and then the final high-resolution frame X is obtained by output convolution and range constraint [0,1]. HR .

[0052] An event-driven, texture-enhanced video super-resolution method for fully illuminated dynamic scenes includes:

[0053] Data acquisition module: includes rigidly installed RGB cameras and event cameras with a common field of view, which acquire low-resolution RGB video sequences, event streams and scene average illumination intensity through shared hardware triggering and a unified time base, and record the corresponding timestamps;

[0054] Preprocessing module: used to correct distortion of RGB frames and event coordinates, reproject events onto the RGB imaging plane, complete event timestamp correspondence according to the exposure range of RGB frames, normalize RGB frames, sort the event stream and remove invalid events, and output spatiotemporally aligned RGB video frames and synchronization event packets.

[0055] Light perception and event adaptive coding module: used to calculate pixel-by-pixel light intensity map and pixel-by-pixel light intensity based on RGB frames, configure the global trigger threshold of event camera, and perform region-by-region software secondary filtering, isolated event and hot pixel filtering, time bin number adaptive voxel coding and 98 percentile truncation normalization on synchronous event stream to generate normalized event voxel grid.

[0056] The dual-branch feature learning module includes a temporal alignment branch and an event texture enhancement branch. The temporal alignment branch uses bidirectional optical flow estimation and a context extractor to generate contextual features, while the event texture enhancement branch uses an event texture extractor and a block-by-block iterative texture updater to generate high-frequency texture features.

[0057] The illumination adaptive fusion module is used to dynamically calculate the weights of the temporal alignment branch and the event texture enhancement branch based on the pixel-by-pixel illumination intensity map, obtain weighted temporal features and weighted texture features respectively, and then concatenate and fused them with the illumination guidance features to obtain the fused features.

[0058] Reconstruction module: used to convolve the fused features into a 3×r matrix. 2 The reconstructed features of each channel are obtained, and pixel rearrangement is used for r-fold upsampling. The upsampling residual is added to the interpolation result of the low-resolution RGB video frame to output a high-resolution high dynamic range video sequence.

[0059] The beneficial effects of this invention are as follows: This invention utilizes a dual-branch architecture that combines temporal alignment and event texture enhancement within a bidirectional temporal propagation architecture, fusing low-resolution RGB video sequences with neuromorphic event stream information. It achieves hierarchical denoising and normalization encoding of the event stream across the entire illumination range through a lighting perception and event adaptive coding module, solving the problems of sparse low-light events and noise redundancy in strong-light events. Furthermore, it extracts high-frequency texture details through a block-by-block progressive texture iteration optimization module using dynamic scheduling of block masks, dynamically assigns feature weights pixel-by-pixel using a lighting adaptive dual-branch fusion mechanism, and optimizes network training using a multi-constraint joint loss function that includes low-light texture weighted loss and dynamic range constraint loss. This improves the reconstruction accuracy and texture restoration quality of video super-resolution in full-illumination scenarios, expands the imaging dynamic range, and simultaneously accommodates real-time inference on edge platforms. Simulation and edge deployment tests show that, under full illumination dynamic scenes ranging from 0.1 lux to 10000 lux and with 4x video super-resolution, compared to pure RGB or event-assisted video super-resolution benchmarks using fixed fusion weights, the peak signal-to-noise ratio (PSNR) of this invention is improved by 1.2 dB to 2.1 dB. In strong backlight scenes where bright and dark areas coexist, the effective dynamic range of the output is extended from approximately 60 dB to over 115 dB. On the NVIDIA Jetson Xavier NX edge platform, the processing speed for 1080p output video reaches 30 frames per second. These quantitative results demonstrate that this invention can balance texture restoration accuracy, brightness and darkness detail preservation, and real-time processing performance under extreme lighting conditions, effectively supporting the intelligent sensing needs of complex applications such as security monitoring, vehicle-mounted assisted driving, and industrial inspection. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a diagram of the overall architecture of the present invention.

[0062] Figure 2 This is a flowchart illustrating the calculation process of the illumination sensing and event adaptive coding module of the present invention.

[0063] Figure 3 This is a network structure diagram of the Progressive Texture Iteration Optimization (PTO) module of this invention.

[0064] Figure 4 This is a logic diagram of the illumination adaptive feature fusion and reconstruction module of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Example 1

[0067] A texture enhancement video super-resolution method based on event-driven principles and oriented towards dynamic scenes under full illumination is proposed. Based on a bidirectional temporal propagation architecture, it employs a dual-branch structure with a temporal alignment branch and an event texture enhancement branch running in parallel. Spatiotemporal synchronization between low-resolution RGB video sequences and event streams is achieved through shared hardware trigger signals, a unified clock reference, and sensor calibration transformation. The method includes an illumination-aware and event-adaptive coding module, a dual-branch feature learning module, a block-by-block progressive texture iterative optimization (PTO) module, an illumination-adaptive fusion module, and a reconstruction module. It is trained using a multi-constraint joint loss function that includes low-illumination texture weighting loss and dynamic range constraint loss. The temporal alignment branch (RGB branch) of the dual-branch feature learning module preserves scene color, brightness, and low-frequency structure, while the event texture enhancement branch supplements edge variations and high-frequency textures under extreme illumination and fast motion conditions. By adjusting the event filtering intensity, texture iteration count, and dual-branch fusion weights according to the illumination region, low-light noise, strong-light redundant events, motion blur, and cross-frame misalignment can be reduced, improving texture restoration accuracy while controlling computational overhead. This invention can improve PSNR by 1.2–2.1 dB under extremely low illumination of 0.1 lux, extending the effective dynamic range from approximately 60 dB to over 115 dB, and achieving 30 frames / s @ 1080p real-time inference on the NVIDIA Jetson Xavier NX edge platform. It is suitable for complex applications such as security monitoring, vehicle-mounted driver assistance systems, and industrial inspection. Figure 1 As shown, this invention forms a complete processing chain from synchronous data acquisition, preprocessing, dual-branch feature learning, adaptive fusion to super-resolution video reconstruction, including the following steps:

[0068] S1: Acquire spatiotemporally synchronized low-resolution RGB video sequences and event streams, and preprocess them separately.

[0069] A low-resolution RGB video sequence consists of multiple frames of color images captured continuously at a fixed frame rate, denoted as I = {I1, I2, ..., I...} n}, where n is the number of color images in the RGB video sequence; the event stream consists of a set of events asynchronously output by the event camera, each event denoted as e. i =(x iy i , t i p i ), where x i y i Let t be the pixel coordinates of the i-th event. i For microsecond-level timestamps, p i ∈{+1, -1} represents the polarity of brightness variation. The RGB camera and the event camera are fixed with a rigid bracket to keep their relative pose unchanged during the acquisition process.

[0070] Time synchronization is achieved using shared hardware triggering and a unified clock reference: the synchronization controller provides the RGB camera and event camera with the same trigger pulse, or the internal clocks of the two types of sensors are calibrated using a PPS / PTP clock; the exposure start time t is recorded for each RGB frame. ns and exposure end time t ne And select timestamps that satisfy t ns ≤t i <t ne Alternatively, an event located between the timestamps of two adjacent frames can be used as the synchronization event packet corresponding to that RGB frame. When a fixed transmission delay exists, the time offset Δt is obtained through offline calibration and denoted as t. i ’ =t i -Δt corrects the event timestamp.

[0071] Spatial synchronization is achieved through joint geometric calibration: the intrinsic and distortion parameters of the RGB camera and the event camera are calibrated separately, and the extrinsic transformation matrix from the event camera coordinate system to the RGB camera coordinate system is obtained using a checkerboard or dot array calibration board; first, the RGB pixels and event coordinates are distorted, and then the event coordinates are reprojected onto the RGB imaging plane according to the extrinsic transformation matrix, and the event position with the same spatial resolution as the RGB frame is obtained through bilinear interpolation. Events that exceed the common field of view after reprojection are deleted, and the two data streams are cropped to a common valid area. Joint geometric calibration and spatial synchronization can achieve a one-to-one spatial correspondence between RGB pixels and event coordinates: on the one hand, it eliminates edge distortion and coordinate deviation caused by lens distortion, and on the other hand, it corrects the positional misalignment caused by parallax of dual-sensor installation, so that the event edges and RGB textures are accurately aligned on the imaging plane.

[0072] The core effect of spatiotemporal synchronization is to avoid problems such as edge misalignment, ghosting, and pseudo-texture in subsequent feature fusion and texture reconstruction, and to ensure accurate matching between high-frequency event information and low-frequency RGB structure. This is the basis for the spatial accuracy of dual-modal joint super-resolution.

[0073] Preprocessing includes: dark current correction, pixel value normalization, and necessary lightweight denoising for RGB frames; sorting the event stream by timestamp, removing out-of-bounds coordinates, timestamp anomalies, and duplicate events, and unifying event polarity encoding. This preprocessing eliminates parallax, lens distortion, and clock skew between the two sensors, ensuring that event edges correspond to RGB textures at the same location and within the same exposure time, reducing ghosting, edge misalignment, and pseudo-textures in subsequent optical flow estimation, feature fusion, and super-resolution reconstruction.

[0074] S2: Perform pixel-by-pixel illumination sensing on the preprocessed low-resolution RGB video frames to generate a refined pixel-by-pixel illumination intensity map; perform dual-layer adaptive coding and spatiotemporal joint denoising on the event stream based on the pixel-by-pixel illumination intensity map to generate a normalized event voxel grid.

[0075] Figure 2 The parallel processing of the low-resolution RGB video frame branch and the synchronous event stream branch is demonstrated. The low-resolution RGB video frames are used only to generate pixel-by-pixel illumination maps and the scene's average illumination intensity; the event stream does not participate in the calculation of the pixel-by-pixel illumination map. The synchronous event stream, after global hardware threshold configuration, undergoes region-by-region software secondary filtering, spatiotemporal noise filtering, event voxel grid encoding, and normalization processing, ultimately outputting a pixel-by-pixel illumination map and a normalized event voxel grid.

[0076] The method for generating the pixel-by-pixel illumination intensity map is as follows: The preprocessed RGB video frame is converted to the YCbCr color space, and the Y channel representing brightness information is extracted as the initial illumination intensity map. A 5×5 Gaussian filter is used to smooth the initial illumination intensity map; that is, a 5×5 Gaussian kernel is used to perform convolution filtering on the initial illumination intensity map to suppress the interference of sensor noise and local texture details on illumination judgment, resulting in a smooth illumination map. The smoothed brightness map is then linearly normalized to normalize the pixel values ​​to [0, 1], obtaining the pixel-by-pixel illumination intensity map. Using the Y channel can characterize the scene's brightness distribution while minimizing the influence of chromaticity components. Gaussian filtering can suppress the interference of sensor random noise and isolated bright spots on illumination judgment, creating a continuous and stable weight transition between low-light, normal-light, and high-light areas. This provides a consistent illumination basis for calculating local event thresholds, block iteration times, and dual-branch fusion weights.

[0077] The specific process of dual-layer adaptive coding and spatiotemporal joint denoising includes: 1) Global hardware threshold pre-configuration: Before data acquisition, the global trigger threshold of the event camera is set according to the average illumination intensity of the scene; 2) Region-by-region software secondary filtering: Local threshold processing is performed on the acquired event stream according to the pixel-by-pixel illumination intensity map; 3) Spatiotemporal noise filtering: A spatiotemporal window composed of a 3×3 spatial neighborhood and a preset time window is used to filter isolated noise events, and hot pixels are identified and filtered using event frequency statistics; 4) Event voxel grid coding and normalization: The number of time bins is adaptively determined according to the event density, the filtered events are encoded into a voxel grid, and 98 percentile truncation normalization is performed to obtain a normalized event voxel grid.

[0078] The scene average illumination intensity is preferably obtained by an illuminance sensor synchronized with the camera: the most recent illuminance value is read before the start of each RGB video frame exposure, and the average of 3 to 5 consecutive readings is taken to reduce instantaneous fluctuations; if no illuminance sensor is configured, an illuminance mapping table can be established based on the pre-calibrated exposure time, analog gain, and Y channel average of the RGB camera, and the spatial average of the Y channel of the current frame can be converted into an estimated illuminance. The obtained scene average illumination intensity is used to select the global trigger threshold of the event camera, while the normalized pixel-by-pixel illumination intensity map is used for subsequent local software filtering and block-based iterative control.

[0079] The global trigger threshold represents the logarithmic change in brightness required to trigger an event. Specifically, it is set to 0.05 for low-light scenes (<1 lux), 0.1 for normal lighting scenes (1–1000 lux), and 0.2 for strong light scenes (>1000 lux). Using a lower threshold under low-light conditions improves the sensor's response to subtle brightness changes and prevents the loss of effective edges due to sparse events. Using a higher threshold under strong light conditions reduces high-frequency invalid triggers caused by flicker, readout disturbances, and small brightness fluctuations, thus reducing event bandwidth, storage requirements, and subsequent computational load. A middle threshold is used under normal lighting conditions to balance event integrity and noise suppression.

[0080] The region-by-region software secondary filtering is performed after the event coordinates are reprojected onto the RGB plane. For event e... i According to its position (x i y i Read the pixel-by-pixel illumination intensity L(x) i y i The normalized event confidence magnitude α is calculated by combining the neighborhood support of the event in the local time surface. i In the low-light region L < 0.2, a lower software threshold of 0.03 is used to retain the normalized event confidence amplitude a. iWeak events ≥ 0.03; in strong light regions L > 0.8, a higher software threshold of 0.15 is used, and the normalized event confidence amplitude α is removed. i Low-confidence events with a threshold value <0.15 are considered low-confidence events. The threshold for normally lit areas is linearly interpolated between 0.03 and 0.15 according to the illumination intensity L. As a result, low-light areas can retain sparse events generated by weak textures, while high-light areas prioritize the retention of real edge events with stable neighborhood support and suppress redundant noise, thus achieving differentiated processing for different lighting areas within the same frame.

[0081] Figure 2 The two types of inputs are processed along independent branches: RGB video frames are used to calculate pixel-by-pixel illumination intensity maps, while the event stream is not input to the illumination intensity map generation module but instead enters the event filtering branch as data to be encoded. The pixel-by-pixel illumination intensity map provides the event filtering branch with control parameters for local thresholds and the number of time bins, thus forming a collaborative relationship where "RGB provides illumination priors and the event stream provides dynamic textures."

[0082] Isolated noise filtering specifically involves: establishing a 3×3 spatial neighborhood and a ±Δt time window centered on the event to be judged; counting the number of neighboring events with the same polarity or continuous time within the time window; when the number of neighboring events is less than a preset threshold, the event is judged as background activity noise and deleted. Hot pixel statistical filtering specifically involves: calculating the event trigger count for each pixel within a statistical window of length T; when the trigger count of a pixel is consistently higher than the sum of the average event frequency and three times the standard deviation of the entire image, and the correlation of its neighboring events is lower than a preset threshold, the pixel is added to a hot pixel mask, and its output is masked in subsequent event packets.

[0083] The event voxel grid encoding process involves: normalizing the event time interval corresponding to the current RGB video frame to [0, B-1], where B is the number of time bins; linearly interpolating and accumulating each event to adjacent time bins according to its polarity and normalized time position to form an event voxel grid of size B×H×W. The number of time bins B is adaptively selected based on the event density per unit time and per pixel: 3 time bins are used when the event density is sparse to avoid excessive event dispersion; 5 time bins are used when the event density is moderate; and 7 time bins are used when the event density is dense or the movement is fast to preserve finer temporal variations. After encoding, the 98th percentile q of the non-zero absolute value of the voxel grid is calculated. 98 The voxel values ​​are truncated to [-q]. 98 q 98 Divide by percentile q 98This normalizes the input to [-1, 1]. This process suppresses the influence of a few abnormally high responses on the feature scale, prevents the network gradient from being dominated by extreme events, and improves the consistency of the input distribution under different lighting and event densities. H and W are the spatial height and spatial width of the event voxel grid, respectively, in pixels; their values ​​are consistent with the event spatial resolution after reprojection onto the RGB imaging plane, and are completely aligned with the width and height of the low-resolution RGB frame, ensuring that the event features and RGB features are strictly matched in spatial dimension, providing a spatial alignment basis for subsequent dual-branch feature fusion.

[0084] S3: A dual-branch structure based on a bidirectional temporal propagation architecture is adopted for multi-scale feature learning. The temporal alignment branch of the dual-branch structure performs bidirectional optical flow estimation, feature deformation, and temporal aggregation on the current RGB video frame and its preceding and following frames to obtain temporally consistent contextual features; the event texture enhancement branch of the dual-branch structure performs multi-scale encoding on the normalized event voxel grid to obtain event texture features in different time chambers. The pixel-by-pixel illumination intensity map does not directly replace image or event features, but is used to generate a block-by-block iterative mask and control the effective update area of ​​the PTO module; the PTO module inputs the contextual features, event texture features, and the iterative mask into the ConvGRU iterative updater to obtain high-frequency texture features.

[0085] Figure 3 This demonstrates the process where the context extractor and event texture extractor extract temporal context features and event texture features respectively, and then the high-frequency texture features are successively updated and output by ConvGRU units and residual block groups driven by block masks. The dual-branch structure is a functional division of the network, and the dual feature extractors are feature extraction components within the dual branches. The temporal alignment branch includes optical flow estimation, bidirectional feature deformation, occlusion validity judgment, and the context extractor. Its function is to supplement stable colors, contours, and low-frequency structures using adjacent RGB frames and to eliminate cross-frame misalignment caused by motion. The event texture enhancement branch includes the event texture extractor and the PTO block-by-block iterative texture updater. Its function is to recover edges and high-frequency textures that are difficult to observe in RGB video frames in fast-moving, low-light, and backlit areas using the high temporal resolution of the event stream. The outputs of the context extractor and the event texture extractor are used together in the PTO module, but they correspond to the two branches mentioned above and do not constitute a third independent branch.

[0086] S31: The temporal alignment branch uses high-precision optical flow estimation to perform sub-pixel-level alignment of adjacent frames before and after RGB video frames, generating contextual features that are consistent with the current target frame in terms of spatial position and motion state.

[0087] High-precision optical flow estimation employs either a pyramid optical flow network or a FlowNet network, with a preference for a pyramid optical flow network possessing multi-scale estimation capabilities. For the current target frame I... k and adjacent frame I k+m 'm' is the inter-frame temporal offset, representing the difference in frame sequence number between adjacent frames and the current target frame, used to identify the frame position in the temporal context. In the bidirectional temporal propagation architecture of this invention, the inter-frame temporal offset 'm' is defaulted to ±1, i.e. The previous adjacent frame of the current frame, The current frame is the next adjacent frame. If the context is extended to multiple frames, m can take values ​​of ±2, ±3, etc., corresponding to a larger temporal neighborhood. The optical flow network outputs a two-dimensional displacement field (u, v) from the adjacent frame to the target frame. Bilinear interpolation sampling is used at the continuous coordinates (x+u, y+v) of the feature map to deform the features of the adjacent frames to the coordinate system of the target frame. Since the displacement and sampling coordinates are both floating-point numbers, sub-pixel alignment of less than one pixel can be achieved, resulting in aligned features. Subsequently, a valid mask is generated using forward-backward optical flow consistency to remove occluded areas and unreliable optical flow areas. The valid forward and backward aligned features are concatenated with the current frame features and aggregated by 3×3 convolution to obtain the temporally consistent context feature F. kc This process can reduce ghosting at the edges of moving objects and supplement the current frame with stable structural information missing from previous and subsequent video frames.

[0088] The context extractor first uses 3×3 convolutions to map the current frame features and aligned features from the forward and backward directions to a uniform number of channels, then sequentially passes them through eight concatenated residual blocks; the output of the previous residual block serves as the input to the next. Each residual block processes the input features in the order of "3×3 convolution - batch normalization - ReLU activation function - 3×3 convolution - residual summation": the first convolutional layer extracts the local spatial structure, the batch normalization layer stabilizes the feature distribution under different frames and lighting conditions, the ReLU activation function introduces non-linear expression, and the second convolutional layer refines the structural features; the output of the second convolutional layer is added element-wise to the input of the residual block, allowing the network to learn incremental contextual information while preserving the original structure and improving gradient propagation in deep networks. The eight residual blocks progressively expand the effective receptive field, and their final output serves as the context feature F. kc .

[0089] S32: The event texture enhancement branch uses the event texture extractor of the block progressive texture iteration optimization (PTO) module to extract multi-scale event texture features from the normalized event voxel grid, and adopts a combination of globally unified maximum iteration count and block mask dynamic update to achieve high-frequency texture restoration with lighting adaptation.

[0090] The PTO module includes a dual feature extractor and a block-by-block iterative texture updater. The context extractor within the dual feature extractor belongs to the temporal alignment branch, and is used to process the current RGB frame and its forward and backward alignment features, outputting the context feature F. kc The event texture extractor, belonging to the event texture enhancement branch, is used to process normalized event voxel meshes and output event texture features according to the time chamber. G is the output identifier of the event texture extractor, representing the raw event texture features extracted directly from the event voxel mesh (without iterative optimization). The two types of features are jointly modeled in the block-by-block iterative texture updater.

[0091] The block-by-block iterative texture updater is built upon a ConvGRU unit that supports block mask input. In the b-th iteration, the hidden state from the previous iteration, context features, and event texture features from the b-th time bin are input into the ConvGRU unit to obtain candidate hidden states. Then, the b-th iteration mask is used to update only the image blocks that need to continue recovery; image blocks that have stopped iterating directly inherit the previous hidden state. The residual block group predicts the residual texture based on the updated hidden state and adds the residual to the texture features from the previous iteration, thereby successively supplementing high-frequency details without destroying the already recovered stable structure.

[0092] The block-by-block iterative texture updater of the PTO module employs the following block-by-block iterative mechanism: a globally unified maximum number of iterations is set to 7; the normalized pixel-by-pixel illumination intensity map is divided into 8×8 pixel non-overlapping image blocks, and the average illumination intensity of each image block is calculated to generate binary iterative masks for different iteration rounds; when the resolution of the illumination map and the feature map are inconsistent, the nearest neighbor method is used to scale the binary iterative mask to the corresponding feature scale. In each iteration, only the hidden state and texture features of the ConvGRU unit are updated for image blocks with a mask of 1, while image blocks with a mask of 0 directly retain the results of the previous iteration, thus allocating more computation to regions with severe texture loss.

[0093] For any 8×8 image patch, the average illumination intensity is obtained by the arithmetic mean of the 64 normalized illumination pixel values ​​within that patch. The generation rule for the iterative mask is as follows: low-light patches (average illumination intensity < 0.2) have a mask of 1 in the first 1 to 7 iterations; normal-light patches (0.2 ≤ average illumination intensity ≤ 0.8) have a mask of 1 in the first 1 to 5 iterations and 0 in the 6th to 7th iterations; and high-light patches (average illumination intensity > 0.8) have a mask of 1 in the first 1 to 3 iterations and 0 in the 4th to 7th iterations. This allows low-light areas to accumulate sparse event texture through more iterations, normal areas to achieve moderate refinement, and high-light areas to avoid introducing false edges and additional computation due to excessive iteration.

[0094] The context extractor consists of eight cascaded residual blocks, each containing two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function. The first 3×3 convolutional layer extracts edges, contours, and local structures within the current receptive field. The batch normalization layer unifies feature scale and accelerates training convergence. The ReLU activation function enhances non-linear expressive power. The second 3×3 convolutional layer reorganizes and refines the extracted structures. Let Z be the input to the residual block. The residual mapping R(Z) obtained after convolution, normalization, and activation is element-wise added to the input Z, resulting in the output Z′=Z+R(Z). This residual connection preserves original information, suppresses deep feature degradation, and enables the eight residual blocks to continuously learn incremental representations from local edges to a larger range of context.

[0095] The event texture extractor employs a 5-layer U-Net structure, comprising a 5-level encoder and a symmetrical 5-level decoder. In the encoding stage, each level extracts event texture through 3×3 convolutions and non-linear activations, and reduces the spatial size and increases the number of channels through stride convolutions or downsampling. In the decoding stage, each level upsamples to restore the spatial size, and the current decoded features are concatenated with the encoder's features at the same scale along the channel dimension, then fused using 3×3 convolutions. The deepest layer features are used to represent a large range of motion and event distribution, while shallow skip connections directly convey precise event location, polarity, and edge details. This structure can simultaneously utilize sparse local events and cross-regional motion context, reducing detail loss caused by downsampling and generating event texture features with consistent spatial resolution for different time chambers.

[0096] The block-by-block iterative texture updater comprises a block-by-block iterative control module, a ConvGRU module, a residual block group, a residual addition module, and an iteration count determination module, connected in sequence. The block-by-block iterative control module generates the b-th iteration mask M based on the pixel-by-pixel illumination intensity map. b The ConvGRU module will change the previous hidden state. Contextual features F kc Event texture features of the b-th time chamber Channel splicing is performed, and the amount of historical information retained and the amount of current event information written are controlled by update and reset gates to obtain candidate hidden states. According to the iterative mask M b Execution hidden state This keeps inactive image blocks unchanged. This indicates element-wise multiplication.

[0097] A residual block group containing 5 cascaded residual blocks will be in a hidden state. Mapped to the residual texture feature Δ of this round k b The residual addition module follows F k b =Fk b-1 +Δ k b Update texture features. Initial texture features F k 0 The initial hidden state is obtained by 3×3 convolution of the output of the event texture extractor. Initialized from context features via 3×3 convolution. The iteration count determination module checks if the effective iteration count for each image patch or the global maximum iteration count of 7 has been reached; if not, it returns to the ConvGRU module for further updates; if reached, it outputs the final high-frequency texture feature F. k T =F k B , where B is the maximum time bin or iteration round actually executed.

[0098] The update formula for the block-by-block iterative texture updater is:

[0099] ;

[0100] Where k represents the sequence number of the current RGB video frame to be reconstructed, b represents the current iteration round or the event time bin corresponding to that round; H k b This is the hidden state after the b-th iteration, used to store the fused temporal context, event changes, and texture restoration information up to the current round. H k b-1 This represents the hidden state of the previous round; Ξ represents the ConvGRU module operation function; F kc The context features are obtained by the context extractor based on the current frame and its forward and backward alignment features; This represents the event texture features extracted by the event texture extractor from the b-th temporal bin of the normalized event voxel grid in the previous adjacent frame; [·, ·] denotes channel concatenation. The ConvGRU module first uses the hidden state H from the previous round... k b-1 With [F kc , Calculate the update gate and reset gate, generate candidate hidden states, and combine them with the block-based iterative mask to obtain the hidden state H. k b Π represents the operation function for a group of 5 residual blocks, Δ k b F represents the residual texture feature predicted in the b-th iteration; k b-1 and F k b These represent the texture features before and after the update, respectively. Texture Feature F k bIf the effective number of iterations has not been reached, it is used as the input for the next round. After the termination condition is met, it is used as the final high-frequency texture feature input for the illumination adaptive fusion module in step S4.

[0101] S4: Dynamically calculate the adaptive fusion weights of the temporal alignment branch and the event texture enhancement branch based on the pixel-by-pixel illumination intensity map, and perform weighted pixel-by-pixel fusion of the temporal alignment features and the high-frequency texture features output by the PTO module to obtain the fused features.

[0102] Figure 4 The process demonstrates how to calculate dual-branch adaptive weights based on pixel-by-pixel illumination intensity, perform feature weighted fusion, and output high-resolution frames after pixel rearrangement and upsampling.

[0103] The formula for calculating adaptive fusion weights is:

[0104] ;

[0105] Among them, w A For time-aligned branch weights, w T The event texture is enhanced with branch weights, where α is an adaptive coefficient ranging from [0.5, 1.0], and L is the illumination intensity of the pixel-by-pixel illumination map; the two weights satisfy w A +w T =1. Figure 4 The feature of the event texture enhancement branch processing is the high-frequency texture feature F of the final output of the PTO module. k T The "T" stands for Texture, used to identify texture features, rather than event voxel meshes or intermediate event features that have not been iterated. When the illumination intensity L in normally lit areas is close to 0.5, the proportion of temporal alignment features is increased; when the illumination intensity L in low-light or high-light areas deviates from 0.5, the contribution of high-frequency texture features is increased to utilize event information to supplement dynamic edges that are difficult to recover from RGB branches.

[0106] Figure 4 In the middle, the time-aligned branch weight w A Timing alignment feature F kc Element-wise multiplication yields the weighted time series feature F. k A =w A ⊙F kc Event texture enhancement branch weight w T With high-frequency texture features F k T Element-wise multiplication yields the weighted texture feature F. k E =w T ⊙F k TThe pixel-by-pixel illumination intensity map is scaled to the same spatial size as the feature map and then copied along the channel dimension or mapped via a 1×1 convolution to form the illumination-guided feature F. k L Subsequently, the weighted time series features F k A Weighted texture features F k E and illumination guidance features F k L The sequence is stitched along the channel dimension and passed through three 3×3 convolutional layers in sequence. The first two convolutional layers are followed by a ReLU activation function, and the last convolutional layer outputs the fused feature F. k f This fusion method preserves the color and stable structure of the RGB branch, writes the high-frequency edges of the event texture enhancement branch to the corresponding positions according to the local lighting requirements, and uses lighting guidance features to reduce weight abrupt changes at the boundary between light and dark.

[0107] The training also includes a multi-stage training process: An end-to-end training optimization is performed on the optical flow estimation and temporal alignment modules, context extractor, event texture extractor, PTO block-by-block iterative texture updater, illumination adaptive fusion module, and reconstruction module using a total loss function that incorporates low-illumination texture weighting loss and dynamic range constraint loss. The optical flow estimation network can first load pre-trained parameters from a public dataset. In the first stage, its parameters are fixed to train the remaining modules. In the second stage, the entire network is jointly fine-tuned with a smaller learning rate to avoid unstable optical flow in the early stages of training from damaging temporal features. Total loss function:

[0108] ;

[0109] Among them, L ch For Charbonnier loss, L low For low-light texture weighting loss, L dr The dynamic range constraint loss is used, with β1=0.8 and β2=0.3 as hyperparameters. The Charbonnier loss is based on the reconstructed high-resolution frame X. HR Compared with the real high-resolution reference frame X GT The calculation of pixel residuals between pixels involves applying the square of each pixel residual to the smoothing constant ε. 2 Take the square root of the sum and then the average; ε is preferably 10. -3This loss function is smoothly differentiable when the residual is close to zero, and it is more robust to outliers than mean squared error in the presence of motion occlusion, anomalous events, or local exposure differences. The total loss function uses Charbonnier loss to constrain overall color and structure reconstruction, low-light texture weighting loss to improve the training weights of dark area edges and details, and dynamic range constraint loss to simultaneously limit overexposure in bright areas and underexposure in dark areas. The joint use of these three losses avoids the problem of smoothing dark textures and saturating bright details caused by only pursuing average PSNR, thus achieving a balance between overall reconstruction accuracy, low-light texture recovery, and preservation of details in both bright and dark areas.

[0110] The formula for calculating the low-light texture weighted loss is:

[0111] ;

[0112] The dynamic range constraint loss is a weighted sum of the detail preservation errors in bright and dark areas, calculated using the following formula:

[0113]

[0114] in, This represents the pixel value of the reconstructed high-resolution image at spatial coordinates (i,j), where i and j are the row and column indices of the image pixel, respectively. The pixel value at spatial coordinates (i,j) of the real high-resolution reference image (Ground Truth), which serves as the training supervision label, is the baseline ground truth for calculating pixel-level reconstruction error. This represents the normalized illumination intensity value at spatial coordinates (i,j) in the pixel-by-pixel illumination intensity map, with a value range of [0,1]. MSE stands for Mean Squared Error, used to calculate the mean pixel error between the reconstructed image and the corresponding region of the ground truth label, quantifying the detail retention deviation in bright and dark areas. and All are high-resolution images X output by the reconstruction module. HR Pixel regions extracted under a specified brightness mask, rather than fused features; and For a true high-resolution reference image X GT The corresponding regions under the same mask. During training, the real high-resolution reference image is preferably acquired by a high-resolution RGB camera that is strictly synchronized with the low-resolution input, or the original high-resolution training video is used as the real label and downsampled by 4 times to generate the low-resolution input. X... GT Convert to YCbCr color space, generate a bright area mask Y > 0.8 and a dark area mask Y < 0.2 based on its Y channel, and then convert them from X... HR and X GTExtract the corresponding region and calculate the mean square error. λ is the weighting coefficient for the detail retention error in the bright area, which is set to 0.5; 0.5 in the formula is the weighting coefficient for the detail retention error in the dark area, so that the bright and dark areas have the same weight in the dynamic range constraint.

[0115] S5: Upsampling reconstruction is performed using fusion features to output high-resolution, high-dynamic-range video sequences.

[0116] Upsampling employs a pixel rearrangement method with an upsampling factor r of 4. First, a 3×3 convolution is used to fuse the features F. k f The mapping is used to reconstruct features, and the number of channels is set to 3×r. 2 =48; then, according to the pixel rearrangement rules, each consecutive r in the reconstructed feature of size H×W×48 is... 2 The channels are rearranged into spatial dimensions to obtain a high-resolution residual image with dimensions 4H×4W×3. The 4x base image obtained by bicubic interpolation of the current low-resolution RGB video frame is added pixel-by-pixel to the high-resolution residual image, and then the final high-resolution frame X is obtained through output convolution and [0,1] range constraints. HR Pixel rearrangement generates the positions of each sub-pixel at once through regular convolution, which avoids the checkerboard artifacts caused by uneven kernel overlap in transposed convolution. The above process is performed frame by frame and combined in the original temporal order to obtain a high-resolution, high-dynamic-range video sequence.

[0117] Upsampling can be achieved by using any of the following alternative pixel rearrangement methods: transposed convolution, nearest neighbor upsampling + convolution, or bicubic upsampling + convolution, to adapt to the computational characteristics of different hardware platforms.

[0118] Example 2

[0119] like Figure 1 As shown, an event-driven, texture enhancement video super-resolution system for dynamic scenes with full illumination is used to implement the method described in Embodiment 1, and includes the following interconnected functional modules:

[0120] Data acquisition module: includes a rigidly mounted RGB camera with a common field of view, a neuromorphic event camera, a synchronization controller, and an optional illuminance sensor, used to acquire low-resolution RGB video sequences, event streams, and scene average illuminance through shared hardware triggering and a unified time base, and record the corresponding timestamps.

[0121] Preprocessing module: Used to read the intrinsic parameters, distortion parameters and extrinsic parameter transformations of the two types of cameras, perform distortion correction on RGB video frames and event coordinates, reproject events onto the RGB imaging plane, complete the event timestamp correspondence according to the exposure range of RGB video frames, normalize RGB video frames, sort the event stream and remove invalid events, and output spatiotemporally aligned RGB video frames and synchronization event packets.

[0122] Light perception and event adaptive coding module: It is used to calculate the pixel-by-pixel light intensity map and the average light intensity of the scene based on RGB video frames, configure the global trigger threshold of the event camera, and perform region-by-region software secondary filtering, isolated event and hot pixel filtering, time bin number adaptive voxel coding and 98th percentile truncation normalization on the synchronous event stream to generate a normalized event voxel grid.

[0123] The dual-branch feature learning module includes a temporal alignment branch and an event texture enhancement branch. The temporal alignment branch uses bidirectional optical flow estimation and a context extractor to generate contextual features, while the event texture enhancement branch uses an event texture extractor and a PTO block-by-block iterative texture updater to generate high-frequency texture features.

[0124] The illumination adaptive fusion module is used to dynamically calculate the temporal alignment branch weights and event texture enhancement branch weights based on the pixel-by-pixel illumination intensity map, obtain weighted temporal features and weighted texture features respectively, and then concatenate and fused them with the illumination guidance features to obtain fused features.

[0125] Reconstruction module: used to convolve the fused features into a 3×r matrix. 2 The reconstructed features of each channel are obtained, and pixel rearrangement is used for r-fold upsampling. The upsampling residual is added to the interpolation result of the low-resolution RGB video frame to output a high-resolution high dynamic range video sequence.

[0126] Example 3

[0127] To verify the texture restoration, dynamic range expansion, and real-time edge inference performance of this invention in dynamic scenes with full illumination, a 4x video super-resolution experiment was conducted on a simulation test platform with synchronous low-resolution RGB video frames and event stream input. The illumination of the test scene covered 0.1 lux to 10000 lux, including extremely low light, normal lighting, and strong backlighting scenes with both bright and dark areas; the output resolution after 4x reconstruction of the input low-resolution video was 1920×1080. Comparison schemes included a pure RGB video super-resolution scheme and an event-assisted video super-resolution scheme using fixed fusion weights, with each scheme using the same test sequence, super-resolution factor, and evaluation conditions.

[0128] The simulation evaluation uses peak signal-to-noise ratio (PSNR), output effective dynamic range (EPR), and edge platform inference speed as the main indicators. PSNR is used to evaluate the overall reconstruction accuracy between the reconstructed frame and the high-resolution reference frame; EPR is used to evaluate the ability to simultaneously preserve details in bright and dark areas in strong backlight scenes; and inference speed is used to evaluate the real-time performance of the method on edge devices. Simulation and deployment test results are shown in Table 1.

[0129] Table 1. Simulation and Deployment Test Results of All-Illumination Video Super-Resolution Based on the Invention

[0130]

[0131] As shown in Table 1, this invention achieved positive reconstruction accuracy gain across the entire illumination test range. In extremely low-light scenes, the illumination perception threshold adjustment and weak event response preservation mechanism can supplement high-frequency textures that are submerged by noise in RGB frames. In strong backlight scenes, event denoising, dynamic range constraints, and pixel-by-pixel adaptive fusion can suppress invalid events in bright areas and preserve dark area structures, thereby achieving an effective output dynamic range of over 115dB. Simultaneously, the block-by-block progressive texture iteration only performs feature updates on image blocks that require further optimization, reducing invalid iteration overhead and enabling this invention to achieve real-time processing of 1080p video at 30 frames / s on the Jetson Xavier NX platform.

[0132] The aforementioned PSNR improvement range is a statistical result of different illuminance test sequences relative to the benchmark scheme. The dynamic range and operating speed are calculated from the output brightness response range and the number of frames processed per unit time, respectively. The results correspond to the beneficial effects of reconstruction accuracy, dynamic range and real-time performance described in the invention.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An event-driven, texture-enhancing video super-resolution method for dynamic scenes with full illumination, characterized in that, The steps are as follows: S1: Acquire spatiotemporally synchronized low-resolution RGB video sequences and event streams, and preprocess them separately; S2: Perform pixel-by-pixel illumination sensing on the preprocessed low-resolution RGB frame to generate a pixel-by-pixel illumination intensity map; perform dual-layer adaptive coding and spatiotemporal joint denoising on the event stream based on the pixel-by-pixel illumination intensity map to generate a normalized event voxel grid. S3: Using the temporal alignment branch of the dual-branch structure, bidirectional optical flow estimation, feature deformation, and temporal aggregation are performed on the current RGB frame and its preceding and following RGB frames to obtain temporally consistent contextual features; The event texture enhancement branch with a dual-branch structure is used to encode the normalized event voxel grid at multiple scales to obtain event texture features in different time chambers; A block-by-block iterative mask is generated based on the pixel-by-pixel illumination intensity map; the context features, event texture features and iterative mask are input together into the block-by-block iterative texture updater of the PTO module for iterative updating to obtain high-frequency texture features; S4: Dynamically calculate adaptive fusion weights based on the pixel-by-pixel illumination intensity map, and perform weighted pixel-by-pixel fusion of context features and high-frequency texture features to obtain fused features; S5: Upsampling reconstruction is performed using fusion features to output high-resolution, high-dynamic-range video sequences.

2. The event-driven, full-illumination dynamic scene texture enhancement video super-resolution method according to claim 1, characterized in that, The spatiotemporal synchronization includes time synchronization and spatial synchronization. The time synchronization is achieved by shared hardware triggering and a unified clock reference. The spatial synchronization is achieved through joint geometric calibration. The preprocessing includes: performing dark current correction, pixel value normalization, and lightweight noise reduction on RGB frames; sorting the event stream by timestamp, deleting out-of-bounds coordinates, abnormal timestamps, and duplicate events, and unifying event polarity encoding. The method for implementing the joint geometric calibration is as follows: the pixel coordinates of the RGB frame and the event coordinates of the event stream are distorted, and the event coordinates are reprojected onto the RGB imaging plane according to the extrinsic transformation matrix from the event camera coordinate system to the RGB camera coordinate system. The event positions with the same spatial resolution as the RGB frame are obtained through bilinear interpolation. Events that exceed the common field of view after reprojection are deleted, and the two data streams are cropped to a common effective area.

3. The event-driven, fully illuminated dynamic scene texture enhancement video super-resolution method according to claim 1 or 2, characterized in that, The method for generating the pixel-by-pixel illumination intensity map is as follows: the preprocessed RGB frame is converted into the YCbCr color space, and the Y channel is extracted as the initial illumination intensity map; the initial illumination intensity map is smoothed by a 5×5 Gaussian filter, and the pixel values ​​are normalized to [0, 1] to obtain the pixel-by-pixel illumination intensity map. The implementation method of dual-layer adaptive coding and spatiotemporal joint denoising includes: 1) global hardware threshold pre-configuration: setting the global trigger threshold of the event camera according to the average illumination intensity of the scene before event stream acquisition; 2) region-by-region software secondary filtering: performing local threshold processing on the acquired event stream according to the pixel-by-pixel illumination intensity map; 3) spatiotemporal noise filtering: using a spatiotemporal window composed of a 3×3 spatial neighborhood and a preset time window to filter isolated noise events, and using an event frequency statistical method to identify and filter hot pixels; 4) event voxel grid coding and normalization: adaptively determining the number of time bins according to the event density, encoding the filtered events into a voxel grid, and performing 98th percentile truncation normalization to obtain a normalized event voxel grid.

4. The event-driven, fully illuminated dynamic scene texture enhancement video super-resolution method according to claim 3, characterized in that, The temporal alignment branch uses optical flow estimation to perform sub-pixel-level alignment of the current RGB frame with the adjacent RGB frames before and after it, and obtains alignment features; A validity mask is generated using forward-backward optical flow consistency, removing occluded and unreliable optical flow regions. Alignment features from adjacent forward and backward RGB frames are concatenated with the current RGB frame features, and then aggregated by a context extractor to obtain temporally consistent context features F. kc ; The event texture enhancement branch uses the event texture extractor of the PTO module to extract multi-scale event texture features from the normalized event voxel grid. The block-by-block iterative texture updater combines a globally unified maximum number of iterations with block-by-block iterative mask dynamic updates to achieve illumination-adaptive high-frequency texture restoration and obtain high-frequency texture features.

5. The event-driven, fully illuminated dynamic scene texture enhancement video super-resolution method according to claim 4, characterized in that, The optical flow estimation employs a pyramid optical flow network or a FlowNet network; The context extractor includes a 3×3 convolution and eight cascaded residual blocks arranged sequentially; the 3×3 convolution maps the features of the current RGB frame to the alignment features of the forward and backward adjacent RGB frames to a uniform number of channels. Each residual block contains a first convolutional layer, a batch normalization layer, a ReLU activation function, a second convolutional layer, and a residual summation layer connected in sequence. The output of the residual summation second convolutional layer is added element-wise to the input of the residual block. The event texture extractor adopts a 5-layer U-Net structure, including a 5-level encoder and a symmetrical 5-level decoder. Each level encoder extracts event textures through 3×3 convolution and non-linear activation, and reduces the spatial size and increases the number of channels through stride convolution or downsampling. Each level decoder upsamples step by step to restore the spatial size, and concatenates the current decoded features with the encoder features of the same scale along the channel dimension, and then fuses them through 3×3 convolution; thus generating event texture features with consistent spatial resolution for different time chambers. The block-by-block iterative texture updater includes a block-by-block iterative control module, a ConvGRU module, a residual block group, a residual addition module, and an iteration count determination module connected in sequence. The block-based iterative control module generates the iterative mask M for the b-th iteration based on the pixel-by-pixel illumination intensity map. b The ConvGRU module will change the previous hidden state. Contextual features F kc Event texture features of the b-th time chamber Channel splicing is performed, and candidate hidden states are obtained by controlling the amount of historical information retained and the amount of current event information written through update and reset gates. According to the iterative mask M b Obtain the hidden state in the b-th iteration. ,in, This represents the element-wise multiplication operator; a residual block group containing 5 concatenated residual blocks will be hidden. Mapped to the residual texture feature Δ of this round k b The residual addition module updates the texture feature F. k b =F k b-1 +Δ k b Among them, F k b-1 and F k b These represent the texture features before and after the update, respectively. The iteration count determination module checks whether the effective iteration count corresponding to each image patch of the pixel-by-pixel illumination intensity map or the globally unified maximum iteration count has been reached. If not, it returns to the ConvGRU module to continue iterating; if reached, it outputs the final high-frequency texture feature F. k T =F k B B represents the maximum timeframe or iteration round actually executed.

6. The event-driven, fully illuminated dynamic scene texture enhancement video super-resolution method according to claim 5, characterized in that, The iteration mask M for generating the b-th iteration b The method is as follows: the maximum number of iterations is set globally to 7, the pixel-by-pixel illumination intensity map is divided into 8×8 pixel non-overlapping image blocks, and the average illumination intensity of each image block is calculated to generate iterative masks for different iteration rounds. When the resolutions of the illumination map and the feature map are inconsistent, the iterative mask is scaled to the corresponding feature scale using the nearest neighbor method. In each iteration, the hidden state and event texture features of the ConvGRU module are updated only for image blocks with an iteration mask of 1, while image blocks with a mask of 0 directly retain the results of the previous iteration. The average illumination intensity of the image patch is obtained by the arithmetic mean of 64 normalized illumination pixel values ​​within the image patch; the generation rule for the iterative mask is: when the average illumination intensity is <0.2, the iterative mask is 1 in the first to seventh iterations; When the average illumination intensity is 0.2 ≤ average illumination intensity ≤ 0.8, the iteration mask is 1 in the first to fifth iterations and 0 in the sixth to seventh iterations; When the average illumination intensity is >0.8, the iteration mask is 1 in the first to third iterations and 0 in the fourth to seventh iterations.

7. The event-driven, fully illuminated dynamic scene-oriented texture enhancement video super-resolution method according to any one of claims 4-6, characterized in that, The formula for calculating the adaptive fusion weights is as follows: Among them, w A For time-aligned branch weights, w T Enhance the branch weights for event textures, where α is an adaptive coefficient with a value range of [0.5, 1.0], and satisfies w A +w T =1; L is the illumination intensity of the pixel-by-pixel illumination intensity map; Align the time-series branch weights w A Timing alignment feature F kc Element-wise multiplication yields the weighted time series feature F. k A =w A ⊙F kc Event texture enhancement branch weight w T With high-frequency texture features F k T Element-wise multiplication yields the weighted texture feature F. k E =w T ⊙F k T The pixel-by-pixel illumination intensity map is scaled to the same spatial size as the feature map and then copied along the channel dimension or mapped via a 1×1 convolution to form the illumination-guided feature F. k L ;Weigh the time series features F k A Weighted texture features F k E and illumination guidance features F k L The sequence is stitched along the channel dimension and passed through three 3×3 convolutional layers in sequence. The first two convolutional layers are followed by a ReLU activation function, and the last convolutional layer outputs the fused feature F. k f .

8. The event-driven, full-illumination dynamic scene texture enhancement video super-resolution method according to claim 7, characterized in that, The upsampling is performed using any one of the following: pixel rearrangement, transposed convolution, nearest neighbor upsampling + convolution, and bicubic upsampling + convolution. An end-to-end training optimization is performed on optical flow estimation, context extractor, event texture extractor, block-by-block iterative texture updater, adaptive fusion, and reconstruction using a total loss function that includes low-illumination texture weighting loss and dynamic range constraint loss. The total loss function is as follows: Among them, L ch For Charbonnier loss, L low For low-light texture weighting loss, L dr The loss is a dynamic range constraint loss, where β1 and β2 are both hyperparameters.

9. The event-driven, full-illumination dynamic scene texture enhancement video super-resolution method according to claim 8, characterized in that, The average illumination intensity of the scene is obtained by an illuminance sensor synchronized with the RGB camera: the most recent illuminance value is read before the start of each RGB frame exposure, and the average of 3 to 5 consecutive illuminance values ​​is taken. When no illuminance sensor is configured, an illuminance mapping table can be established based on the pre-calibrated exposure time, analog gain, and Y channel mean of the RGB camera, and the spatial mean of the Y channel of the current frame can be converted into an estimated illuminance. The global trigger threshold represents the logarithmic change in brightness required to trigger an event. When the logarithmic change in brightness is <1 lux, the global trigger threshold is set to 0.05; when the logarithmic change in brightness is 1 to 1000 lux, the global trigger threshold is set to 0.1; and when the logarithmic change in brightness is >1000 lux, the global trigger threshold is set to 0.

2. The region-by-region software secondary filtering is performed after the event coordinates are reprojected onto the RGB imaging plane. The implementation method is as follows: based on the position (x, y) of the i-th event... i y i Read the illumination intensity L at the corresponding position in the pixel-by-pixel illumination intensity map, and calculate the normalized event confidence magnitude a by combining it with the neighborhood support of the i-th event in the local time surface. i When the light intensity L < 0.2, retain the normalized event confidence amplitude a. i Weak events ≥ 0.03; when light intensity L > 0.8, the normalized event confidence amplitude a is removed. i Low confidence events < 0.15; normalized event confidence magnitude a i When the value is between 0.03 and 0.15, it is linearly interpolated according to the light intensity L; The method for filtering isolated noise is as follows: a 3×3 spatial neighborhood and a time window with a fixed transmission delay of ±Δt are established with the event to be judged as the center, where Δt is the time offset of the fixed transmission delay. The number of neighborhood events with the same polarity or continuous time with the event to be judged within the time window is counted. When the number of neighborhood events is less than a preset threshold, the event to be judged is identified as background activity noise and deleted. The method for filtering hot pixels is as follows: calculate the number of event triggers for each pixel within the statistical window. When the number of event triggers for a certain pixel is consistently higher than the sum of the average event frequency of the entire image and three times the standard deviation, and the correlation of the neighboring events of the pixel is lower than a preset threshold, add the pixel to the hot pixel mask and block its output in subsequent event packets. The event voxel grid encoding method is as follows: normalize the event time interval corresponding to the current RGB frame to [0, B-1], where B is the number of time bins; each event is linearly interpolated and accumulated to the adjacent time bins according to its polarity and normalized time position to form an event voxel grid with size B×H×W, where H and W are the spatial height and spatial width of the event voxel grid, respectively. Calculate the 98th percentile q of the non-zero absolute value of the event voxel grid. 98 The voxel values ​​of the event voxel grid are truncated to [-q]. 98 q 98 Divide by percentile q 98 Normalized to [-1, 1]; The number of time bins, B, is adaptively selected based on the event density per unit time and per unit pixel: event density ≤ 0.05 events / (pixel) . s) retrieves 3 timeframes; 05 events / (pixel . s) < event density ≤ 0.2 events / (pixel) . s) ) Take 5 time bins, event density > 0.2 events / (pixel) . s) or when the movement is faster, take 7 time slots; Both the first and second convolutional layers are 3×3 convolutions; Initial texture features F k 0 The event texture features output by the event texture extractor are obtained by a 3×3 convolution, and the initial hidden state is obtained. Initialized from contextual features via 3×3 convolution; The Charbonnier loss L ch Based on the reconstructed high-resolution frame X HR Compared with the real high-resolution reference frame X GT Calculation of pixel residuals between them; The low-light texture weighted loss is: ; The dynamic range constraint loss is a weighted sum of the detail preservation error in bright areas and the detail preservation error in dark areas, and: ； in, This represents the pixel value of the reconstructed high-resolution image at spatial coordinates (i,j); The pixel value of the real high-resolution reference image at spatial coordinates (i,j) as the training supervision label; This represents the normalized illumination intensity value of the pixel-by-pixel illumination map at spatial coordinates (i,j); MSE represents the mean square error. and All are high-resolution images X output by the reconstruction module. HR The pixel region extracted under a specified luminance mask; and For a true high-resolution reference image X GT Corresponding regions under the same brightness mask; λ is the weighting coefficient for the detail retention error in bright areas; The upsampling factor r of the pixel rearrangement method is 4; the fused features F are then processed using a 3×3 convolution. k f The mapping is used to reconstruct features, and the number of channels is set to 3×r. 2 =48; According to the pixel rearrangement rule, each consecutive r in the reconstructed feature of size H×W×48 is... 2 The channels are rearranged into spatial dimensions to obtain a high-resolution residual image of size 4H×4W×3; the base image (4 times the size of the current low-resolution RGB frame obtained by bicubic interpolation) is added pixel by pixel to the high-resolution residual image, and then the final high-resolution frame X is obtained by output convolution and range constraint [0,1]. HR .

10. An event-driven, texture-enhancing video super-resolution method for fully illuminated dynamic scenes, characterized in that... include: Data acquisition module: includes rigidly installed RGB cameras and event cameras with a common field of view, which acquire low-resolution RGB video sequences, event streams and scene average illumination intensity through shared hardware triggering and a unified time base, and record the corresponding timestamps; Preprocessing module: used to correct distortion of RGB frames and event coordinates, reproject events onto the RGB imaging plane, complete event timestamp correspondence according to the exposure range of RGB frames, normalize RGB frames, sort the event stream and remove invalid events, and output spatiotemporally aligned RGB video frames and synchronization event packets. Light perception and event adaptive coding module: used to calculate pixel-by-pixel light intensity map and pixel-by-pixel light intensity based on RGB frames, configure the global trigger threshold of event camera, and perform region-by-region software secondary filtering, isolated event and hot pixel filtering, time bin number adaptive voxel coding and 98 percentile truncation normalization on synchronous event stream to generate normalized event voxel grid. The dual-branch feature learning module includes a temporal alignment branch and an event texture enhancement branch; The temporal alignment branch uses bidirectional optical flow estimation and a context extractor to generate contextual features, while the event texture enhancement branch uses an event texture extractor and a block-by-block iterative texture updater to generate high-frequency texture features. The illumination adaptive fusion module is used to dynamically calculate the weights of the temporal alignment branch and the event texture enhancement branch based on the pixel-by-pixel illumination intensity map, obtain weighted temporal features and weighted texture features respectively, and then concatenate and fused them with the illumination guidance features to obtain the fused features. Reconstruction module: used to convolve the fused features into a 3×r matrix. 2 The reconstructed features of each channel are obtained, and pixel rearrangement is used for r-fold upsampling. The upsampling residual is added to the interpolation result of the low-resolution RGB video frame to output a high-resolution high dynamic range video sequence.

Images