A multi-scale extremely sparse event enhancement method and device based on an RGB image
By employing a multi-scale sparse event enhancement method based on RGB images, and utilizing deep learning models and multi-scale U-shaped networks, the problem of missing structures and broken contours in sparse event maps is solved, achieving high-precision event map enhancement and obtaining more complete and pure event representations.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN POLYTECHNIC
- Filing Date
- 2026-05-09
- Publication Date
- 2026-07-31
AI Technical Summary
Existing technologies lack targeted structural constraints when processing extremely sparse event graphs, and cannot effectively fill in missing structures or repair contour breaks, thus failing to meet the high-precision requirements for extremely sparse event graphs in practical applications.
We employ a multi-scale extremely sparse event enhancement method based on RGB images. By constructing sparse ROI priors and training them with a deep learning model, we combine a multi-scale U-shaped network and RGB edge priors to achieve structural constraint injection. Event enhancement is then performed by combining ROI weighted reconstruction, new event region supervision, incremental sparse regularization, and background suppression as a joint loss.
Under the condition of extremely sparse event observation, the missing structure was completed and the background pseudo-events were suppressed, resulting in a more complete and reliable event representation, which improved the accuracy and purity of the extremely sparse event graph.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for enhancing multi-scale extremely sparse events based on RGB images. Background Technology
[0002] As a novel biomimetic vision sensor inspired by biological vision, the event camera differs from the traditional frame camera's fixed-frequency capture of the entire image. It only outputs pixel-level brightness change information, including pixel position, timestamp, and brightness change polarity, in the form of an asynchronous event stream when the brightness change of a single pixel exceeds a preset threshold. It has significant advantages such as high microsecond-level temporal resolution, low latency, high dynamic range, and low data redundancy. It has shown broad application prospects in many computer vision-related fields such as robot vision, autonomous driving, and augmented reality, and has become one of the research hotspots in the fields of event camera vision, multimodal fusion, and deep learning image processing technology.
[0003] In related technologies, some solutions involving the fusion of RGB images and events and biased towards event enhancement have failed to design specific optimization mechanisms for the characteristics of extremely sparse event maps with short time windows. In particular, they lack the design of targeted structural constraints for the missing regions of extremely sparse events, and cannot effectively fill in the missing structures in the event map or repair the contour breakage problem, thus failing to meet the high-precision requirements for the enhancement of extremely sparse event maps in practical applications.
[0004] Therefore, this invention proposes a multi-scale extremely sparse event enhancement method and apparatus based on RGB images to solve the above problems. Summary of the Invention
[0005] This invention describes a multi-scale extremely sparse event enhancement method and apparatus based on RGB images, which can enhance extremely sparse event maps with high precision.
[0006] According to a first aspect, the present invention provides a multi-scale extremely sparse event enhancement method based on RGB images, comprising: Based on historical RGB images and historical event streams of extremely sparse events, determine the ROI mask of extremely sparse events; The pre-defined deep learning model is trained using the dataset of the ROI extremely sparse region mask to obtain the trained extremely sparse event augmentation model. The extremely sparse event enhancement model is used to enhance the fused image to be processed, and the output increment is obtained; wherein, the fused image to be processed is obtained by multi-scale fusion of the RGB image to be processed and the event stream to be processed; Based on the output increment, the RGB image enhancement event map to be processed is determined.
[0007] According to a second aspect, the present invention provides a multi-scale extremely sparse event enhancement device based on RGB images, comprising: The first data processing unit is configured to determine the ROI mask of extremely sparse events based on historical RGB images and historical event streams of extremely sparse events. The second data processing unit is configured to train a preset deep learning model using the dataset of the ROI sparse region mask to obtain a trained sparse event augmentation model. The third data processing unit is configured to enhance the fused image to be processed using the extremely sparse event enhancement model to obtain an output increment; wherein the fused image to be processed is obtained by multi-scale fusion of the RGB image to be processed and the event stream to be processed. The fourth data processing unit is configured to determine the RGB image enhancement event map to be processed based on the output increment.
[0008] Thirdly, embodiments of this specification also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method described in any embodiment of this specification.
[0009] Fourthly, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods described in any embodiment of this specification.
[0010] The multi-scale extremely sparse event enhancement method and apparatus based on RGB images provided by this invention first acquires historical RGB images and historical event streams of extremely sparse events. A sparse ROI prior is constructed based on event density statistics, and random occlusion is implemented within the ROI during the training phase to enhance the utilization of the RGB prior. An early-fusion multi-scale U-shaped network is used to predict event increments, and structural constraint injection is achieved through multi-scale gating modulation of the RGB edge prior. An extremely sparse event enhancement model is trained by combining ROI weighted reconstruction, new event region supervision, incremental sparsity regularization, and background suppression joint loss. The current RGB image is input into the extremely sparse event enhancement model to obtain the output increment. Finally, the current RGB image and the output increment are fused to obtain the enhanced event map. Thus, this invention can achieve missing structure completion and suppress background pseudo-event leakage under extremely sparse event observation conditions, thereby obtaining a more complete and reliable event representation. Extremely sparse events refer to observation phenomena where only a very small number of pixels are triggered within a short time window, or even where no events occur in local areas, under scenarios such as long-distance observation, weak texture regions, low angular velocity motion, or near-static illumination changes. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 A flowchart illustrating a multi-scale extremely sparse event enhancement method based on RGB images according to one embodiment is shown. Figure 2 A schematic block diagram of a multi-scale extremely sparse event enhancement device based on an RGB image according to one embodiment is shown.
[0013] Figure 3 A schematic diagram comparing short-window event input, enhanced output, and long-window event graphs is shown in one embodiment.
[0014] Figure 4 A schematic diagram comparing an RGB frame, the original short-window event graph, the baseline method output, the output of the present invention, and the long-window event graph of an embodiment is shown.
[0015] Figure 5 A schematic diagram of the structure of an embodiment of an extremely sparse event enhancement model is shown. Detailed Implementation
[0016] The solution provided by the present invention will now be described with reference to the accompanying drawings.
[0017] Figure 1 This diagram illustrates a flowchart of a multi-scale extremely sparse event enhancement method based on RGB images according to one embodiment. It is understood that this method can be executed by any device, apparatus, platform, or cluster of devices with computing and processing capabilities. Figure 1 As shown, the method includes: Step 100: Based on the historical RGB images and historical event streams of extremely sparse events, determine the ROI mask of extremely sparse events; Step 102: Use the dataset of ROI extremely sparse region masks to train the preset deep learning model to obtain the trained extremely sparse event augmentation model. Step 104: Use the extremely sparse event enhancement model to enhance the fused image to be processed and obtain the output increment; wherein, the fused image to be processed is obtained by multi-scale fusion of the RGB image to be processed and the event stream to be processed. Step 106: Determine the RGB image enhancement event graph to be processed based on the output increment.
[0018] In this embodiment, historical RGB images and historical event streams of extremely sparse events are first acquired. A sparse ROI prior is constructed based on event density statistics. During the training phase, random occlusion is implemented within the ROI to enhance the utilization of the RGB prior. An early-fusion multi-scale U-shaped network is used to predict event increments, and structural constraint injection is achieved through multi-scale gating modulation of the RGB edge prior. An extremely sparse event enhancement model is trained by combining ROI weighted reconstruction, new event region supervision, incremental sparsity regularization, and background suppression joint loss. The current RGB image is input into the extremely sparse event enhancement model to obtain the output increment. Finally, the current RGB image and the output increment are fused to obtain the enhanced event map. Thus, this invention can achieve missing structure completion and suppress background pseudo-event leakage under extremely sparse event observation conditions, thereby obtaining a more complete and reliable event representation. Extremely sparse events refer to observation phenomena where only a very small number of pixels are triggered within a short time window, or even where no events occur in local areas, under scenarios such as long-distance observation, weak texture regions, low angular velocity motion, or near-static illumination changes.
[0019] like Figure 3 As shown in the figure, in this embodiment, Output(Ours) is the RGB image enhancement event graph using the method of the present invention, Input(E) is the fused image to be processed, and Long-window(Ed) is the image to be processed. This is a diagram of the long window event, by Figure 3 It can be seen that compared with the input fused image, the output result fully restores the clear outline and details of key targets in the scene such as houses, roads, and trees, solving the core pain points of insufficient event information and missing structure in short window events. At the same time, compared with long window event maps, the event map output by this method has a cleaner background and more prominent targets while maintaining the same structural integrity, effectively avoiding the problems of noise redundancy and motion blur caused by excessively long time windows in long window events.
[0020] The specific structure of the extremely sparse event enhancement model of the present invention is described below.
[0021] Please see Figure 5As a preferred implementation, the sparse event augmentation model includes an input layer, an edge prior module (Encoder), a backbone network (U-Net), an output head (Decoder), an output module (Head), and a training-only module. The input layer receives two types of data: 3×H×W RGB image frames and 1×H×W sparse event maps. Channel concatenation generates a 4×H×W fused feature X, laying the foundation for subsequent feature extraction. Simultaneously, the RGB frames are independently fed into the edge prior module, undergoing grayscale conversion, Sobel edge detection, and edge normalization to generate multi-scale edge-guided features. These features are then injected into the encoder, bottleneck layer, and decoder of the U-Net, respectively. Element-wise multiplication enhances structure awareness and addresses the issue of blurred details in event reconstruction. The backbone network adopts an improved U-Net structure: the encoder (Enc1~Enc3) downsamples the input feature X and extracts multi-scale semantic features layer by layer; the bottleneck layer completes the feature fusion of the highest level through two convolutional blocks; the decoder (Dec3~Dec1) gradually restores the spatial resolution through upsampling, and at the same time fuses the features of the corresponding level of the encoder through a skip connection structure to achieve complementarity between details and semantics. The decoded features are sent to the output head, which is activated by a 1×1 convolution and a sigmoid function to generate a 1×H×W output increment. ; After being clipped in the [0,1] interval, the event is added element-wise to the input sparse events, and finally outputs a 1×H×W RGB image enhancement event map.
[0022] During the training phase, an additional dedicated supervision module is enabled: the input data is processed by the ROI mask estimator to generate region of interest masks, and after the robustness is enhanced by ROI Dropout, it is combined with long window dense event ground truth (Dense Target) and the network parameters are optimized by backpropagation through structured loss, which specifically solves the sparsity problem of event data and improves the density effect and structural consistency.
[0023] In one embodiment of the present invention, determining the sparse region mask of the ROI of an extremely sparse event based on historical RGB images and historical event streams includes: Based on historical RGB images and historical event streams, determine historical short-window event graphs that are time-aligned with the historical RGB images; The historical short-window event graph is divided into grids and counted within each grid to obtain a density map; Thresholding the density map yields the sparse region mask of the ROI for extremely sparse events.
[0024] In this embodiment, using the historical RGB images and historical event streams corresponding to extremely sparse events as input, the event stream within the corresponding short time window is first extracted based on the timestamp of the historical RGB images and projected onto a 2D plane of the same size as the RGB images to generate a time-aligned historical short-window event map, solving the time matching problem between asynchronous event streams and synchronous frame data. Then, the event map is divided into uniform grids, and the number of events in each grid is counted, transforming discrete event locations into a quantifiable event density map, thereby highlighting the spatial distribution difference between effective event regions and background noise. Finally, the density map is thresholded and binary processed, retaining only regions with event densities higher than the threshold as effective ROIs, ultimately obtaining the sparse region mask of the ROI for extremely sparse events. This scheme does not require complex model training and, through a lightweight process of multimodal alignment, spatial statistics, and threshold segmentation, accurately locates the effective regions of extremely sparse events, effectively filtering background noise interference, providing reliable region constraints for subsequent visual tasks related to extremely sparse events, and can be flexibly adjusted by the threshold to adapt to different scenario requirements.
[0025] In one embodiment of the present invention, the historical short-window event graph is determined by the following formula:
[0026] In the formula, This is a graph of historical short window events. Count the number of pixel events in the historical event stream. This is the trimming constant.
[0027] In one embodiment of the present invention, the sparse region mask of the ROI is determined by the following formula:
[0028] In the formula, For the first The normalized density values of each grid cell. For the first The number of events within each grid, The side length of the grid cell. The normalized saturation coefficient, For density plots, For upsampling, This is the sparse region mask for the ROI. For indicator functions, This is the sparsity threshold.
[0029] In this embodiment, the historical short-window event graph is used. Divide into b×b grids, for the first... Count of events per grid The grid density and pixel-level density map are obtained by normalization. ; through threshold right Thresholding is performed to obtain a sparse region mask. And use it as a sparse region mask for the ROI. .
[0030] In one embodiment of the present invention, when depth, parallax or constructible long-distance prior exists, the ROI sparse region mask is modified to obtain the modified ROI sparse region mask. The corrected ROI sparse region mask is determined by the following formula:
[0031] In the formula, This is the corrected sparse region mask for the ROI. For sparse regions of the ROI, For long-distance binary masking, For Hadama accumulation, For long-distance threshold, For depth disparity maps, This is an indicator function.
[0032] In this embodiment, firstly based on the depth disparity map and long distance threshold via indicator function Generate a binary mask for a long-distance region Only retain distant regions with depth and parallax greater than the threshold, and set the rest to 0); then mask the sparse regions of the ROI. and Perform the Hadamard product (element-wise multiplication) operation to obtain the corrected ROI sparse region mask. This formula addresses the pain points of traditional ROI sparse region masks, which are based solely on 2D semantic generation, are susceptible to near-range interference, and lack accuracy. It introduces long-range 3D geometric priors, carried by depth and parallax, into the mask correction process. By designing indicator functions and thresholds, it generates long-range binary masks and then uses lightweight Hadamard product operations to achieve accurate fusion of 2D ROI semantic constraints and 3D spatial distance constraints. This eliminates the need for complex model training and high computational costs, fundamentally eliminating near-range invalid regions within the ROI and significantly improving mask accuracy. Furthermore, it adapts to various scenario requirements through adjustable thresholds, pioneering a new technical paradigm of "3D geometric prior correcting 2D semantic masks." This fills the technical gap in accurate ROI mask correction in sparse vision tasks and possesses both high technical innovation and engineering practicality.
[0033] In one embodiment of the present invention, the total loss of the deep learning model during the training phase is determined by the following formula:
[0034] In the formula, For the total loss, For the overall reconstruction loss, Reconstruction loss weighted by ROI For the newly added area of supervision losses, For incremental sparse regularization, For background suppression, The weighting coefficients for the reconstruction loss of ROI are used. The new regional supervision loss weighting coefficient, The weight coefficients for the incremental sparse regularization term. The weight coefficients for the background suppression term. Enhance the event graph for RGB images. This is a diagram of the long window incident. It is the minimum value. For indicator functions, To output the increment, For incremental pseudo-true values, To add a mask, For dynamic region threshold, This serves as a reference baseline for the short-window event graph.
[0035] In this embodiment, addressing the core issues of event cameras in extremely sparse scenes, such as missing events, structural breaks, and background pseudo-responses, this loss function constructs a multi-dimensional closed-loop constraint system consisting of global constraints, region reinforcement, incremental supervision, sparsity regularization, and background suppression. Firstly, it uses global reconstruction loss... To ensure global consistency between augmented events and original sparse events and avoid destroying existing valid information; secondly, ROI-weighted reconstruction loss is introduced. This approach focuses constraint weights on the target / missing region, addressing the issue of insufficient constraints on sparse regions by the global loss; furthermore, it incorporates a new region-monitored loss. An incremental mask is constructed to precisely supervise only the incremental parts that need to be completed, achieving structural completion that "only completes what needs to be completed," while avoiding over-modification of the original events; further, incremental sparse regularization terms are used... The sparsity of the generated increments is constrained, unfounded redundant completions are suppressed, and the physical sparsity of events is maintained; finally, a background suppression term is used. By penalizing the original, event-free background regions, the problem of pseudo-event leakage in the background is addressed at its root. The total loss is calculated using learnable hyperparameters. By weighting and balancing each module, the system can accurately complete the missing structure of extremely sparse events while effectively suppressing background pseudo-enhancement. It takes into account the structural integrity, physical rationality and background purity of the enhancement results, and perfectly adapts to the requirements of RGB-guided extremely sparse event enhancement tasks.
[0036] In one embodiment of the present invention, the RGB image enhancement event map is determined by the following formula:
[0037] In the formula, Enhance the event graph for RGB images. The current short window event graph The image to be fused. This is the current RGB image.
[0038] In this embodiment, the current short window event graph is... The image is stitched together with the current RGB image in the channel dimension to form the fused image to be processed. Input multiscale Shape-encoder-decoder network to obtain output increment An RGB image enhancement event map is obtained through residual synthesis.
[0039] like Figure 4 As shown, in this embodiment, using an extremely sparse short-window event map as input, compared with the baseline method (c), its enhanced output (d) achieves a significant improvement in structural consistency, fully restoring key scene structures such as road direction, guardrail outline, and tree shape that are highly matched with the RGB reference frame (a), and accurately completing the missing details of the short-window events; at the same time, it greatly optimizes the purity, effectively suppressing a large number of block artifacts and background noise in the baseline method, resulting in a cleaner output and clearer targets.
[0040] The foregoing has described specific embodiments of the invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0041] According to another embodiment, the present invention provides a multi-scale extremely sparse event enhancement device based on RGB images. Figure 2 A schematic block diagram of a multi-scale extremely sparse event enhancement apparatus based on an RGB image is shown according to one embodiment. It will be understood that this apparatus can be implemented by any device, apparatus, platform, or cluster of devices with computing and processing capabilities. Figure 2 As shown, the device includes: a first data processing unit 200, a second data processing unit 202, a third data processing unit 204, and a fourth processing unit 206. The main functions of each component are as follows: The first data processing unit 200 is configured to determine the ROI mask of extremely sparse events based on historical RGB images and historical event streams of extremely sparse events. The second data processing unit 202 is configured to train a preset deep learning model using the dataset of the ROI sparse region mask to obtain a trained sparse event augmentation model. The third data processing unit 204 is configured to enhance the fused image to be processed using the extremely sparse event enhancement model to obtain an output increment; wherein the fused image to be processed is obtained by multi-scale fusion of the RGB image to be processed and the event stream to be processed. The fourth data processing unit 206 is configured to determine the RGB image enhancement event map to be processed based on the output increment.
[0042] In one embodiment of the present invention, determining the sparse region mask of the ROI based on historical RGB images and historical event streams of extremely sparse events includes: Based on the historical RGB image and the historical event stream, a historical short-window event map aligned with the time of the historical RGB image is determined; The historical short-window event graph is divided into grids and counted within each grid to obtain a density map; Thresholding the density map yields the sparse region mask of the ROI for extremely sparse events.
[0043] In one embodiment of the present invention, the historical short-window event graph is determined by the following formula:
[0044] In the formula, This is the historical short-window event graph. Count the number of pixel events in the historical event stream. This is the trimming constant.
[0045] In one embodiment of the present invention, the ROI sparse region mask is determined by the following formula:
[0046] In the formula, For the first The normalized density values of each grid cell. For the first The number of events within each grid, The side length of the grid cell. The normalized saturation coefficient, For the density map, For upsampling, This is the sparse region mask for the ROI. For indicator functions, This is the sparsity threshold.
[0047] In one embodiment of the present invention, when there is depth, parallax or constructible long-distance prior, the ROI sparse region mask is modified to obtain the modified ROI sparse region mask. The corrected ROI sparse region mask is determined by the following formula:
[0048] In the formula, This is the modified ROI sparse region mask. This is the sparse region mask for the ROI. For long-distance binary masking, For Hadama accumulation, For long-distance threshold, For depth disparity maps, This is an indicator function.
[0049] In one embodiment of the present invention, the total loss of the deep learning model during the training phase is determined by the following formula:
[0050] In the formula, For the total loss, For the overall reconstruction loss, Reconstruction loss weighted by ROI For the newly added area of supervision losses, For incremental sparse regularization, For background suppression, The weighting coefficients for the reconstruction loss of ROI are used. The new regional supervision loss weighting coefficient, The weight coefficients for the incremental sparse regularization term. The weight coefficients for the background suppression term. Enhance the event graph for RGB images. This is a diagram of the long window incident. It is the minimum value. For indicator functions, To output the increment, For incremental pseudo-true values, To add a mask, For dynamic region threshold, This serves as a reference baseline for the short-window event graph.
[0051] In one embodiment of the present invention, the RGB image enhancement event map is determined by the following formula:
[0052] In the formula ， This is an event graph for enhancing the RGB image. This is the current short window event graph. The image to be fused. This is the current RGB image.
[0053] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 1 The method described.
[0054] According to another embodiment, an electronic device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 1 The method described.
[0055] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0056] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.
[0057] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.
Claims
1. A multi-scale extremely sparse event enhancement method based on RGB images, characterized in that, include: Based on historical RGB images and historical event streams of extremely sparse events, determine the ROI mask of extremely sparse events; The pre-defined deep learning model is trained using the dataset of the ROI extremely sparse region mask to obtain the trained extremely sparse event augmentation model. The extremely sparse event enhancement model is used to enhance the fused image to be processed, and the output increment is obtained; wherein, the fused image to be processed is obtained by multi-scale fusion of the RGB image to be processed and the event stream to be processed; Based on the output increment, the RGB image enhancement event map to be processed is determined.
2. The method according to claim 1, characterized in that, The determination of the sparse region mask of the ROI based on historical RGB images and historical event streams of extremely sparse events includes: Based on the historical RGB image and the historical event stream, a historical short-window event map aligned with the time of the historical RGB image is determined; The historical short-window event graph is divided into grids and counted within each grid to obtain a density map; Thresholding the density map yields the sparse region mask of the ROI for extremely sparse events.
3. The method according to claim 2, characterized in that, The historical short-window event graph is determined by the following formula: In the formula, This is the historical short-window event graph. Count the number of pixel events in the historical event stream. This is the trimming constant.
4. The method according to claim 2, characterized in that, The sparse region mask of the ROI is determined by the following formula: In the formula, For the first The normalized density values of each grid cell. For the first The number of events within each grid, The side length of the grid cell. The normalized saturation coefficient, For the density map, For upsampling, This is the sparse region mask for the ROI. For indicator functions, This is the sparsity threshold.
5. The method according to claim 4, characterized in that, When depth, parallax, or long-distance priors exist, the ROI sparse region mask is modified to obtain the modified ROI sparse region mask. The corrected ROI sparse region mask is determined by the following formula: In the formula, This is the modified ROI sparse region mask. This is the sparse region mask for the ROI. For long-distance binary masking, For Hadama accumulation, For long-distance threshold, For depth disparity maps, This is an indicator function.
6. The method according to claim 1, characterized in that, The total loss of the deep learning model during the training phase is determined by the following formula: In the formula, For the total loss, For the overall reconstruction loss, Reconstruction loss weighted by ROI For the newly added area of supervision losses, For incremental sparse regularization, For background suppression, The weighting coefficients for the reconstruction loss of ROI are used. The new regional supervision loss weighting coefficient, The weight coefficients for the incremental sparse regularization term. The weight coefficients for the background suppression term. Enhance the event graph for RGB images. This is a diagram of the long window incident. It is the minimum value. For indicator functions, To output the increment, For incremental pseudo-true values, To add a mask, For dynamic region threshold, This serves as a reference baseline for the short-window event graph.
7. The method according to claim 1, characterized in that, The RGB image enhancement event graph is determined by the following formula: In the formula ， This is an event graph for enhancing the RGB image. This is the current short window event graph. The image to be fused. This is the current RGB image.
8. A multi-scale extremely sparse event enhancement device based on RGB images, characterized in that, include: The first data processing unit is configured to determine the ROI mask of extremely sparse events based on historical RGB images and historical event streams of extremely sparse events. The second data processing unit is configured to train a preset deep learning model using the dataset of the ROI sparse region mask to obtain a trained sparse event augmentation model. The third data processing unit is configured to enhance the fused image to be processed using the extremely sparse event enhancement model to obtain an output increment; wherein the fused image to be processed is obtained by multi-scale fusion of the RGB image to be processed and the event stream to be processed. The fourth data processing unit is configured to determine the RGB image enhancement event map to be processed based on the output increment.
9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed in a computer, causes the computer to perform the method described in any one of claims 1-7.