Dark light scene data enhancement method and system

Abstract

The invention discloses a dark light scene data enhancement method and system, and relates to the field of intelligent parking management, and the method comprises the steps: constructing paired frame-based image and event data through employing the advantages of an event camera in a dark light scene, covering a normal light scene and a dark light scene, carrying out the feature learning and fusion of two types of modal data, and carrying out the feature learning and fusion of the two types of modal data; in view of different advantages of two kinds of modal data, when feature fusion is carried out, event and image features are integrally fused firstly, then image features are extracted from a region with a high signal-to-noise ratio and event features are extracted from a region with a low signal-to-noise ratio under the guidance of the signal-to-noise ratio, more texture and structure information of the regions can be reserved, and the accuracy of feature fusion is improved. And finally, the overall features and the regional features of the image and the event are fused to obtain enhanced image data, so that recovery of the low-illumination image is realized, and the efficiency and the accuracy of vehicle management are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent parking management, and in particular to a dark-light scene data enhancement method and system. Background Art

[0002] In recent years, with the development of urban areas, the population has continued to increase, the number of cars has continued to increase, the traffic flow has continued to increase, and the complexity of traffic has continued to increase. At the same time, with the development of software and hardware technology and algorithms, by installing camera-based monitoring equipment at various traffic locations such as roadsides, intersections, and highways, the collection of traffic data from video images can provide data information support for the analysis of traffic scenes, which plays an important role in improving the accuracy of traffic scene analysis. By installing high-position video cameras on the roadside, the parking of vehicles can be judged and managed. By detecting the vehicle and the body posture, and then analyzing the data with the known parking position, the status of the vehicle can be judged, such as whether the vehicle is in the parking space, whether the vehicle is parked on the line, whether the vehicle is parked in the prohibited parking area, etc. In addition, the vehicle can be judged whether it is illegally parked according to the body posture of the vehicle, which has a positive role in promoting various aspects of urban traffic management and driving safety.

[0003] However, in low-light scenes such as at night, the image quality captured by the camera is low, the visibility of vehicle targets is low, and there is too much noise information in the image, resulting in low accuracy in tasks such as vehicle position judgment, vehicle re-identification, and license plate detection and recognition in low-light scenes. For the enhancement of such low-light images, currently, image enhancement is mainly achieved based on the RGB images generated by the camera by utilizing algorithms such as generative adversarial networks or diffusion models. However, in many cases with poor lighting conditions, low-light images cannot be restored, and some noise information is easily introduced, which affects the judgment of various visual algorithms, thereby resulting in low efficiency and accuracy of vehicle management. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a dark-light scene data enhancement method and system, which can solve the problem that the existing low-light image cannot be restored, thereby resulting in low efficiency and accuracy of vehicle management.

[0005] To achieve the above object, the present invention provides a dark light scene data enhancement method, the method comprising:

[0006] Extract the overall feature information of the image and event of each video frame;

[0007] Extracting regional image feature information in an image region of each video frame where the signal-to-noise ratio is greater than a preset threshold;

[0008] Extracting regional event feature information in the event region of each video frame where the signal-to-noise ratio is less than a preset threshold;

[0009] The overall feature information of the image and event of each video frame, the regional image feature information, and the regional event feature information are fused.

[0010] Furthermore, before the step of extracting the overall feature information of the image and event of each video frame, the method further includes:

[0011] According to formula I init =I⊙L and L=F(I,I prior ) enhances the image and event data of each video frame, where I init represents the image after initial enhancement, the low-light image is defined as I, L represents the brightness map, F represents the brightness estimation algorithm, and I prior represents the image illumination prior map, I prior =max(I).

[0012] Furthermore, the step of extracting regional image feature information in the image region of each video frame whose signal-to-noise ratio is greater than a preset threshold comprises:

[0013] According to the formula Get the signal-to-noise ratio of the video frame, where M snr represents the signal-to-noise ratio, I gray Indicates that the initial enhancement image I init The converted grayscale image, I gray Grayscale image to be denoised;

[0014] The image feature F of each video frame img Input into the preset residual convolution model to extract local area image features

[0015] According to the formula Get regional image feature information, where: Based on M snr The signal-to-noise ratio after normalization, F img-sele is the image feature after region enhancement.

[0016] Furthermore, the step of extracting regional event feature information in the event region of each video frame whose signal-to-noise ratio is less than a preset threshold comprises:

[0017] According to the formula Get regional event feature information, where: is the regional event characteristic, Based on M snrThe signal-to-noise ratio after normalization is

[0018] F event-sele It is the event feature after regional enhancement.

[0019] Furthermore, the step of fusing the overall feature information, regional image feature information, and regional event feature information of the image and event of each video frame comprises:

[0020] According to formula F whole-init =Concat(F img-sele ,F event-sele ), F whole-mid =Attention(F whole-init )+F whole-init 、F whole =FFN(LN(F whole-mid ))+F whole-mid 、F cat =Concat(F img-sele ,F event-sele ,F whole ), F fusion =F3(σ(F1(Fcat))⊙F2(Fcat)+Fcat) for data fusion, where Fwhole-init is the initial overall feature, Attention represents the self-attention structure in the channel dimension, LN represents the layer attention function, FFN represents the preset feedforward neural network, and F whole-mid represents the overall feature in the middle, F whole represents the final overall feature, F1, F2, and F3 represent different convolution modules, σ represents the sigmoid function, ⊙ represents the pixel-by-pixel multiplication operation, and F fusion Represents the obtained fused features.

[0021] Furthermore, the present invention provides a dark light scene data enhancement system, the system comprising:

[0022] An extraction module is used to extract the overall feature information of the image and event of each video frame;

[0023] The extraction module is further used to extract regional image feature information in the image region of each video frame whose signal-to-noise ratio is greater than a preset threshold;

[0024] The extraction module is further used to extract regional event feature information in the event region of each video frame whose signal-to-noise ratio is less than a preset threshold;

[0025] The fusion module is used to fuse the overall feature information of the image and event of each video frame, the regional image feature information, and the regional event feature information.

[0026] Furthermore, the system further comprises: an enhancement module;

[0027] The enhancement module is used to init =I⊙L and L=F(I,I prior ) enhances the image and event data of each video frame, where I init represents the image after initial enhancement, the low-light image is defined as I, L represents the brightness map, F represents the brightness estimation algorithm, and I prior represents the image illumination prior map, I prior =max(I).

[0028] Furthermore, the extraction module is further configured to extract Get the signal-to-noise ratio of the video frame, where M snr represents the signal-to-noise ratio, I gray Indicates that the initial enhancement image I init The converted grayscale image, I gray The grayscale image to be denoised; the image feature F of each video frame img Input into the preset residual convolution model to extract local area image features According to the formula Get regional image feature information, where: Based on M snr The signal-to-noise ratio after normalization, F img-sele is the image feature after region enhancement.

[0029] Furthermore, the extraction module is further configured to extract Get regional event feature information, where: is the regional event characteristic, Based on M snr The signal-to-noise ratio after normalization, F event-sele It is the event feature after regional enhancement.

[0030] Furthermore, the fusion module is specifically used to calculate the whole-init =Concat(F img-sele ,F event-sele ) 、F whole-mid =Attention(F whole-init )+F whole-init 、F whole =FFN(LN(F whole-mid ))+F whole-mid 、F cat =Concat(Fimg-sele ,F event-sele ,F whole ), F fusion =F3(σ(F1(F cat ))⊙F2(F cat )+F cat ) to perform data fusion, where F whole-init is the initial overall feature, Attention represents the self-attention structure in the channel dimension, LN represents the layer attention function, FFN represents the preset feedforward neural network, and F whole-mid represents the overall feature in the middle, F whole represents the final overall feature, F1, F2, and F3 represent different convolution modules, σ represents the sigmoid function, ⊙ represents the pixel-by-pixel multiplication operation, and F fusion Represents the obtained fused features.

[0031] The present invention provides a dark-light scene data enhancement method and system, which utilizes the advantages of event cameras in dark-light scenes to construct paired frame-based images and event data, covering normal light scenes and dark-light scenes, and perform feature learning and fusion of the two modal data. In view of the different advantages of the two modal data, when performing feature fusion, the event and image features are first fused as a whole, and then, under the guidance of the signal-to-noise ratio, image features are extracted from areas with high signal-to-noise ratio, and event features are extracted from areas with low signal-to-noise ratio, which can retain more texture and structural information of the area. Finally, the overall features and regional features of the image and event are fused to obtain enhanced image data, thereby realizing the restoration of low-light images, thereby improving the efficiency and accuracy of vehicle management. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of a dark light scene data enhancement method provided by the present invention;

[0033] Figure 2 It is a schematic diagram of a dark light scene data enhancement system provided by the present invention. DETAILED DESCRIPTION

[0034] The device structure and implementation of the present invention are further described in detail below through the accompanying drawings and embodiments.

[0035] The present invention provides a dark light scene data enhancement method, such as Figure 1 As shown, the specific steps include:

[0036] 101. Extract the overall feature information of the image and event of each video frame.

[0037] For example, the overall feature extraction part of the image and event data is composed of a convolutional neural network, including but not limited to ResNet, AlexNet and other networks, including multiple convolutional layers, normalization layers, non-linear activation layers, etc., to obtain the output features of the image and event.

[0038] For the embodiment of the present invention, before the step of extracting the overall feature information of the image and event of each video frame, the method further includes: for the image, event and data preprocessing part, firstly, data definition is performed, the low-light image is defined as I, and the event sequence paired with it is defined as Indicates that the event sequence includes N events; the model output is defined as I en , represents the enhanced image data; according to formula I init =I⊙L and L=F(I,I prior ) enhances the image and event data of each video frame, where I init represents the image after initial enhancement, the low-light image is defined as I, L represents the brightness map, F represents the brightness estimation algorithm, and I prior represents the image illumination prior map, I prior =max(I).

[0039] For the embodiments of the present invention, paired image-event datasets in low-light and normal-light scenarios can also be pre-constructed; specifically, the dataset can be acquired in two ways, one is the collection of real data, and the other is by utilizing open source data; for the method of collecting real data, by utilizing an event camera and an ordinary camera, the camera is fixed on a stable tripod to ensure that the captured images are aligned at the pixel level, and then roadside parking scenes such as vehicle parking are simulated, firstly, images under normal lighting conditions are captured to obtain images and event sequences under normal lighting, and then, by adjusting the exposure time of the camera, reducing the exposure time, and capturing short-exposure images, images under low-light conditions are simulated to obtain images and event sequences under low light, thereby obtaining pairs of images and event sequences under different lighting scenarios; for open source data, including but not limited to event camera datasets such as DVS-Dark and EvLowLight.

[0040] 102. Extract regional image feature information in an image region of each video frame whose signal-to-noise ratio is greater than a preset threshold.

[0041] It should be noted that regional features are selectively extracted from images or events based on the signal-to-noise ratio; this part includes two modules, namely, the image region feature selection module and the event region feature selection module; wherein, the purpose of the image region feature selection module is to select image features with a high signal-to-noise ratio to avoid the influence of noise in the image, and the purpose of the event region feature selection module is to enhance the features of the edge areas in low-light areas and high-noise areas.

[0042] Specifically, according to the formula Get the signal-to-noise ratio of the video frame, where M snr represents the signal-to-noise ratio, I gray Indicates that the initial enhancement image I init The converted grayscale image, I gray The grayscale image to be denoised; the image feature F of each video frame img Input into the preset residual convolution model to extract local area image features According to the formula Get regional image feature information, where: Based on M snr The signal-to-noise ratio after normalization, F img-sele is the image feature after region enhancement.

[0043] For example, for the image region feature selection module, the input is the extracted image feature F img , and the signal-to-noise ratio map M snr ; First, the image feature F img Input residual convolution module to extract local area features and obtain output image features The residual convolution module includes multiple convolution layers, normalization layers, etc. Then, the signal-to-noise ratio map M snr Expand in the channel dimension, which is the same as the image feature dimension, normalize the values ​​in the signal-to-noise ratio map to [0,1], and define the signal-to-noise ratio threshold T snr , set the value less than the threshold to 0, T snr Can be set to 0.3; get the processed signal-to-noise ratio graph Finally, the pixel-by-pixel multiplication method is used to obtain the image features after region enhancement, which is expressed as:

[0044] 103. Extract regional event feature information in the event region of each video frame whose signal-to-noise ratio is less than a preset threshold.

[0045] Specifically, according to the formula Get regional event feature information, where: is the regional event characteristic, Based on M snr The signal-to-noise ratio after normalization, F event-sele It is the event feature after regional enhancement.

[0046] 104. Perform data fusion on the overall feature information of the image and event of each video frame, the regional image feature information, and the regional event feature information.

[0047] Specifically, according to formula F whole-init =Concat(F img-sele ,F event-sele ), F whole-mid =Attention(F whole-init )+F whole-init 、F whole =FFN(LN(F whole-mid ))+F whole-mid 、F cat =Concat(F img-sele ,F event-sele ,F whole ), F fusion =F3(σ(F1(F cat ))⊙F2(F cat )+F cat ) to perform data fusion, where F whole-init is the initial overall feature, Attention represents the self-attention structure in the channel dimension, LN represents the layer attention function, FFN represents the preset feedforward neural network, and F whole-mid represents the overall feature in the middle, F whole represents the final overall feature, F1, F2, and F3 represent different convolution modules, σ represents the sigmoid function, ⊙ represents the pixel-by-pixel multiplication operation, and F fusion Represents the obtained fused features.

[0048] For example, for the overall feature extraction part, the image region features and event region features are first concatenated in the channel dimension to obtain the initial overall feature, which is expressed as: F whole-init =Concat(F img-sele ,F event-sele ); then the multi-head attention module and feedforward neural network are used to further extract the initial overall features, which is expressed as:

[0049] F whole-mid =Attention(F whole-init )+F whole-init ,

[0050] F whole =FFN(LN(F whole-mid ))+Fwhole-mid , where Attention represents the self-attention structure in the channel dimension, LN represents the layer attention function, and FFN represents the feedforward neural network; F whole Represents the overall feature obtained; for the regional and overall feature fusion part, first, the image region features, event region features, and overall features are spliced ​​in the channel dimension, expressed as: F cat =Concat(F img-sele ,F event-sele ,F whole ), and then use multiple convolutional layers to further extract the concatenated features, expressed as: F fusion =F3(σ(F1(F cat ))⊙F2(F cat )+F cat ), where F1, F2, and F3 represent convolution modules, σ represents the sigmoid function, and ⊙ represents the pixel-by-pixel multiplication operation; fusion Represents the obtained fused features, which are used to predict the output of the final enhanced image.

[0051] For the embodiment of the present invention, a loss function for model training can also be constructed to perform model optimization and parameter update; specifically, the training loss function of the model consists of the following two parts: L = ε1L l1 +ε2L ssim , where α1 and α2 represent weight parameters; specifically, for L l1 and L ssim , as pixel fidelity loss, can be expressed as: in, Represents the two vectors that need to be calculated, the ratio is the estimated value and the target value of the model, m represents the total amount of data, and the ultimate goal of the loss function is to minimize the absolute difference between the total estimated value and the target value; Among them, SSIM represents the structural similarity of images.

[0052] A dark-light scene data enhancement method provided by an embodiment of the present invention utilizes the advantages of an event camera in a dark-light scene to construct paired frame-based image and event data, covering normal light scenes and dark light scenes, and performs feature learning and fusion of the two modal data. In view of the different advantages of the two modal data, when performing feature fusion, the event and image features are first fused as a whole, and then, under the guidance of the signal-to-noise ratio, image features are extracted from areas with high signal-to-noise ratio, and event features are extracted from areas with low signal-to-noise ratio, so as to retain more texture and structural information of the area. Finally, the overall features and regional features of the image and event are fused to obtain enhanced image data, thereby realizing the restoration of low-light images, thereby improving the efficiency and accuracy of vehicle management.

[0053] As Figure 1 The specific implementation of the method shown in the figure, the embodiment of the present invention provides a dark light scene data enhancement system, such as Figure 2 As shown, the system includes: an extraction module 21, for extracting the overall feature information of the image and event of each video frame;

[0054] The extraction module 21 is further used to extract regional image feature information in the image region of each video frame whose signal-to-noise ratio is greater than a preset threshold;

[0055] The extraction module 21 is further used to extract regional event feature information in the event region of each video frame whose signal-to-noise ratio is less than a preset threshold;

[0056] The fusion module 22 is used to fuse the overall feature information of the image and event of each video frame, the regional image feature information, and the regional event feature information.

[0057] Furthermore, the system further comprises: an enhancement module 23;

[0058] The enhancement module 23 is used to init =I⊙L and L=F(I,I prior ) enhances the image and event data of each video frame, where I init represents the image after initial enhancement, the low-light image is defined as I, L represents the brightness map, F represents the brightness estimation algorithm, and I prior represents the image illumination prior map, I prior =max(I).

[0059] Furthermore, the extraction module 21 is further configured to extract Get the signal-to-noise ratio of the video frame, where M snr represents the signal-to-noise ratio, I gray Indicates that the initial enhancement image I init The converted grayscale image, I gray The grayscale image to be denoised; the image feature F of each video frame img Input into the preset residual convolution model to extract local area image features According to the formula Get regional image feature information, where: Based on M snr The signal-to-noise ratio after normalization, F img-sele is the image feature after region enhancement.

[0060] Furthermore, the extraction module 21 is further configured to extract Get regional event feature information, where: is the regional event characteristic, Based on M snr The signal-to-noise ratio after normalization, F event-sele It is the event feature after regional enhancement.

[0061] Furthermore, the fusion module 22 is specifically configured to: whole-init =Concat(F img-sele ,F event-sele ), F whole-mid =Attention(F whole-init )+F whole-init 、F whole =FFN(LN(F whole-mid ))+F whole-mid 、F cat =Concat(F img-sele ,F event-sele ,F whole ), F fusion =F3(σ(F1(F cat ))⊙F2(F cat )+F cat ) to perform data fusion, where F whole-init is the initial overall feature, Attention represents the self-attention structure in the channel dimension, LN represents the layer attention function, FFN represents the preset feedforward neural network, and F whole-mid represents the overall feature in the middle, F whole represents the final overall feature, F1, F2, and F3 represent different convolution modules, σ represents the sigmoid function, ⊙ represents the pixel-by-pixel multiplication operation, and F fusion Represents the obtained fused features.

[0062] A dark-light scene data enhancement system provided by an embodiment of the present invention utilizes the advantages of event cameras in dark-light scenes to construct paired frame-based images and event data, covering normal light scenes and dark-light scenes, and performs feature learning and fusion of the two modal data. In view of the different advantages of the two modal data, when performing feature fusion, the event and image features are first fused as a whole, and then, under the guidance of the signal-to-noise ratio, image features are extracted from areas with high signal-to-noise ratio, and event features are extracted from areas with low signal-to-noise ratio, which can retain more texture and structural information of the area. Finally, the overall features and regional features of the image and event are fused to obtain enhanced image data, thereby realizing the restoration of low-light images, thereby improving the efficiency and accuracy of vehicle management.

[0063] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0064] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0065] The disclosed embodiments are described above to enable any person skilled in the art to implement or use the present invention. Various modifications of these embodiments are obvious to those skilled in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0066] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

[0067] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.

[0068] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0069] The steps of the method or algorithm described in the embodiments of the present invention can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.

[0070] In one or more exemplary designs, the above functions described in the embodiments of the present invention can be implemented in hardware, software, firmware or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium, or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by any general or special computer. For example, such computer-readable media can include but are not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program codes in the form of instructions or data structures and other forms that can be read by general or special computers, or general or special processors. In addition, any connection can be appropriately defined as a computer-readable medium, for example, if the software is transmitted from a website site, server or other remote resource through a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disk and disc include compact disk, laser disk, optical disk, DVD, floppy disk and blue-ray disk. Disks usually copy data magnetically, while discs usually copy data optically with lasers. The above combination can also be included in computer readable media.

[0071] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A dark light scene data enhancement method, characterized in that: The method comprises: Extract the overall feature information of the image and event of each video frame; Extracting regional image feature information in an image region of each video frame where the signal-to-noise ratio is greater than a preset threshold; Extracting regional event feature information in the event region of each video frame where the signal-to-noise ratio is less than a preset threshold; The overall feature information of the image and event of each video frame, the regional image feature information, and the regional event feature information are fused.

2. The dark light scene data enhancement method according to claim 1, characterized in that: Before the step of extracting the overall feature information of the image and event of each video frame, the method further comprises: According to formula I init =I⊙L and L=F(I,I prior ) enhances the image and event data of each video frame, where I init represents the image after initial enhancement, the low-light image is defined as I, L represents the brightness map, F() represents the brightness estimation algorithm, and I prior represents the image illumination prior map, I prior =max(I).

3. The dark light scene data enhancement method according to claim 1, characterized in that: The step of extracting regional image feature information in the image region of each video frame whose signal-to-noise ratio is greater than a preset threshold comprises: According to the formula Get the signal-to-noise ratio of the video frame, where M snr represents the signal-to-noise ratio, I gray Indicates that the initial enhancement image I init The converted grayscale image, I gray Grayscale image to be denoised; The image feature F of each video frame img Input into the preset residual convolution model to extract local area image features According to the formula Get regional image feature information, where: Based on M snr The signal-to-noise ratio after normalization, F img-sele is the image feature after region enhancement.

4. The dark light scene data enhancement method according to claim 3, characterized in that: The step of extracting regional event feature information in the event region of each video frame whose signal-to-noise ratio is less than a preset threshold comprises: According to the formula Get regional event feature information, where: is the regional event characteristic, Based on M snr The signal-to-noise ratio after normalization, F event-sele It is the event feature after regional enhancement.

5. The dark light scene data enhancement method according to claim 4, characterized in that: The step of fusing the overall feature information, regional image feature information, and regional event feature information of each video frame includes: According to formula F whole-init =Concat(F img-sele ,F event-sele ), F whole-mid =Attention(F whole-init )+F whole-init 、F whole =FFN(LN(F whole-mid ))+F whole-mid 、F cat =Concat(F img-sele ,F event-sele ,F whole ), F fusion =F3(σ(F1(F cat ))⊙F2(F cat )+F cat ) to perform data fusion, where F whole-init is the initial overall feature, Attention represents the self-attention structure in the channel dimension, LN represents the layer attention function, FFN represents the preset feedforward neural network, and F whole-mid represents the overall feature in the middle, F whole represents the final overall feature, F1, F2, and F3 represent different convolution modules, σ represents the sigmoid function, ⊙ represents the pixel-by-pixel multiplication operation, and F fusion Represents the obtained fused features.

6. A dark light scene data enhancement system, characterized in that: The system comprises: An extraction module is used to extract the overall feature information of the image and event of each video frame; The extraction module is further used to extract regional image feature information in the image region of each video frame whose signal-to-noise ratio is greater than a preset threshold; The extraction module is further used to extract regional event feature information in the event region of each video frame whose signal-to-noise ratio is less than a preset threshold; The fusion module is used to fuse the overall feature information of the image and event of each video frame, the regional image feature information, and the regional event feature information.

7. The dark light scene data enhancement system according to claim 6, characterized in that: The system further comprises: an enhancement module; The enhancement module is used to init =I⊙L and L=F(I,I prior ) enhances the image and event data of each video frame, where I init represents the image after initial enhancement, the low-light image is defined as I, L represents the brightness map, F represents the brightness estimation algorithm, and I prior represents the image illumination prior map, I prior =max(I).

8. The dark light scene data enhancement system according to claim 6, characterized in that: The extraction module is further configured to: Get the signal-to-noise ratio of the video frame, where M snr represents the signal-to-noise ratio, I gray Indicates that the initial enhancement image I init The converted grayscale image, I gray The grayscale image to be denoised; the image feature F of each video frame img Input into the preset residual convolution model to extract local area image features According to the formula Get regional image feature information, where: Based on M snr The signal-to-noise ratio after normalization, F img-sele is the image feature after region enhancement.

9. The dark light scene data enhancement system according to claim 8, characterized in that: The extraction module is further configured to: Get regional event feature information, where: is the regional event characteristic, Based on M snr The signal-to-noise ratio after normalization, F event-sele It is the event feature after regional enhancement.

10. The dark light scene data enhancement system according to claim 9, characterized in that: The fusion module is specifically used to calculate the whole-init =Concat(F img-sele ,F event-sele ), F whole-mid =Attention(F whole-init )+F whole-init 、F whole =FFN(LN(F whole-mid ))+F whole-mid 、F cat =Concat(F img-sele ,F event-sele ,F whole ), F fusion =F3(σ(F1(F cat ))⊙F2(F cat )+F cat ) to perform data fusion, where F whole-init is the initial overall feature, Attention represents the self-attention structure in the channel dimension, LN represents the layer attention function, FFN represents the preset feedforward neural network, and F whole-mid represents the overall feature in the middle, F whole represents the final overall feature, F1, F2, and F3 represent different convolution modules, σ represents the sigmoid function, ⊙ represents the pixel-by-pixel multiplication operation, and F fusion Represents the obtained fused features.

Images