Low-light image enhancement method and device based on data fusion

Abstract

The application relates to the computer technical field, in particular to a low-light image enhancement method and device based on data fusion, wherein the method comprises the following steps: acquiring an underexposed image and event stream data from an event camera; intercepting event stream corresponding to the underexposed image within an exposure time of the underexposed image; constructing a pulse convolutional neural network; inputting the event stream into the pulse convolutional neural network for forward propagation to obtain a feature vector of the event stream data; constructing a multi-modal fusion network; and splicing the feature vectors of the underexposed image and the event stream data to use the multi-modal fusion network for forward propagation to generate an exposed enhanced image. According to the application, the underexposed image and the event stream collected by the event camera in a low-light environment are used to respond to the scene and output the event stream to generate a normal exposure image, so that the low-light image enhancement is realized, the quality and effect of the collected image are improved, and the obtained image data is more accurate and practical.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a low-light image enhancement method and apparatus based on data fusion. Background Technology

[0002] With the development of digital imaging, the requirements for image quality have also increased. When a camera is in a low-light environment, the quality of the captured images will be severely degraded, affecting the normal use of the images.

[0003] An event camera is a new type of visual sensor that has advantages over traditional cameras, such as high dynamic range, high temporal resolution, and no dynamic blur. It works asynchronously with pixels and outputs an event stream based on changes in light intensity to acquire image data. However, the algorithms and methods applicable to traditional cameras and traditional image processing cannot be directly applied to event cameras and event data.

[0004] In related technologies, traditional cameras struggle to acquire effective images in environments with limited light intensity, often resulting in underexposure. This leads to a decline in the quality of image data acquired in low-light conditions, hindering effective observation of low-light scenes and reducing the practicality of the obtained image results, which urgently needs to be addressed. Summary of the Invention

[0005] This application provides a low-light image enhancement method and apparatus based on data fusion to solve the problems in related technologies, such as the difficulty of traditional cameras to acquire effective images in environments with limited light intensity, which easily leads to underexposure, resulting in a decrease in the quality of image data acquired in low-light environments, making it impossible to achieve effective observation of low-light scenes, and making the obtained image results less practical.

[0006] The first aspect of this application provides a low-light image enhancement method based on data fusion, comprising the following steps: acquiring an underexposed image and event stream data collected under preset low-light environmental conditions from a moving event camera; extracting the event stream data corresponding to the exposure time of the underexposed image; constructing a spike convolutional neural network using a Spike Response model as the neuron dynamics model; inputting the event stream into the spike convolutional neural network for forward propagation to obtain the feature vector of the event stream data; constructing a multimodal fusion network and concatenating the feature vectors of the underexposed image and the event stream data, so as to use the multimodal fusion network for forward propagation to generate an enhanced image.

[0007] Specifically, in one embodiment of this application, the event stream interception algorithm is as follows:

[0008] ε={(x i ,y i ,ti ,pi)|t start ≤t i ≤t end},

[0009] Where ε is the intercepted event stream, (x i ,y i ) represents the pixel coordinates of event i, and t represents the pixel coordinates of event i. i p is the timestamp of event i. i Let t be the event polarity of event i. start It is the time when the underexposed image I begins to be exposed, t end It is the time when the underexposed image I ends its exposure, i = 1, 2, 3...

[0010] Optionally, in one embodiment of this application, the pulsed convolutional neural network includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. The input convolutional layer has 2 input channels, corresponding to positive and negative polarity events in the event stream. The kernel size of the input convolutional layer is 3*3, the stride is 1, and the number of output channels is 16. The hidden convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 16 output channels. The output convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 32 output channels.

[0011] Optionally, in one embodiment of this application, the multimodal fusion network uses a U-Net structure, wherein the number of input channels in the input layer of the multimodal fusion network is 32+k, and the number of output channels in the output layer is k, where k is the number of channels of the underexposed image.

[0012] Optionally, in one embodiment of this application, the step of stitching the feature vectors of the underexposed image and the event stream data includes: stitching the feature vectors of the underexposed image and the event stream data along the channel dimension.

[0013] A second aspect of this application provides a low-light image enhancement device based on data fusion, comprising: an acquisition module for acquiring an underexposed image and event stream data collected from a moving event camera under preset low-light environmental conditions; a cropping module for cropping the event stream data corresponding to the event stream within the exposure time of the underexposed image; a construction module for constructing a spike convolutional neural network using a Spike Response model as a neuron dynamics model; an input module for inputting the event stream into the spike convolutional neural network for forward propagation to obtain a feature vector of the event stream data; and an enhancement module for constructing a multimodal fusion network and concatenating the feature vectors of the underexposed image and the event stream data to perform forward propagation using the multimodal fusion network to generate an enhanced image with improved exposure.

[0014] Specifically, in one embodiment of this application, the event stream interception algorithm is as follows:

[0015] ε={(x i ,y i ,t i ,p i )|t start ≤t i ≤t end},

[0016] Where ε is the intercepted event stream, (x i ,y i ) represents the pixel coordinates of event i, and t represents the pixel coordinates of event i. i p is the timestamp of event i. i Let t be the event polarity of event i. start It is the time when the underexposed image I begins to be exposed, t end It is the time when the underexposed image I ends its exposure, i = 1, 2, 3...

[0017] Optionally, in one embodiment of this application, the pulsed convolutional neural network includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. The input convolutional layer has 2 input channels, corresponding to positive and negative polarity events in the event stream. The kernel size of the input convolutional layer is 3*3, the stride is 1, and the number of output channels is 16. The hidden convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 16 output channels. The output convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 32 output channels.

[0018] Optionally, in one embodiment of this application, the multimodal fusion network uses a U-Net structure, wherein the number of input channels in the input layer of the multimodal fusion network is 32+k, and the number of output channels in the output layer is k, where k is the number of channels of the underexposed image.

[0019] Optionally, in one embodiment of this application, the enhancement module includes: a stitching unit, used to stitch the feature vectors of the underexposed image and the event stream data along the channel dimension.

[0020] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the low-light image enhancement method based on data fusion as described in the above embodiments.

[0021] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data fusion-based low-light image enhancement method described above.

[0022] This application embodiment can acquire underexposed images and event stream data from an event camera, extract the event stream data corresponding to the exposure time of the underexposed image, construct a pulsed convolutional neural network, input the event stream into the pulsed convolutional neural network for forward propagation to obtain the feature vector of the event stream data, construct a multimodal fusion network, and concatenate the feature vectors of the underexposed image and event stream data. Using the multimodal fusion network for forward propagation, an enhanced exposure image is generated, thereby improving the quality and effect of the acquired image and making the obtained image data more accurate and practical. This solves the problems in related technologies where traditional cameras struggle to acquire effective images in light-limited environments, easily resulting in underexposure, leading to a decrease in the quality of image data acquired in low-light environments, hindering effective observation of low-light scenes, and reducing the practicality of the obtained image results.

[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0025] Figure 1 This is a flowchart of a low-light image enhancement method based on data fusion according to an embodiment of this application;

[0026] Figure 2This is a schematic diagram illustrating the process of a low-light image enhancement method based on data fusion according to an embodiment of this application;

[0027] Figure 3 This is event stream data of an underexposed image acquired according to one embodiment of this application;

[0028] Figure 4 This is a cropping event stream of an underexposed image acquired according to one embodiment of this application;

[0029] Figure 5 This is a schematic diagram of the structure of a low-light image enhancement device based on data fusion according to an embodiment of this application;

[0030] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0031] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0032] The following describes a low-light image enhancement method and apparatus based on data fusion according to embodiments of this application, with reference to the accompanying drawings. Addressing the issue mentioned in the background section that traditional cameras struggle to acquire effective images in low-light environments, often resulting in underexposure, which degrades the quality of image data acquired in low-light conditions, hindering effective observation of low-light scenes and reducing the practicality of the obtained image results, this application provides a low-light image enhancement method based on data fusion. This method acquires underexposure images and event stream data from an event camera, extracts the event stream data corresponding to the exposure time of the underexposure image, constructs a pulsed convolutional neural network, inputs the event stream into the pulsed convolutional neural network for forward propagation, obtains the feature vector of the event stream data, constructs a multimodal fusion network, and concatenates the feature vectors of the underexposure image and event stream data. The multimodal fusion network is then used for forward propagation to generate an enhanced image, thereby improving the quality and effect of the acquired images and making the obtained image data more accurate and practical. This solves the problems in related technologies, such as the difficulty of traditional cameras to acquire effective images in environments with limited light intensity, which easily leads to underexposure, resulting in a decrease in the quality of image data acquired in low-light environments, making it impossible to effectively observe low-light scenes, and reducing the practicality of the obtained image results.

[0033] Specifically, Figure 1 This is a schematic flowchart of a low-light image enhancement method based on data fusion provided in an embodiment of this application.

[0034] like Figure 1 As shown, this low-light image enhancement method based on data fusion includes the following steps:

[0035] In step S101, underexposed images and event stream data that meet preset low-light environment conditions are acquired from the moving event camera.

[0036] It is understood that the preset low light environment conditions in this application embodiment can be camera shooting environments where the light intensity cannot meet the normal exposure conditions of the image. The event stream data can be a summary of the event data output by each pixel. Each event includes the horizontal and vertical coordinates of the pixel, the timestamp, and the event polarity. The event polarity can include two cases: the pixel light intensity decreases and the pixel light intensity increases.

[0037] It should be noted that the preset low-light environment conditions are set by those skilled in the art according to the actual situation, and are not specifically limited here.

[0038] In actual execution, the underexposed image I and event stream data E, acquired under low-light conditions, can be obtained from the moving event camera through raw data acquisition, specifically represented as follows:

[0039]

[0040] Where E represents event stream data, (x i ,y i ) represents the pixel coordinates of event i, and t represents the pixel coordinates of event i. i p is the timestamp of event i. i Let represent the event polarity of event i, and N be the total number of events collected.

[0041] This application embodiment can acquire underexposed images and event stream data collected under preset low-light environment conditions from a moving event camera, thereby obtaining the required output information of the event camera. The obtained scene visual data has the high dynamic range characteristics of the event, so as to further divide and extract the data according to the following steps.

[0042] In step S102, the event stream data corresponding to the exposure time of the underexposed image is extracted.

[0043] It is understood that the exposure time of the underexposed image in the embodiments of this application can be the time information contained in the event stream collected in the low light environment obtained in the above steps, and the event stream corresponding to the exposure time of the underexposed image can be obtained by intercepting the event stream data obtained in the above steps.

[0044] This application embodiment can extract the event stream data corresponding to the exposure time of an underexposed image. By utilizing the high temporal resolution of the event camera, the required exposure time can be obtained, thereby obtaining the corresponding event stream data, making the results more targeted and accurate.

[0045] Specifically, in one embodiment of this application, the event stream interception algorithm is as follows:

[0046] ε={(x i ,y i ,t i ,p i )|t start ≤t i ≤t end},

[0047] Where ε is the intercepted event stream, (x i ,y i ) represents the pixel coordinates of event i, and t represents the pixel coordinates of event i. i p is the timestamp of event i. i Let t be the event polarity of event i. start It is the time when the underexposed image I begins to be exposed, t end It is the time when the underexposed image I ends its exposure, i = 1, 2, 3...

[0048] It is understood that, in the embodiments of this application, the event stream ε corresponding to the exposure time can be obtained by extracting the total event stream data from the start and end times of the underexposed image. The event polarity p can take the value 1 or -1, where -1 indicates that the light intensity of the pixel decreases and 1 indicates that the light intensity of the pixel increases.

[0049] As shown in the above formula, the corresponding algorithm can be used to process the data and obtain the desired event stream. By completing data preprocessing, the temporal resolution accuracy of the event camera can be further explored, making the application of event stream data more comprehensive.

[0050] In step S103, the Spike Response model is used as the neuron dynamics model to construct a spike convolutional neural network.

[0051] It is understood that the pulsed convolutional neural network constructed in the embodiments of this application can be an artificial neural network with the ability to consider the influence of time information. When its membrane potential reaches a certain specific value, it is activated and then generates a signal to be transmitted to other neurons to increase or decrease their membrane potential.

[0052] The embodiments of this application can use the Spike Response model as a neuron dynamics model to construct a spike convolutional neural network, making the simulated neurons closer to reality, thus making them more applicable when processing time-series pulse signals.

[0053] Optionally, in one embodiment of this application, the pulsed convolutional neural network includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. The input convolutional layer has 2 input channels, corresponding to positive and negative events in the event stream. The kernel size of the input convolutional layer is 3*3, the stride is 1, and the number of output channels is 16. The hidden convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 16 output channels. The output convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 32 output channels.

[0054] It is understood that in the embodiments of this application, the number of input channels of the input convolutional layer, the hidden convolutional layer, and the output convolutional layer are 2, 16, and 16, respectively, the kernel size is 3*3, the stride is 1, and the number of output channels is 16, 16, and 32. The number of input channels of the input convolutional layer is 2, which corresponds to the positive and negative polarity events of the event stream, such as the increase and decrease of the light intensity of the pixel.

[0055] In this embodiment, the spiking convolutional neural network includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. By limiting the number of input channels, the kernel size, the stride, and the number of output channels, the practicality of the spiking convolutional neural network is improved, making it easier to apply for processing event stream data in the following steps.

[0056] In step S104, the event stream is input into the pulse convolutional neural network for forward propagation to obtain the feature vector of the event stream data.

[0057] It is understood that, in the embodiments of this application, in the pulse convolutional neural network, the forward propagation process of event stream data can be used to activate neurons and transmit signals to other neurons to perform membrane potential changes, thereby completing the processing of time-series pulse signals.

[0058] In actual execution, the event stream ε calculated in the above steps can be input into the constructed spiking neural network for forward propagation to complete the event stream encoding calculation and obtain the feature vector F of the event stream data.

[0059] In this embodiment, the event stream can be input into a pulse convolutional neural network for forward propagation to obtain the feature vector of the event stream data. The event stream encoding process is completed by performing neuromorphic calculations, which improves the denoising effect of the event stream data and makes the obtained data processing results more reliable.

[0060] In step S105, a multimodal fusion network is constructed, and the feature vectors of the underexposed image and event stream data are stitched together to perform forward propagation using the multimodal fusion network to generate an exposed enhanced image.

[0061] It is understood that the multimodal fusion network in this application embodiment can realize data fusion, concatenate the feature vectors of the underexposed image and event stream data obtained in the above steps, input them into the multimodal fusion network to propagate the data forward, thereby completing the data enhancement processing of the required image.

[0062] The embodiments of this application can construct a multimodal fusion network and stitch together the feature vectors of underexposed images and event stream data to use the multimodal fusion network for forward propagation to generate an enhanced exposed image, thereby improving the quality and effect of the acquired images and making the obtained image data more accurate and practical.

[0063] Optionally, in one embodiment of this application, the multimodal fusion network uses a U-Net structure, the number of input channels in the input layer of the multimodal fusion network is 32+k, and the number of output channels in the output layer is k, where k is the number of channels in the underexposed image.

[0064] It is understood that the number of channels k of the underexposed image in the embodiments of this application can be obtained from the underexposed image of the obtained original data. For example, when the obtained underexposed image frame is a grayscale image, k = 1, and when the obtained underexposed image frame is an RGB (Red, Green, Blue) image, k = 3.

[0065] In this embodiment, the multimodal fusion network uses a U-Net structure. The input layer of the multimodal fusion network has 32+k input channels, and the output layer has k output channels, where k is the number of channels in the underexposed image. By limiting the structure of the multimodal fusion network and the number of input and output channels, the enhancement process for low-light images can be further realized.

[0066] Optionally, in one embodiment of this application, stitching the feature vectors of the underexposed image and the event stream data includes: stitching the feature vectors of the underexposed image and the event stream data along the channel dimension.

[0067] It is understood that the underexposed image in this application embodiment can be obtained from the original data in the above steps, and the feature vector of the event stream data can be obtained by forward propagation in the pulse convolutional neural network in the above steps. The two are then concatenated along the channel dimension and input into the multimodal fusion network to obtain the enhanced image with normal exposure.

[0068] The embodiments of this application can concatenate the feature vectors of underexposed images and event stream data along the channel dimension. By using event streams and underexposed images as input, the effectiveness of multimodal data information is enhanced, so as to achieve more effective observation of low-light scenes.

[0069] like Figure 2The diagram shown is a schematic representation of a low-light image enhancement method based on data fusion according to an embodiment of this application. The following is a detailed explanation... Figure 2 The working content of the embodiments of this application will be described in detail with reference to a specific example.

[0070] First, low-light image data and event stream data are acquired. Raw data is obtained from a moving event camera, capturing underexposed images (I) and event stream data (E) in low-light conditions. The event stream data of the acquired underexposed images is as follows: Figure 3 As shown.

[0071] Secondly, data preprocessing is performed. Based on the exposure time of the obtained underexposed image, the event stream ε corresponding to this exposure time is extracted. Specifically, the start and end exposure times of the underexposed image are defined as t. start =100ms, t end =150ms, then the captured event stream is as follows Figure 4 As shown.

[0072] Next, a spiking neural network was constructed, using the Spike Response model as the neuron dynamics model to construct a spiking convolutional neural network.

[0073] Next, the event stream encoding calculation is performed, and the calculated event stream ε is input into the constructed spiking neural network for forward propagation to obtain the feature vector F of the event stream data.

[0074] Next, a multimodal fusion network is constructed. The input underexposed image is an RGB image, and the number of channels in the underexposed image is k=3. Therefore, the number of input channels in the input layer of the data fusion network is 35, and the number of output channels in the output layer is 3.

[0075] Finally, low-light image enhancement calculations are performed. The resulting underexposed image I and the resulting event stream data feature vector F are concatenated and forward-propagated using a multimodal fusion network to generate a normally exposed enhanced image.

[0076] The low-light image enhancement method based on data fusion proposed in this application can acquire underexposed images and event stream data from an event camera, extract the event stream data corresponding to the exposure time of the underexposed image, construct a pulsed convolutional neural network, input the event stream into the pulsed convolutional neural network for forward propagation to obtain the feature vector of the event stream data, construct a multimodal fusion network, and concatenate the feature vectors of the underexposed image and event stream data. The multimodal fusion network is then used for forward propagation to generate an enhanced image, thereby improving the quality and effect of the acquired image and making the obtained image data more accurate and practical. This solves the problems in related technologies where traditional cameras struggle to acquire effective images in light-limited environments, easily resulting in underexposure, leading to a decrease in the quality of image data acquired in low-light environments, hindering effective observation of low-light scenes, and reducing the practicality of the obtained image results.

[0077] Next, referring to the accompanying drawings, a low-light image enhancement apparatus based on data fusion according to an embodiment of this application is described.

[0078] Figure 6 A block diagram of a low-light image enhancement device based on data fusion according to an embodiment of this application.

[0079] like Figure 6 As shown, the low-light image enhancement device 10 based on data fusion includes: an acquisition module 100, a cropping module 200, a construction module 300, an input module 400, and an enhancement module 500.

[0080] The acquisition module 100 is used to acquire underexposed images and event stream data from a moving event camera that meet preset low-light environmental conditions.

[0081] The capture module 200 is used to capture the event stream data corresponding to the exposure time of the underexposed image.

[0082] Module 300 is used to construct a spike convolutional neural network using the Spike Response model as the neuron dynamics model.

[0083] The input module 400 is used to input the event stream into the pulse convolutional neural network for forward propagation to obtain the feature vector of the event stream data.

[0084] Enhancement module 500 is used to construct a multimodal fusion network and stitch together the feature vectors of the underexposed image and event stream data to generate an exposed enhanced image through forward propagation using the multimodal fusion network.

[0085] Specifically, in one embodiment of this application, the event stream interception algorithm is as follows:

[0086] ε={(xi ,y i ,t i ,p i )|t start ≤t i ≤t end},

[0087] Where ε is the intercepted event stream, (x i ,y i ) represents the pixel coordinates of event i, and t represents the pixel coordinates of event i. i p is the timestamp of event i. i Let t be the event polarity of event i. start It is the time when the underexposed image I begins to be exposed, t end It is the time when the underexposed image I ends its exposure, i = 1, 2, 3...

[0088] Optionally, in one embodiment of this application, the pulsed convolutional neural network includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. The input convolutional layer has 2 input channels, corresponding to positive and negative events in the event stream. The kernel size of the input convolutional layer is 3*3, the stride is 1, and the number of output channels is 16. The hidden convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 16 output channels. The output convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 32 output channels.

[0089] Optionally, in one embodiment of this application, the multimodal fusion network uses a U-Net structure, the number of input channels in the input layer of the multimodal fusion network is 32+k, and the number of output channels in the output layer is k, where k is the number of channels in the underexposed image.

[0090] Optionally, in one embodiment of this application, the enhancement module 500 includes: a splicing unit.

[0091] The stitching unit is used to stitch the feature vectors of the underexposed image and the event stream data along the channel dimension.

[0092] It should be noted that the foregoing explanation of the low-light image enhancement method based on data fusion also applies to the low-light image enhancement device based on data fusion in this embodiment, and will not be repeated here.

[0093] The low-light image enhancement device based on data fusion proposed in this application can acquire underexposed images and event stream data from an event camera, extract the event stream data corresponding to the exposure time of the underexposed image, construct a pulsed convolutional neural network, input the event stream into the pulsed convolutional neural network for forward propagation to obtain the feature vector of the event stream data, construct a multimodal fusion network, and concatenate the feature vectors of the underexposed image and event stream data. The multimodal fusion network is then used for forward propagation to generate an enhanced image, thereby improving the quality and effect of the acquired image and making the obtained image data more accurate and practical. This solves the problems in related technologies where traditional cameras struggle to acquire effective images in light-limited environments, easily resulting in underexposure, leading to a decrease in the quality of image data acquired in low-light environments, hindering effective observation of low-light scenes, and reducing the practicality of the obtained image results.

[0094] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0095] The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0096] When the processor 602 executes the program, it implements the low-light image enhancement method based on data fusion provided in the above embodiments.

[0097] Furthermore, electronic devices also include:

[0098] Communication interface 603 is used for communication between memory 601 and processor 602.

[0099] The memory 601 is used to store computer programs that can run on the processor 602.

[0100] The memory 601 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0101] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0102] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0103] The processor 602 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0104] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data fusion-based low-light image enhancement method described above.

[0105] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0106] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0107] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0108] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0109] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0110] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0111] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0112] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A low-light image enhancement method based on data fusion, characterized in that, Includes the following steps: Acquire underexposed images and event stream data from a moving event camera that meet preset low-light environmental conditions; Extract the event stream data corresponding to the event stream within the exposure time of the underexposed image; Using the Spike Response model as the neuron dynamics model, a spike convolutional neural network was constructed. The event stream is input into the pulsed convolutional neural network for forward propagation to obtain the feature vector of the event stream data; as well as A multimodal fusion network is constructed, and the feature vectors of the underexposed image and the event stream data are stitched together to generate an exposed enhanced image by forward propagation using the multimodal fusion network. The algorithm for intercepting the event stream is as follows: in, For the captured event stream, For the event The pixel horizontal and vertical coordinates, For the event timestamp, For the event event polarity, It is an underexposed image. The time of initial exposure It is an underexposed image. End of exposure time, ; The feature vector that stitches together the underexposed image and the event stream data includes: The feature vectors of the underexposed image and the event stream data are concatenated along the channel dimension.

2. The method according to claim 1, characterized in that, The pulsed convolutional neural network includes an input convolutional layer, hidden convolutional layers, and an output convolutional layer. The input convolutional layer has 2 input channels, corresponding to positive and negative events in the event stream. The kernel size of the input convolutional layer is 3*3, the stride is 1, and the number of output channels is 16. The hidden convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 16 output channels. The output convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 32 output channels.

3. The method according to claim 1, characterized in that, The multimodal fusion network uses a U-Net architecture, and the number of input channels in the input layer of the multimodal fusion network is 32+. k The number of output channels in the output layer is k ,in, The number of channels in the underexposed image.

4. A low-light image enhancement device based on data fusion, characterized in that, include: The acquisition module is used to acquire underexposed images and event stream data from a moving event camera that meet preset low-light environmental conditions. The interception module is used to intercept the event stream data corresponding to the event stream within the exposure time of the underexposed image; Build modules are used to construct spike convolutional neural networks using the Spike Response model as a neuron dynamics model; The input module is used to input the event stream into the pulse convolutional neural network for forward propagation to obtain the feature vector of the event stream data; as well as An enhancement module is used to construct a multimodal fusion network and stitch together the feature vectors of the underexposed image and the event stream data to perform forward propagation using the multimodal fusion network to generate an exposed enhanced image; The algorithm for intercepting the event stream is as follows: in, For the captured event stream, For the event The pixel horizontal and vertical coordinates, For the event timestamp, For the event event polarity, It is an underexposed image. The time of initial exposure It is an underexposed image. End of exposure time, ; The step of stitching together the feature vectors of the underexposed image and the event stream data includes: stitching together the feature vectors of the underexposed image and the event stream data along the channel dimension.

5. The apparatus according to claim 4, characterized in that, The pulsed convolutional neural network includes an input convolutional layer, hidden convolutional layers, and an output convolutional layer. The input convolutional layer has 2 input channels, corresponding to positive and negative events in the event stream. The kernel size of the input convolutional layer is 3*3, the stride is 1, and the number of output channels is 16. The hidden convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 16 output channels. The output convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 32 output channels.

6. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the low-light image enhancement method based on data fusion as described in any one of claims 1-3.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the low-light image enhancement method based on data fusion as described in any one of claims 1-3.

Images