Method for reinforcing reconstructed image for event camera merging visible light images, medium, and device
The method addresses noise issues in event camera images by fusing visible light images with event camera outputs, resulting in improved image quality and reduced noise in reconstructed images.
Patent Information
- Application Number
- JP2023158282
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2023-09-22
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2043-09-22
AI Technical Summary
Event cameras produce noisy intensity images due to their sensitivity to environmental brightness changes, leading to poor natural image reality and noise in reconstructed images.
A method that fuses visible light images with event camera outputs using a reinforcement module to suppress noise and enhance image quality by selecting reference images based on event stream statistics and performing texture feature transitions.
The method significantly improves the visual quality of reconstructed images by reducing noise and enhancing their similarity to true visible light images, while maintaining high dynamic range characteristics.
Smart Images

Figure 2024162960000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to the field of image processing, and more particularly to a method for enhancing the reconstructed image of an event camera by fusing visible light images. [Background technology]
[0002] An event camera detects the luminance change of each pixel individually in an asynchronous manner, and outputs a corresponding polarity signal when the relative luminance change at a certain position exceeds a set threshold.Compared with conventional visible light imaging cameras that capture scene luminance information at a fixed frame rate, an event camera has the characteristics of high temporal resolution and high dynamic range.
[0003] Currently, many machine vision-related systems (e.g., target detection, identification, and tracking) take grayscale or color images presented in frame format as input, and thus there is a technical demand for reconstructing high frame rate intensity images from event streams. However, the principle by which an event camera records event information determines that it is very sensitive to changes in environmental brightness, and the output event stream inevitably contains a large amount of noise signals, so that the intensity images reconstructed from the event streams usually also contain a large amount of noise. Fortunately, the currently mainstream event cameras based on DAVIS sensors (Dynamic and Active-pixel Vision Sensors) not only record the perceived scene brightness changes in the form of an event stream, but also output grayscale images at a certain frame rate (e.g., 25 frames / second), which opens the possibility of using these grayscale images to augment and process the intensity images reconstructed from the event streams. Summary of the Invention
[0004] The object of the present invention is to solve the problems existing in the prior art and to provide a method for enhancing the reconstructed image of an event camera by fusing visible light images.
[0005] The specific technical solutions of the present invention are as follows: In a first aspect, the present invention provides a method for enhancing a reconstructed image of an event camera by fusing a visible light image, the method comprising: S1, the event stream collected by the event camera is non-overlappingly divided based on fixed time windows, and the event stream in each time window is used as the input of the reconstruction module to obtain a reconstructed image of the event stream in each time window. S2, for each time window, a time stamp corresponding to a center position of the time window is taken as a reference time, and the grayscale images of the most recent frames before and after the reference time are collected from the visible light grayscale image sequence synchronously recorded by the event camera as a first visible light image and a second visible light image, respectively, and the time stamps of the first visible light image and the second visible light image are recorded as a first time and a second time, respectively. In the time window, if the total number of events between the first time and the reference time is less than the total number of events between the reference time and the second time, the first visible light image is taken as the reference image corresponding to the time window, and otherwise the second visible light image is taken as the reference image corresponding to the time window. S3, constructing a reconstructed image and a corresponding reference image of the event stream in each time window into a sample pair, inputting the sample pair into a pre-trained augmentation module, and performing texture feature transition on the reconstructed image by the augmentation module to make the reconstructed image have an appearance expression close to that of the reference image, and suppressing noise in the reconstructed image, and finally forming an augmented image corresponding to each time window.
[0006] In a preferred embodiment of the first aspect, the reconstruction module may be SPADE-E2VID or Firenet.
[0007] As a preferred embodiment of the first aspect, in the event stream collected by the event camera, each event is a quaternion represented by a position coordinate, a timestamp, and an event polarity, where the event polarity values are +1 and -1, where +1 represents an event with increasing brightness and -1 represents an event with decreasing brightness.
[0008] As a preferred embodiment of the first embodiment, in S2, when selecting a reference image from the first visible light image and the second visible light image for each time window, it is necessary to calculate a sum of polarity absolute values of all events between the first time and a reference time and record the sum as a first statistical value. The sum of polarity absolute values of all events between the reference time and a second time is calculated and recorded as a second statistical value. If the first statistical value is smaller than the second statistical value, the first visible light image is selected as the reference image corresponding to the time window, and if not, the second visible light image is selected as the reference image corresponding to the time window.
[0009] In a preferred embodiment of the first aspect, the enhancement module includes an input layer, an image feature extraction module, a texture feature extraction module, a correlation calculation module, a feature transition module, and an output layer.
[0010] The input layer includes a convolution layer followed by a ReLU activation function, and the reconstructed image z and the reference image v in the input sample pair share the input layer, and a feature map M corresponding to the reconstructed image z after being processed by the input layer is obtained. z and the feature map M of the reference image v v Output.
[0011] The image feature extraction module and the texture feature extraction module are cascaded in parallel to the input layer, after which the image feature extraction module is composed of five sequentially connected residual blocks, and the texture feature extraction module adopts a U-Net network. z and feature diagram M v After passing through the image feature extraction module, they respectively form image feature maps Z and V, and feature map M Zand feature diagram M v After passing through the texture feature extraction module, the image feature map Z, the image feature map V, the texture feature map Q and the texture feature map K are formed respectively, and the sizes of the image feature map Z, the image feature map V, the texture feature map Q and the texture feature map K are the same.
[0012] In the correlation calculation module, first, each position on the texture feature map Q and the texture feature map K is scanned, and each scanned position is assigned a unique number during the scanning process. Then, all channel values at the current scanning position are extracted from the scanned feature map to form a feature vector. The feature vector extracted from the position numbered i on the texture feature map Q is q. i The feature vector extracted from the position numbered j in the texture feature map K is k j Next, the correlation between each position on the texture feature map Q and each position on the texture feature map K is calculated. Here, the correlation between the position numbered i on the texture feature map Q and the position numbered j on the texture feature map K is expressed as follows: i and feature vector k j Finally, each position on the texture feature map Q is traversed, and for a position (m, n) whose number is i, the maximum correlation between this position and all positions on the texture feature map K and the number j of the position (m', n') corresponding to the maximum correlation between this position and all positions on the texture feature map K are calculated. * The maximum correlation value found is recorded at position (m,n) of the weight map S, and the position number j is stored. * is recorded at position (m, n) of the position index map P, thereby generating a weight map S and a position index map P having the same size as the texture feature map Q.
[0013] In the feature transition module, the position number j recorded at each position (m, n) in the position index map P *Based on this, it determines the position (m', n') in the texture feature map K, then extracts all channel values at the position (m', n') from the image feature map V, and sets them as the channel values at the position (m, n) in the reconstructed feature map T, thereby constructing the reconstructed feature map T. Then, the image feature map Z and the reconstructed feature map T are stitched along the channel dimension, and the stitching result is converted into a fused feature map Z by a convolution operation. S Forming the fusion feature diagram Z S The weight map S is weighted by element-wise multiplication and then added to the image feature map Z to form the feature map Z0 after the feature transition.
[0014] In the output layer, the feature map Z0 after the feature transition passes through two convolution layers and then undergoes a sigmoid activation operation to form the final augmented image.
[0015] In a preferred embodiment of the first aspect, in the input layer, the convolution layer has a size of 3×3×C in It contains 64 convolution cores, C in is the number of channels of the reconstructed image and the reference image, and the convolution step size of the convolution layer is 1.
[0016] In a preferred embodiment of the first aspect, the two convolutional layers in the output layer include a first convolutional layer that includes 32 3×3 convolutional cores and a second convolutional layer that includes three 3×3 convolutional cores, and the convolution step sizes of both convolutional layers are 1.
[0017] In a preferred embodiment of the first aspect, the network parameters of each layer in the enhancement module are trained in advance using a training sample set, where each training sample is composed of a set of corresponding reconstructed images, reference images and true visible light images, where the reconstructed images and the reference images are input to the enhancement module, and the true visible light images are the true value tags, and the total loss function used in training the enhancement module is a weighted sum of a structural similarity loss function, a mean square error loss function and a sensing loss function, where the sensing loss function uses VGG19 as the sensing model.
[0018] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored therein, the computer program being capable of realizing a method for enhancing a reconstructed image of an event camera by fusing a visible light image as described in any one of the first aspects when executed by a processor.
[0019] In a third aspect, the present invention provides a computer electronic device, which includes a memory and a processor. The memory is used to store computer programs. The processor, when executing the computer program, is used to implement the method for enhancing a reconstructed image of an event camera by fusing visible light images as described in any of the first aspects above.
[0020] Compared with the prior art, the present invention has the following beneficial effects: To address the problem of event cameras in that the intensity image reconstructed from the event stream is easily affected by noise and the natural realism of the image is poor, the reconstructed image and a reference image selected from a grayscale image sequence captured by an optical sensor are input, and texture feature transition is performed on the reconstructed image using an improved Transformer model, so that the reconstructed image is not only closer to the natural image collected by a visible light imaging camera, but also the noise in the reconstructed image can be partially suppressed. [Brief description of the drawings]
[0021] [Figure 1] FIG. 2 is a schematic diagram of a method for enhancing a reconstructed image of an event camera by fusing visible light images according to the present invention; [Diagram 2] 1 is a schematic diagram of an event stream and visible light images collected by an event camera, and a division of a time window. [Diagram 3] FIG. 2 is a structural schematic diagram of a reinforcement module. [Figure 4] 1 is an example of the results of applying the method provided by the present invention to augment the reconstructed image of an event camera. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0022] The present invention will now be further described with reference to the drawings and specific embodiments in order to enable those skilled in the art to better understand the nature of the present invention.
[0023] As shown in Fig. 1, in one preferred embodiment of the present invention, a method for enhancing a reconstructed image of an event camera by fusing a visible light image is provided, which takes an intensity image output from a reconstruction module and a reference image selected from a sequence of visible light images captured by an optical sensor as input, and uses an enhancement module based on an improved Transformer model to perform enhancement processing on the reconstructed image. In this embodiment, the enhancement method specifically includes the following three basic steps, and the specific operation method of each step will be described in detail below.
[0024] Step (1): Divide the events collected by the event camera into time windows without overlapping regions using a fixed interval length, and the event stream in each time window is input to a reconstruction module, which outputs a reconstructed image corresponding to the event stream in each time window.
[0025] In an embodiment of the present invention, only if the events in the event stream collected by the event camera are divided into time windows without overlapping regions, can they be the input of the reconstruction model, and the output of the reconstruction module is the reconstructed intensity image, i.e., the reconstructed image. In the event stream collected by the event camera, each event is a quaternion set represented by a position coordinate, a time stamp, and an event polarity, specifically as follows: tuple e=(x n ,y n ,t n ,p n ), where n is a number, and (x n ,y n ) represents the location coordinate where the event occurred, and t n represents a timestamp, and pn represents the polarity, and its value is (x n ,y n ) position depends on whether the luminance change from the previous event trigger time to the current time exceeds a preset threshold ε. If the luminance change exceeds ε and the luminance is increased, p n = ±1, which represents an event in which the brightness increases. Conversely, when the brightness change exceeds ε and the brightness decreases, p n =-1 represents an event with decreasing brightness. Next, if the start time of a certain time window is t0 and the section length of the time window is ΔT, any event with a timestamp equal to or greater than t0 and less than t0+ΔT is classified as an event within the time window, and these events are input to a reconstruction module to reconstruct a reconstruction intensity image corresponding to the time window.
[0026] In an embodiment of the present invention, the reconstruction module may be SPADE-E2VID (SPADE-E2VID: Spatially-Adaptive Denormalization for Event-Based Video Reconstruction, IEEE Transactions on Image Processing, vol. 30, 2021). Of course, in other embodiments, the reconstruction module may adopt Firenet (Fast Image Reconstruction with an Event Camera, Proc. IEEE / CVF WACV, 2020.3). Both of the above two types of reconstruction modules belong to the prior art, and will not be described.
[0027] Step (2): A reference image corresponding to the reconstructed image in each time window is determined. For each time window, a time stamp corresponding to a center position of the time window is taken as a reference time, a visible light grayscale image of one frame immediately before the reference time is collected from the visible light grayscale image sequence synchronously recorded by the event camera to be a first visible light image, and at the same time, a visible light grayscale image of one frame immediately after the reference time is collected to be a second visible light image, and the time stamps of the first visible light image and the second visible light image are recorded as a first time and a second time, respectively. In the time window, if the total number of events located between the first time and the reference time is less than the total number of events located between the reference time and the second time, the first visible light image is taken as the reference image corresponding to the time window, and otherwise the second visible light image is taken as the reference image corresponding to the time window.
[0028] Specifically, see FIG. 2, where the event camera records the perceived scene luminance changes in the form of an event stream on the one hand, and simultaneously captures images with an optical sensor to output a visible light grayscale image sequence in the form of a grayscale image, but the inter-frame interval of the visible light grayscale image sequence is greater than the length of a time window in the event stream. Therefore, there are many time windows corresponding to the event stream between the visible light grayscale images of two adjacent frames. Let t k Let t k At some time t before k and some later time t k+ A visible light grayscale image is collected for each of the first visible light images f k- and the second visible light image f k+ It is noteworthy that t k- t k It is located before t k The time where there is one visible light grayscale image closest to t k+ t k It is located after t kThe time for which there is one visible light grayscale image closest to t should be determined. The total number of events in a time window can be achieved by counting the sum of the polarity absolute values of all events in the time window. Thus, in an embodiment of the present invention, t k- From k The sum of the polarity absolute values of all events within the time range up to is calculated, and the first statistical value P - t k From k+ The sum of the polarity absolute values of all events within the time range up to is calculated, and the second statistical value P + It is written as follows. - <P + If so, the first visible light image f k- Select the second visible light image f as the reference image, otherwise k+ Select this to use as the reference image.
[0029] In step (3), the reconstructed image and the corresponding reference image of the event stream in each time window are organized into a sample pair, and the sample pair is input to a pre-trained augmentation module, which performs texture feature transition on the reconstructed image to make the reconstructed image have an appearance expression close to the reference image, and suppresses the noise in the reconstructed image, and finally forms an augmented image corresponding to each time window.
[0030] Referring to Fig. 3, in an embodiment of the present invention, the improved Transformer model is used as an enhancement module for any sample pair (z, v) consisting of a reconstructed image z and a reference image v with the same width W and height H. The enhancement module includes an input layer, an image feature extraction module, a texture feature extraction module, a correlation calculation module, a feature transition module and an output layer. Based on the data flow direction during forward propagation, the process flow in the enhancement module includes steps A to E, which are described in detail below:
[0031] In step A, the input data is processed in the input layer. First, in a sample pair input to the augmentation module, the reconstructed image z and the reference image v are input to a common input layer, and after being processed by the input layer, a feature map M corresponding to z is generated. z and the feature map M corresponding to v v The input layer includes a convolutional layer followed by a ReLU activation function, and the convolutional layer in this embodiment has a size of 3×3×C in It contains 64 convolution cores, C in is the number of channels of the reconstructed or reference image, and the grayscale image C in =1, color image C in = 3, and the convolution step size of the convolution layer is 1.
[0032] In step B, image features and texture features are generated by an image feature extraction module and a texture feature extraction module, respectively. After the input layer are two parallel feature extraction modules, one of which is an image feature extraction module and the other is a texture feature extraction module. z and M v After passing through the image feature extraction module, the image feature maps Z and V are formed, respectively, and M z After passing through the texture feature extraction module, Z, V, Q and K respectively form texture feature maps Q and K, and the size of the feature maps Z, V, Q and K is W×H×C, where C is the number of channels of the feature map. In the embodiment of the present invention, the number of channels is C=64. Thus, K, Q and V form the three elements of the Transformer model, namely, key, query and value, which are used for subsequent correlation calculation and feature transition.
[0033] In a preferred embodiment of the present invention, the image feature extraction module connected after the input layer is composed of five sequentially connected residual blocks. Each residual block has the same structure, which includes two parts, and the feature map input to the residual block is A0. In the first part of the residual block, A0 is first subjected to a channel-wise layer normalization operation, and then immediately follows two convolution layers, the first convolution layer includes 128 convolution cores of 1×1 size, and the convolution step size is 1, and the second convolution layer uses grouped convolution, includes 128 convolution cores of 3×3 size, the filling size is 1, the number of groups is 128, and the convolution step size is 1. The feature map output from the second convolution layer, which includes 128 channels, is divided into two feature maps according to channel dimension, each of which includes 64 channels, and a new feature map is generated by calculating from the two feature maps in the form of multiplying corresponding elements, which is denoted as A1. A1 is subjected to a self-adaptive average pooling operation to form a 1x1x64 feature map A2, and the values at all positions of a certain channel of A1 are multiplied by the values of the certain channel of A2, and the above operation is repeated for all channels to form a corrected feature map A'1. A'1 passes through a convolution layer with a convolution step size of 1, including 64 convolution cores of 1x1 size connected later, and the formed feature map is output as the first part of the residual block after element-wise addition with A0, which is denoted as A3. In the second part of the residual block, A3 is first subjected to a layer normalization operation in the direction of one channel, and immediately after that is a convolution layer with a convolution step size of 1, including 128 convolution cores of 1x1 size. The feature map output from the convolution layer is divided into two feature maps according to the channel dimension, each of which contains 64 channels, and a new feature map is generated by multiplying corresponding elements from the two feature maps, denoted as A4. Finally, A4 is added element by element to the feature map output after convolution operation by the subsequent convolution layer with a size of 1×1 and a convolution step size of 1, A3, and A4 is the final output of the residual block.
[0034] In a preferred embodiment of the present invention, the texture feature extraction module following the input layer adopts a U-Net network structure. The feature map input to the texture feature extraction module is B0, and the feature map B1 is formed through two sequentially connected convolution layers and subsequent pooling operations, in which the two convolution layers each include 128 convolution cores of 3×3 size, the convolution step size is both 1, the pooling layer adopts a maximum pooling operation, and the sampling step size is 2. B1 forms a feature map B2 through two sequentially connected convolution layers and subsequent pooling operations, in which the two convolution layers each include 256 convolution cores of 3×3 size, the convolution step size is both 1, the pooling layer adopts a maximum pooling operation, and the sampling step size is 2. B2 forms a feature map B3 through two sequentially connected convolution layers, in which the two convolution layers each include 128 convolution cores of 3×3 size, the convolution step size is both 1. B3 is enlarged to a feature map of the same size as B1 by bicubic interpolation, and then the feature map is added element-wise with B1 to form feature map B4. B4 passes through a convolution layer with 64 convolution cores of 3×3 size and a convolution step size of 1 to form feature map B5. B5 is enlarged to a feature map of the same size as B0 in the form of bicubic interpolation, and then the feature map and B0 are added element-wise to form feature map B6. Finally, B6 passes through a convolution layer with 64 convolution cores of 3×3 size and a convolution step size of 1 to generate the final output of the texture feature extraction module.
[0035] In step C, a position index map and a weight map are generated by the correlation calculation module. In the correlation calculation module, the correlation between the texture feature map Q and the texture feature map K is calculated by the normalized inner product method, and a position index map and a weight map are generated. Specifically, the W×H size plane corresponding to the texture feature map Q is scanned in row order priority, and a unique number i is assigned to each scanning position, where 1≦i≦H×W. At the scanning position corresponding to number i, the values of all channels of the feature map Q are sequentially taken in channel order to form a feature vector; q i Next, similarly, the plane of size W×H corresponding to the texture feature map K is scanned in the row order priority order, and a unique number j is assigned to each scanning position, where 1≦j≦H×W. At the scanning position corresponding to the number j, the values of all channels of the feature map K are sequentially taken in the channel order to form one feature vector, k j Then, the correlation between each position on the texture feature map Q and each position on the texture feature map K is calculated by the normalized inner product method, that is, the vector q is calculated according to the correlation formula: i and k j Calculate the correlation between:
number
[0036] Finally, each position on the texture feature map Q is traversed, and for any position (m, n) whose number is i, the maximum correlation between this position and all positions on the texture feature map K and the correlation maximum between the position (m', n') whose number is j on the texture feature map K are calculated. * The maximum correlation value found is recorded in position (m,n) of the weight map S, and the position number j *is recorded at position (m,n) of the position index map P, thereby generating a weight map S and a position index map P of the same size as the texture feature map Q. Specifically, for a position corresponding to number i on any texture feature map Q, H×W correlation values are calculated according to the correlation formula above, and the maximum value is selected from these correlation values, and recorded in the weight map S corresponding to the feature map Q. At the same time, the number j corresponding to this maximum value is calculated. * and record it in the position index map P.
[0037] The weight map In TIFF2024162960000003.tif5170, each element characterizes the similarity between a feature in feature map Q and its most similar feature in K, and is determined according to the following weighting formula:
number
number
[0038] In step D, feature transition is performed by the feature transition module. The value at the (m,n) position of the position index map P is the index value, i.e., the number j *The index value has a one-to-one correspondence with the position on the H×W plane corresponding to the texture feature map K, and is denoted as (m', n'). Since the texture feature map K and the image feature map V have the same size, (m', n') on the texture feature map K can be mapped onto the image feature map V. In an embodiment of the present invention, the position number j recorded at each position (m, n) in the position index map P is * Based on this, it determines the position (m',n') in the texture feature map K, and then extracts all channel values at the position (m',n') from the image feature map V, and these channel values can be the channel values at the position (m,n) in the reconstructed feature map T, thereby constructing and obtaining the reconstructed feature map T. This operation is equivalent to extracting all channel values one by one in the channel order at the position (m',n') of the feature map V, and forming them into a vector consisting of C elements. The above vector extraction operation is repeated for all positions in the position index map, and the obtained H×W vectors are reconstructed into a feature map of H×W×C, i.e., the reconstructed feature map T. The image feature map Z and the reconstructed feature map T are stitched together in the channel dimension to form a spliced feature map of H×W×2C, which is converted into a fused feature map Z after performing a convolution operation through a convolution layer with a convolution step size of 1, including 64 3×3 convolution cores. S Form the fusion feature diagram Z S All elements of Z S (m,n,c) and the weight map S are weighted by element-wise multiplication, i.e., Z S (m,n,c) needs to be multiplied by the value S(m,n) at the (m,n) position of the weight map to achieve weighting, where 1≦m≦H, 1≦n≦W, and 1≦c≦C. The weighted result is further added to the feature map Z element by element to form the feature map Z0 after feature transition. The above feature transition operation can be expressed as follows:
number
[0039] For illustration, feature map K is a feature map formed after the reference image is processed by the texture feature extraction module. Feature map V is a feature map formed after the reference image is processed by the image feature extraction module. The feature map formed by feature transition is stitched with the feature map formed after the reconstructed image is processed by the image feature extraction module based on the channel dimension. Therefore, the embodiment of the present invention takes, for a position corresponding to each index, the values of all channels at that position from feature map V (not feature map K) to form a feature map for stitching.
[0040] In step E, an augmented image is generated from the output layer. The feature map Z0 undergoes two convolutional layers and subsequent Sigmoid operation to form the final augmented image, in which the first convolutional layer contains 32 3×3 convolutional cores, the second convolutional layer contains three 3×3 convolutional cores, and the convolution step sizes of both convolutional layers are 1.
[0041] It should be noted that before performing enhancement processing on the reconstructed image of the event camera using the enhanced Transformer model-based enhancement module provided by the present invention, the enhancement module needs to be trained, and the parameters of each layer of the enhancement module are determined in a learning manner.
[0042] The training sample set used for model training is Set it to TIFF2024162960000009.tif5170, where z k represents the intensity image output from the reconstruction module, and v k represents the reference image, and g k is reconstructed z k is the true visible light image sampled by the optical sensor during a period corresponding to the event stream of z, where N is the number of samples in the set. k and reference image vk serves as the input to the augmentation module, and the true visible light image g k functions as a truth tag.
[0043] During training, for any sample z in D, k and v k can be input to the augmentation module, and the augmented image output from the augmentation module is If the file is TIFF2024162960000010.tif5170, use the following formula: Calculate the total loss function L corresponding to TIFF2024162960000011.tif4170:
number
number
number
[0044] As shown in FIG. 4, the method provided by the embodiment of the present invention is a part of the results obtained by performing augmentation processing on the reconstructed image of the event camera. Eight different scenes are shown in the figure, and the images in each row are, from left to right, the output result of the augmentation module, the image reconstructed by the reconstruction module (i.e., SPADE-E2VID), and the grayscale image captured with true visible light. As can be seen from the figure, the image after augmentation is not only more similar to the true grayscale image, but also retains the high dynamic range characteristics of the event camera, and reconstructs the details missing due to the wide dynamic range of the grayscale image, such as the details of the person and the chair in the left image in the first row of FIG. 4. At the same time, the embodiment of the present invention takes the true visible light image as a reference, and takes the mean square error MSE, the structural similarity index SSIM, and the learnable perceptual image block similarity LPIPS as the objective basis, and performs experimental tests on multiple test data sets. Table 1 shows the test results. The "Reconstructed Image" column in the table gives the test index value calculated between the output result from the reconstruction module and the reference image, and the "Augmented Image" column gives the index value calculated between the image after augmentation by the augmentation module of the embodiment of the present invention and the reference image. As can be seen from the results of Figure 4 and Table 1, the enhancement method of the reconstructed image of the event camera provided by the embodiment of the present invention can greatly improve the visual effect of the reconstructed image, reduce the error calculated by MSE and LPIPS between the reconstructed image and the true visible light image, and increase the structural similarity index calculated by SSIM. [Table 1]
[0045] Similarly, based on the same inventive concept, in another preferred embodiment of the present invention, there is further provided a computer electronic device corresponding to the method for enhancing a reconstructed image of an event camera by fusing a visible light image provided by the above embodiment, which includes a memory and a processor.
[0046] The memory is used to store computer programs. The processor, when executing the computer program, is used to implement the method for enhancing a reconstructed image of an event camera by fusing a visible light image as described above.
[0047] Moreover, the logic instructions in the memory can be realized in the form of a software functional unit and stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention essentially or a part that contributes to the prior art or a part of the technical solution can be expressed in the form of a software product, and the computer software product is stored in a storage medium and includes multiple commands to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method according to each embodiment of the present invention.
[0048] Thus, based on the same inventive concept, in another preferred embodiment of the present invention, a computer-readable storage medium corresponding to the method for enhancing a reconstructed image of an event camera by fusing visible light images provided by the above-mentioned embodiment is provided, and when a computer program is stored in the storage medium and the computer program is executed by a processor, the method for enhancing a reconstructed image of an event camera by fusing visible light images as described above can be realized.
[0049] As will be understood, the storage medium and memory may be Random Access Memory (RAM) or at least one non-volatile memory (NVM) such as a magnetic disk memory. The storage medium may be a medium capable of storing various program codes, such as a USB memory, a removable hard disk, a magnetic disk, or an optical disk.
[0050] The processor may be a general-purpose processor including a Central Processing Unit (CPU), a Network Processor (NP), etc. It may also be a Digital Signal Processing (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.
[0051] In addition, as should be explained, those skilled in the art can clearly understand that, for convenience and brevity, the specific operation process of the above-described device can refer to the corresponding process in the above-described method embodiment, and the description will be omitted here. In each embodiment provided by the present application, the division of steps or modules in the above-described device and method is merely a logical function division, and other division methods may be used in actual implementation, such as combining or integrating multiple modules or steps, or dividing a module or step.
[0052] The above are merely preferred embodiments of the present invention, but the scope of the claims of the present invention is not limited thereto, and any modifications or substitutions made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.
Claims
1. 1. A method for enhancing a reconstructed image of an event camera that fuses visible light images, comprising: S1: an event stream collected by an event camera is non-overlappingly divided based on fixed time windows, and the event stream in each time window is input to a reconstruction module to obtain a reconstructed image of the event stream in each time window; S2, for each time window, a time stamp corresponding to a center position of the time window is taken as a reference time, and grayscale images of recent frames before and after the reference time are collected from the visible light grayscale image sequence synchronously recorded by the event camera to be a first visible light image and a second visible light image, respectively, and the time stamps of the first visible light image and the second visible light image are recorded as a first time and a second time, respectively, and if a total number of events between the first time and the reference time in the time window is less than a total number of events between the reference time and the second time, the first visible light image is taken as a reference image corresponding to the time window, and otherwise the second visible light image is taken as a reference image corresponding to the time window; S3. Configuring the reconstructed image and the corresponding reference image of the event stream in each time window into a sample pair, inputting the sample pair into a pre-trained augmentation module, and performing texture feature transition on the reconstructed image by the augmentation module to make the reconstructed image have an appearance expression close to that of the reference image, and suppressing noise in the reconstructed image, and finally forming an augmented image corresponding to each time window. A method for enhancing a reconstructed image of an event camera by fusing a visible light image, comprising:
2. The reconstruction module is SPADE-E2VID or Firenet; The method for enhancing a reconstructed image of an event camera by fusing visible light images according to claim 1 .
3. In the event stream collected by the event camera, each event is a quaternion represented by a position coordinate, a timestamp, and an event polarity, where the event polarity values are +1 and -1, where +1 represents an event with increasing brightness and -1 represents an event with decreasing brightness. The method for enhancing a reconstructed image of an event camera by fusing visible light images according to claim 1 .
4. In S2, when selecting a reference image from the first visible light image and the second visible light image for each time window, a sum of polarity absolute values of all events from the first time to a reference time needs to be statistically calculated and recorded as a first statistical value, a sum of polarity absolute values of all events from the reference time to a second time needs to be statistically calculated and recorded as a second statistical value, and if the first statistical value is smaller than the second statistical value, the first visible light image is selected as the reference image corresponding to the time window, otherwise the second visible light image is selected as the reference image corresponding to the time window. The method for enhancing a reconstructed image of an event camera by fusing a visible light image according to claim 3.
5. The enhancement module includes an input layer, an image feature extraction module, a texture feature extraction module, a correlation calculation module, a feature transition module, and an output layer; The input layer includes a convolution layer followed by a ReLU activation function, and the reconstructed image z and the reference image v in the input sample pair share the input layer, and a feature map M corresponding to the reconstructed image z after being processed by the input layer is obtained. z and the feature map M of the reference image v v and The image feature extraction module and the texture feature extraction module are cascaded in parallel to the input layer, after which the image feature extraction module is composed of five sequentially connected residual blocks, the texture feature extraction module adopts a U-Net network, and the feature map M z and feature diagram M v After passing through the image feature extraction module, the image feature map Z and the image feature map V are respectively formed, and the feature map M z and feature diagram M v After passing through the texture feature extraction module, the texture feature map Q and the texture feature map K are respectively formed, and the image feature map Z, the image feature map V, the texture feature map Q and the texture feature map K have the same size; In the correlation calculation module, first, each position on the texture feature map Q and the texture feature map K is scanned, and each scanned position is assigned a unique number during the scanning process. Then, all channel values at the current scanning position are extracted from the scanned feature map to form a feature vector. The feature vector extracted from the position numbered i on the texture feature map Q is q i The feature vector extracted from the position numbered j in the texture feature map K is k j Then, the correlation between each position on the texture feature map Q and each position on the texture feature map K is calculated. Here, the correlation between the position numbered i on the texture feature map Q and the position numbered j on the texture feature map K is expressed as follows: i and feature vector k j Finally, each position on the texture feature map Q is traversed, and for a position (m, n) whose number is i, the maximum correlation between this position and all positions on the texture feature map K and the number j of the position (m'n') corresponding to the maximum correlation between this position and all positions on the texture feature map K are calculated. * The maximum correlation value found is recorded at position (m, n) of the weight map S, and the position number j * at position (m, n) of the position index map P, thereby generating a weight map S and a position index map P having the same size as the texture feature map Q; In the feature transition module, the position number j recorded at each position (m, n) in the position index map P is *Based on the above, it determines the position (m', n') in the texture feature map K, then extracts all channel values at the position (m', n') from the image feature map V, and sets them as the channel values at the position (m, n) in the reconstructed feature map T, thereby constructing the reconstructed feature map T, and then stitches the image feature map Z and the reconstructed feature map T along the channel dimension, and the stitching result is converted into a fused feature map Z by a convolution operation. S and form a fusion feature map Z S The weight map S is weighted by element-by-element multiplication and then added to the image feature map Z. The feature map Z after the feature transition is obtained. 0 Forming In the output layer, the feature map Z after the feature transition 0 The final augmented image is then generated through two convolutional layers and a sigmoid activation operation. The method for enhancing a reconstructed image of an event camera by fusing visible light images according to claim 1 .
6. In the input layer, the convolution layer has a size of 3 × 3 × 3 × C in It contains 64 convolution cores of C in is the number of channels of the reconstructed image and the reference image, and the convolution step size of the convolution layer is 1; The method for enhancing a reconstructed image of an event camera by fusing a visible light image according to claim 5.
7. The two convolution layers in the output layer include: a first convolution layer includes 32 3×3 convolution cores; a second convolution layer includes three 3×3 convolution cores; and the convolution step sizes of the two convolution layers are both 1; The method for enhancing a reconstructed image of an event camera by fusing a visible light image according to claim 5.
8. The network parameters of each layer in the enhancement module are trained in advance using a training sample set, where each training sample is composed of a set of corresponding reconstructed images, reference images and true visible light images, where the reconstructed images and the reference images are input to the enhancement module, and the true visible light images are used as true value tags, and the total loss function used in training the enhancement module is a weighted sum of a structural similarity loss function, a mean squared error loss function and a sensing loss function, where the sensing loss function uses VGG19 as the sensing model. The method for enhancing a reconstructed image of an event camera by fusing visible light images according to claim 1 .
9. A computer-readable storage medium, comprising: When a computer program is stored in the storage medium and executed by a processor, the method for enhancing a reconstructed image of an event camera by fusing visible light images according to any one of claims 1 to 8 can be realized. A computer-readable storage medium comprising:
10. 1. A computer electronic device comprising: A memory and a processor are included. the memory is used to store a computer program; When the processor executes the computer program, the computer program is used to realize the method for enhancing a reconstructed image of an event camera by fusing a visible light image according to any one of claims 1 to 8.
16. A computer electronic device comprising:
Citation Information
Patent Citations
Weak light image enhancement method based on event camera
CN115082341A
Three-dimensional convolution operation device, visual odometry system, and three-dimensional convolution program
JP2019211879A
