Aerial image detail enhancement method based on deep learning

Abstract

The invention provides an aerial image detail enhancement method based on deep learning. The aerial image detail enhancement method comprises the following steps: constructing a cyclic convolutional neural network with a memory state; the cyclic convolutional neural network with the memory state comprises a denoising network, an event reconstruction image network and a fusion network; in a low-illumination scene, a low-quality video frame sequence VL * and a low-quality event stream FL * of a target area are acquired respectively in each acquisition period through a visual camera and a common camera, the low-quality video frame sequence VL * and the low-quality event stream FL * are input into the trained cyclic convolutional neural network with the memory state, a reconstructed first high-quality video frame sequence VHR1 * is output, and a reconstructed second high-quality video frame sequence VHR2 * is output; the video frame sequence is a finally obtained video frame sequence after image detail enhancement. According to the model designed by the invention, the high dynamic range information of the event stream data and the high texture detail information of the image frame sequence data are utilized at the same time, and the visual perception enhanced image with the high dynamic range and the high texture detail information at the same time can be constructed through fusion.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for enhancing aerial image details based on deep learning. Background Art

[0002] With the rapid development of modern industry and the widespread application of computer technology, the structure of image acquisition equipment is becoming more and more sophisticated, the market stock is growing, and the use is becoming more and more convenient. At the same time, with the development of storage devices and Internet technology, the number of digital images has exploded worldwide, and the quality of digital images has become higher and higher. This is mainly reflected in the increasing resolution of images, more delicate colors, and clearer pictures. Therefore, the huge amount of image data that can be easily obtained is sufficient to support a variety of computer vision tasks, which also lays a solid foundation for the development of the field of computer vision. However, due to various environmental factors such as long distance and low light that may appear in the shooting scene, and the inevitable signal errors and noise in the process of capturing images by digital image acquisition equipment, the acquired digital images are sometimes distorted or even unrecognizable. For this reason, image enhancement has attracted widespread attention in the field of computer vision in recent years.

[0003] Existing image enhancement technologies mainly focus on reducing the noise signal contained in the image, correcting the local blur in the image, improving the brightness contrast of low-light images, and other related tasks. The specific implementation methods can be mainly divided into traditional methods and machine learning methods. For traditional methods, such as grayscale transformation, spatial domain filtering, frequency domain filtering, histogram enhancement and other traditional image enhancement methods, their implementation is relatively simple and the effect is relatively fixed, which is difficult to meet the current demand for image enhancement. For machine learning methods, including various methods based on deep learning, they are the focus of academic research today.

[0004] At present, in the field of deep learning, traditional cameras are generally used to capture scene images, which are then used in deep learning networks. This method has the following problems:

[0005] When the distance between a traditional camera and the visual target in the scene to be photographed is large, exceeding the concept of "close-up shooting" that people usually think of, the visual target is considered to be a long-distance visual target. For example, aerial photography is a typical shooting scene for long-distance visual targets. When the traditional camera is far away from the scene to be photographed and the visual target therein, the visual target has a smaller range in the scene, and the amount of light reflected by the target entering the camera is small, which easily leads to low brightness of the target and the image as a whole. At the same time, the reduction in the amount of light entering will cause the noise signal of the image to occupy a larger proportion of the total information signal of the image, that is, resulting in a lower signal-to-noise ratio. In addition, since the image range contained in the long-distance aerial photography scene is large, if the camera has a slight shake at this time, it will cause the entire image to change significantly in a short period of time, making it more likely to be blurred when the exposure time is long.

[0006] Therefore, how to process aerial images taken by traditional cameras and effectively improve the quality of aerial images is a key issue that needs to be solved at present. Summary of the invention

[0007] In view of the defects of the prior art, the present invention provides an aerial image detail enhancement method based on deep learning, which can effectively solve the above problems.

[0008] The technical solution adopted by the present invention is as follows:

[0009] The present invention provides a method for enhancing aerial image details based on deep learning, comprising the following steps:

[0010] Step S1, obtaining a training sample set; each training sample in the training sample set simultaneously includes a high-quality event stream, a low-quality event stream, a low-quality video frame sequence, and a high-quality video frame sequence corresponding to the same time interval and actually acquired from the same area;

[0011] Wherein: the high-quality event stream and the low-quality event stream are acquired by a visual camera; the high-quality video frame sequence and the low-quality video frame sequence are acquired by an ordinary camera;

[0012] Step S2, for each training sample, perform data preprocessing to obtain a preprocessed training sample; the preprocessed training sample includes a preprocessed high-quality event stream, a low-quality event stream, a low-quality video frame sequence, and a high-quality video frame sequence, and the high-quality event stream, the low-quality event stream, the low-quality video frame sequence, and the high-quality video frame sequence are respectively represented as: a high-quality event stream FH, a low-quality event stream FL, a low-quality video frame sequence VL, and a high-quality video frame sequence VH;

[0013] Step S3, constructing a recurrent convolutional neural network with a memory state; the recurrent convolutional neural network with a memory state includes a denoising network, an event reconstruction image network and a fusion network;

[0014] Step S4, using the preprocessed training samples to train the recurrent convolutional neural network with memory state, to obtain the trained recurrent convolutional neural network with memory state, the specific training method is:

[0015] Step S4.1, inputting the low-quality event stream FL into the denoising network, performing denoising processing through the denoising network, and generating an event stream FL1;

[0016] Step S4.2, inputting the event stream FL1 generated in step S4.1 into the event reconstruction image network, and the event reconstruction image network outputs a reconstructed video frame sequence VR;

[0017] Step S4.3, inputting the reconstructed video frame sequence VR and the low-quality video frame sequence VL obtained in step S4.2 into the fusion network, and the fusion network outputs a reconstructed first high-quality video frame sequence VHR1;

[0018] Step S4.4, inputting the high-quality event stream FH into the event reconstruction image network, and the event reconstruction image network outputs a reconstructed second high-quality video frame sequence VHR2;

[0019] Step S4.5, calculating a first loss function value between the reconstructed video frame sequence VR and the reconstructed second high-quality video frame sequence VHR2;

[0020] Calculate a second loss function value between the reconstructed video frame sequence VR and the high-quality video frame sequence VH;

[0021] Calculate a third loss function value between the reconstructed first high-quality video frame sequence VHR1 and the high-quality video frame sequence VH;

[0022] Combining the first loss function value, the second loss function value, and the third loss function value to obtain a comprehensive loss function value;

[0023] Step S4.6, updating the network parameters of the recurrent convolutional neural network with memory state according to the comprehensive loss function value;

[0024] Return to step S4.1, use the next training sample, and continue training the recurrent convolutional neural network with memory state until the recurrent convolutional neural network with memory state reaches the training termination condition;

[0025] Step S5: In a low-light scene, a low-quality video frame sequence VL of the target area is acquired by using a visual camera and a common camera in each acquisition cycle. * and low-quality event stream FL * , input to the trained recurrent convolutional neural network with memory state, and output the reconstructed first high-quality video frame sequence VHR1 * , which is the final video frame sequence after image detail enhancement.

[0026] Preferably, in step S2, the low-quality video frame sequence and the high-quality video frame sequence are preprocessed in the following manner:

[0027] Each frame image in the low-quality video frame sequence and the high-quality video frame sequence actually acquired is converted into a grayscale image.

[0028] Preferably, it is converted into a grayscale image by formula (1):

[0029] I gray =0.2989×I rgb [r]+0.5870×I rgb [g]+0.1140×I rgb [b] (1)

[0030] Where: I gray is a grayscale image, I rgb [r]、I rgb [g] and I rgb [b] are the RGB three-channel image components of each frame of image.

[0031] Preferably, in step S2, the high-quality event stream and the low-quality event stream are preprocessed by converting the event stream into a voxel grid.

[0032] Preferably, the event stream is converted into a voxel grid, specifically:

[0033] ① Divide the entire event stream into several event stream units with a time interval length of ΔT according to the event timestamp;

[0034] ② For each event stream unit, the start time is denoted as t0, and the end time is: t0+ΔT. Suppose in the time interval length [t0, t0+ΔT], there are a total of B timestamps, denoted as: t0, t1, …, t B-1 ; Therefore, any timestamp is denoted as t i , i=0,1,…,B-1;

[0035] ③For each event stream unit, each timestamp t i A container is generated for all events;

[0036] ④Use formula (2) to convert the timestamp t i Standardize to get the standardized timestamp

[0037]

[0038] ⑤Convert all events in each event stream unit into a voxel grid E;

[0039] The size of the voxel grid E is B×H×W, where B is the number of timestamps and the number of containers; H represents the number of pixels in the length direction, and W represents the number of pixels in the width direction; then for each pixel, its two-dimensional coordinate is (x l ,y m ), where l = 0, 1, ..., H-1, m = 0, 1, ..., W-1; its three-dimensional coordinates with time attributes are: (x l ,y m ,t i ), using formula (3), we get the voxel pixel weighted value E(x l ,y m ,t i ):

[0040]

[0041] in:

[0042] ω is the three-dimensional coordinate (x l ,y m ,t i ) at the pixel value;

[0043] p jki The meaning is: the event camera asynchronously senses the brightness signal change of the incident light at each pixel position, and generates an event when the absolute value of the brightness signal change exceeds the threshold CT, using the four-tuple e jki =(x j ,y k ,p jki ,t i ) means, where (x j ,y k ) represents the horizontal and vertical coordinates of the pixel where the event is triggered, t i Indicates the timestamp of the event; p jki Indicates the polarity of the event, p jki ∈{-1,1}, that is: p jki -1 means darkening; p jki is 1, indicating brightening; if no event occurs at the corresponding pixel position, then p jkiis a null value;

[0044] α is the mixing weight coefficient, the size of which is determined by the proportion of events generated in the voxel grid E, and the value ranges from 0 to 1, that is: α∈[0,1];

[0045] Therefore, in determining the voxel pixel weight value E(x l ,y m ,t i ), the voxel grid E is determined.

[0046] Preferably, when calculating the loss function value between each video frame sequence, the voxel grid E of each video frame sequence is first regularized, and then the regularized voxel grid E is normalized. The loss is calculated using Euclidean distance.

[0047] Preferably, the voxel grid E is regularized by:

[0048] ① For each voxel pixel weighted value E(x l ,y m ,t i ), using formula (4), we get the intermediate variable Mask(x l ,y m ,t i ):

[0049]

[0050] ② Using formula (5), all intermediate variables Mask(x l ,y m ,t i ) and sum them to obtain the weighted values ​​of voxels E(x l ,y m ,t i ), denoted as N;

[0051]

[0052] ③Use formula (6) to obtain the mean μ:

[0053]

[0054] ④Use formula (7) to obtain the variance σ:

[0055]

[0056] ⑤Use formula (8) to obtain the voxel pixel weighted value E(x l ,y m ,t i )

[0057]

[0058] By weighting each voxel pixel by E(x l ,y m ,t i ) regularization, that is, regularizing the voxel grid E.

[0059] The aerial image detail enhancement method based on deep learning provided by the present invention has the following advantages:

[0060] 1. The model designed by the present invention utilizes the high dynamic range information of event stream data and the high texture detail information of image frame sequence data, and can be integrated to construct a visual perception enhanced image with both high dynamic range and high texture detail information;

[0061] 2. In the process of converting event stream data into voxel grids, the contribution of pixel values ​​is also considered, and the method of the present invention utilizes the information of voxel grids and pixel coordinates at the same time by means of mixed weights.

[0062] 3. The method of the present invention has the ability to improve image quality and enhance image details to a certain extent, and has application value in the field of aerial photography. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A schematic diagram of the overall model structure of an aerial image detail enhancement method based on deep learning provided by the present invention;

[0064] Figure 2 A schematic diagram of the network structure used in the present invention;

[0065] Figure 3 A schematic diagram of the fusion of the reconstructed image and the low-quality image provided by the present invention;

[0066] Figure 4 The real data low-light scene image enhancement effect provided by the present invention;

[0067] Figure 5 This is the fusion effect of the event reconstruction image and the low-light scene image provided by the present invention. DETAILED DESCRIPTION

[0068] In order to make the technical problems, technical solutions and beneficial effects solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0069] An event camera is a new type of visual sensor, also known as a neuromorphic visual sensor. Inspired by neuroscience, it imitates the working principle of the eye, asynchronously captures the brightness changes of each pixel, and organizes the data in an "event stream" structure. This is fundamentally different from the way traditional cameras expose at a fixed time to capture fixed frame rate light intensity images. Due to the various advantages of event cameras, event cameras have broad development prospects in many application fields such as autonomous driving, robots, and wearable devices. At present, in many scientific research fields of computer vision, event cameras have also produced many related research and application results.

[0070] Event cameras also have their shortcomings. Limited by their imaging principles, the event data collected when the scene changes little is relatively sparse, making it difficult to parse the target. In addition, the current event camera imaging noise is large, the price is expensive, and there is little public data. Research in the field of event cameras still faces many problems. By combining event cameras with traditional cameras and using the obtained multi-camera fusion model to capture high temporal resolution, high dynamic range event streams and low frame rate, high texture detail intensity image sequences in the same scene, the advantages of the two can be complemented to generate high temporal resolution, high dynamic range, and high texture detail intensity images, which can then be used to support subsequent visual target parsing tasks.

[0071] Therefore, the present invention uses a multi-camera fusion model of event cameras and traditional cameras and the heterogeneous visual stream data generated by them to improve the various problems caused by the long distance between the camera and the scene to be photographed, such as improving the overall brightness of the image, reducing the blurred part in the image, and reducing the noise signal in the image signal, thereby improving the perceptibility of long-distance visual targets and providing better support for subsequent visual target parsing tasks.

[0072] The present invention is particularly suitable for the field of aerial remote sensing image data processing technology. It is aimed at the practical application of aerial image processing and takes improving the texture details of aerial images as the main purpose. It proposes an image texture enhancement method optimized for aerial scenes based on deep learning. The method includes two core steps: first, low-quality event denoising and image reconstruction; second, the fusion of the reconstructed image and the actual collected low-quality image. The method aims to utilize the high dynamic range information of the event stream data and the high texture detail information of the actual collected image frame sequence data to fuse and construct a perception-enhanced image with both high dynamic range and high texture detail information. The method of the present invention has a certain ability to improve image quality and enhance image perception, and has application value in the field of aerial photography.

[0073] See also Figure 1 The present invention provides a method for enhancing aerial image details based on deep learning, comprising the following steps:

[0074] Step S1, obtaining a training sample set; each training sample in the training sample set includes a high-quality event stream, a low-quality event stream, a low-quality video frame sequence, and a high-quality video frame sequence corresponding to the same time interval obtained by real aerial photography of the same area;

[0075] Wherein: the high-quality event stream and the low-quality event stream are acquired by a visual camera; the high-quality video frame sequence and the low-quality video frame sequence are acquired by an ordinary camera;

[0076] Step S2, for each training sample, perform data preprocessing to adapt to the input format of the network model, and obtain a preprocessed training sample;

[0077] The preprocessed training samples include a preprocessed high-quality event stream, a low-quality event stream, a low-quality video frame sequence, and a high-quality video frame sequence, and the high-quality event stream, the low-quality event stream, the low-quality video frame sequence, and the high-quality video frame sequence are respectively represented as: a high-quality event stream FH, a low-quality event stream FL, a low-quality video frame sequence VL, and a high-quality video frame sequence VH;

[0078] In step S2, the low-quality video frame sequence and the high-quality video frame sequence are preprocessed in the following manner:

[0079] Each frame image in the low-quality video frame sequence and the high-quality video frame sequence actually acquired is converted into a grayscale image. Specifically, the grayscale image is converted into a grayscale image using formula (1):

[0080] I gray =0.2989×I rgb [r]+0.5870×I rgb [g]+0.1140×I rgb [b] (1)

[0081] Where: I gray is a grayscale image, I rgb [r]、I rgb [g] and I rgb [b] are the RGB three-channel image components of each frame of image.

[0082] In step S2, the high-quality event stream and the low-quality event stream are preprocessed by: converting the event stream into a voxel grid; converting the event stream into a voxel grid, specifically:

[0083] Voxels are analogous to pixels, and are the smallest unit in a three-dimensional space. Voxel grids can be compared to an image plane composed of pixels, and are three-dimensional cubes composed of voxels. The event voxel grid contains three dimensions, two of which are the horizontal and vertical coordinates of the pixel, and the third dimension is set as a container variable (Bin), which represents the degree of subdivision of the voxel grid in time series.

[0084] ① Divide the entire event stream into several event stream units with a time interval length of ΔT according to the event timestamp;

[0085] ② For each event stream unit, the start time is denoted as t0, and the end time is: t0+ΔT. Suppose in the time interval length [t0, t0+ΔT], there are a total of B timestamps, denoted as: t0, t1, …, t )-1 ; Therefore, any timestamp is denoted as t i , i=0,1,…,B-1;

[0086] ③For each event stream unit, each timestamp t i A container is generated for all events;

[0087] ④Use formula (2) to convert the timestamp t i Standardize to get the standardized timestamp

[0088]

[0089] ⑤Convert all events in each event stream unit into a voxel grid E;

[0090] The size of the voxel grid E is B×H×W, where B is the number of timestamps and the number of containers; H represents the number of pixels in the length direction, and W represents the number of pixels in the width direction; then for each pixel, its two-dimensional coordinate is (x l ,y m ), where l = 0, 1, ..., H-1, m = 0, 1, ..., W-1; its three-dimensional coordinates with time attributes are: (x l ,y m ,t i ), using formula (3), we get the voxel pixel weighted value E(x l ,y m ,t i ):

[0091]

[0092] in:

[0093] ω is the three-dimensional coordinate (x l,y m , t i ) at the pixel value;

[0094] p jki The meaning is: the event camera asynchronously senses the brightness signal change of the incident light at each pixel position, and generates an event when the absolute value of the brightness signal change exceeds the threshold CT (Contrast Threshold, CT), using the four-tuple e jki =(x j ,y k , p hki , t i ) means, where (x j ,y k ) represents the horizontal and vertical coordinates of the pixel where the event is triggered, t i Indicates the timestamp of the event; p jki Indicates the polarity of the event, p jki ∈{-1, 1}, that is: p jki -1 means darkening; p jki is 1, indicating brightening; if no event occurs at the corresponding pixel position, then p jki is a null value;

[0095] α is the mixing weight coefficient, the size of which is determined by the proportion of events generated in the voxel grid E, and the value ranges from 0 to 1, that is: α∈[0,1];

[0096] Therefore, in determining the voxel pixel weight value E(x l ,y m , t i ), the voxel grid E is determined.

[0097] In the present invention, each event within ΔT is assigned its polarity value to its two adjacent containers according to the time-series distance relationship, and has no effect on the containers outside the adjacent containers. The voxel grid generated in this way not only meets the fixed size characteristics, but also retains the time sequence information of the event stream data to a certain extent, which makes it possible to maximize the use of useful information of the event stream data in the subsequent training process. At the same time, considering the influence of the pixel value ω, the proportion of the pixel value in the grid E can be weighted and mixed with the voxel value through user-defined control, so that the event contains richer information and the expression method is more flexible than a single method.

[0098] Step S3, constructing a recurrent convolutional neural network with a memory state; the recurrent convolutional neural network with a memory state includes a denoising network, an event reconstruction image network and a fusion network;

[0099] The recurrent convolutional neural network with memory state is used to map the input heterogeneous visual stream data into a perceptually enhanced image through function calculation by designing a network structure, and output it;

[0100] As a specific embodiment, the denoising network, the event reconstruction image network and the fusion network have the following structural characteristics:

[0101] (I) Denoising Network

[0102] Denoising network: used to remove noise from the input low-quality event stream, using the improved U-Net network structure. Through experimental tests, the denoising effect is best when the number of downsampling units and upsampling units in the downsampling module and upsampling module is set to 4; U-Net network such as Figure 2 As shown in the figure, the left side is a downsampling module, which includes multiple downsampling units, each of which sequentially includes two convolutional layers with a convolution kernel size of 3×3, a linear rectifier function (Rectified Linear Unit, ReLU), and a pooling layer of 2×2 size and a stride of 2. The right side of the network is an upsampling module, which includes several upsampling units, each of which sequentially includes an upsampling layer, a deconvolution layer, and a cropping operation, and after cropping, the feature map is spliced ​​with the corresponding downsampled feature map.

[0103] Considering the temporal characteristics of event data, the convolutional layer in the downsampling module is modified to a long short-term memory network (LSTM) layer, so that it can memorize the characteristics of the previous data to better complete the denoising task;

[0104] Adjust the output of the denoising grid structure and set the number of input channels in the denoising network structure to 3 to ensure that the same l ,y m ,t i ) has the same output format dimension, namely: E(x l ,y m ,t i ) contains 3 output data. Therefore, the number of input channels and output channels in the denoising network structure are set to 3 to meet the format of the event voxel grid data.

[0105] (II) Event Reconstruction Image Network

[0106] The high-quality event stream FH and the generated event stream FL1 share the same set of event reconstruction image networks and share weights.

[0107] 3. Converged Network

[0108] The specific approach is: the reconstructed video frame sequence VR and the low-quality video frame sequence VL obtained through the low-quality event stream are spliced ​​by channel and input into the fusion network, and finally a high-quality video frame image with high dynamic range and high texture detail features is synthesized.

[0109] The fusion network structure refers to the U-Net network structure, and the number of downsampling units and upsampling units in the downsampling module and upsampling module are set to 3. In addition, in the fusion network structure, the number of input channels is set to 2 and the number of output channels is set to 1 to meet the requirements of two spliced ​​grayscale images as input and one fused grayscale image as output. The complete reconstructed image and low-quality image fusion structure is shown in the figure. Figure 3 shown.

[0110] Step S4, using the preprocessed training samples to train the recurrent convolutional neural network with memory state, to obtain the trained recurrent convolutional neural network with memory state, the specific training method is:

[0111] Step S4.1, inputting the low-quality event stream FL into the denoising network, performing denoising processing through the denoising network, and generating an event stream FL1;

[0112] Step S4.2, inputting the event stream FL1 generated in step S4.1 into the event reconstruction image network, and the event reconstruction image network outputs a reconstructed video frame sequence VR;

[0113] Step S4.3, inputting the reconstructed video frame sequence VR and the low-quality video frame sequence VL obtained in step S4.2 into the fusion network, and the fusion network outputs a reconstructed first high-quality video frame sequence VHR1;

[0114] Step S4.4, inputting the high-quality event stream FH into the event reconstruction image network, and the event reconstruction image network outputs a reconstructed second high-quality video frame sequence VHR2;

[0115] Step S4.5, calculating a first loss function value between the reconstructed video frame sequence VR and the reconstructed second high-quality video frame sequence VHR2;

[0116] Calculate a second loss function value between the reconstructed video frame sequence VR and the high-quality video frame sequence VH;

[0117] Calculate a third loss function value between the reconstructed first high-quality video frame sequence VHR1 and the high-quality video frame sequence VH;

[0118] Combining the first loss function value, the second loss function value, and the third loss function value to obtain a comprehensive loss function value;

[0119] Step S4.6, updating the network parameters of the recurrent convolutional neural network with memory state according to the comprehensive loss function value;

[0120] Return to step S4.1, use the next training sample, and continue training the recurrent convolutional neural network with memory state until the recurrent convolutional neural network with memory state reaches the training termination condition;

[0121] For example, by setting the loss function, calculating the gradient and passing it back to each weight, the network weights are updated through the gradient descent method to achieve iterative optimization of the entire network.

[0122] The specific method is:

[0123] The voxel grid generated by the event stream is measured according to the calculation formula of the mean absolute error loss between images. First, the voxel grid is regularized, and then the event loss of two voxel grids is calculated using the Euclidean distance (L2Loss) to calculate the error. By setting the loss function, the gradient is calculated and passed back to each weight, and the weight is updated by the gradient descent method to achieve iterative optimization of the entire network.

[0124] The specific steps are:

[0125] Event Loss is used to measure the gap between event data. Specifically, the present invention measures the voxel grid generated by the event stream according to the calculation formula of the mean absolute error loss between images. The voxel value distribution between voxel grids of different data is quite different and there is no maximum value limit. Before measuring the loss between voxel grids, the voxel grid needs to be regularized.

[0126] In the present invention, if the voxel grid is used as a unit, when calculating each loss function, it is necessary to regularize the voxel grid E of each video frame sequence, and then regularize the voxel grid E of each video frame sequence. The loss is calculated using Euclidean distance.

[0127] The voxel grid E is regularized as follows:

[0128] ① For each voxel pixel weighted value E(x l ,y m ,t i ), using formula (4), we get the intermediate variable Mask(x l ,y m ,t i ):

[0129]

[0130] ② Using formula (5), all intermediate variables Mask(x l ,y m ,t i ) and sum them to obtain the weighted values ​​of voxels E(x l ,y m ,t i ), denoted as N;

[0131]

[0132] ③Use formula (6) to obtain the mean μ:

[0133]

[0134] ④Use formula (7) to obtain the variance σ:

[0135]

[0136] ⑤Use formula (8) to obtain the voxel pixel weighted value E(x l ,y m ,t i )

[0137]

[0138] By weighting each voxel pixel by E(x l ,y m ,t i ) regularization, that is, regularizing the voxel grid E.

[0139] Step S5: In a low-light scene, a low-quality video frame sequence VL of the target area is acquired by using a visual camera and a common camera in each acquisition cycle. * and low-quality event stream FL * , input to the trained recurrent convolutional neural network with memory state, and output the reconstructed first high-quality video frame sequence VHR1 * , which is the final video frame sequence after image detail enhancement.

[0140] The aerial scene visual perception enhancement method proposed in this invention was tested and verified on real data. Figure 4 It can be seen that the method proposed in the present invention has a good effect on the image enhancement of low-light heterogeneous visual flow data of aerial scenes taken in the field, and is clearer and brighter than aerial low-light images.

[0141] Figure 5The fusion effect of the present invention on the event reconstruction image and the low-light scene image is demonstrated. Comparing the event reconstruction image with the fused image, the fused image has more texture details than the event reconstruction image, that is, the method of the present invention obtains texture detail information from the low-quality image and combines it with the dynamic range information of the event reconstruction image, and finally generates a high-quality image with both high dynamic range and high texture details.

[0142] The present invention relates to the field of aerial remote sensing image data enhancement in image processing. Aiming at the practical application of aerial image processing, with the main purpose of improving the texture details of aerial images, a deep learning-based image texture enhancement method optimized for aerial scenes is proposed.

[0143] The present invention aims at aerial remote sensing images of poor quality, enhances the proportion of useful information in the images, emphasizes certain specific overall features or local features in the images, clarifies the originally unclear images, emphasizes certain specific image features while suppressing other image features, thereby improving the image quality, enriching the detailed information of the images, and enhancing the perceptual effect of the images.

[0144] The present invention utilizes the advantages of both video frame sequence data and event stream data, complements the advantages of the two, and generates high-quality images with high dynamic range and high texture detail features, thereby enhancing the visual detail effect of the aerial photography scene.

[0145] The principle of the present invention is:

[0146] From the perspective of image enhancement, for low-quality video frame sequences captured when scene illumination is low, the image perception effect can be enhanced by means of the high dynamic range information introduced by the paired low-quality event stream data, including brightness improvement and noise removal; from the perspective of event stream reconstruction images, for low-quality event stream data captured when scene illumination is low, the reconstruction effect of event stream reconstruction images can be improved by means of the high texture detail information introduced by the paired low-quality video frame sequence data. By utilizing the advantages of both video frame sequence data and event stream data, high-quality images with both high dynamic range and high texture detail features can be generated, thereby achieving innovation in enhancing the visual detail effects of aerial photography scenes.

[0147] The advantages of the present invention compared with the prior art are:

[0148] 1. The model designed by the present invention utilizes the high dynamic range information of event stream data and the high texture detail information of image frame sequence data, and can be integrated to construct a visual perception enhanced image with both high dynamic range and high texture detail information;

[0149] 2. In the process of converting event stream data into voxel grids, the contribution of pixel values ​​is also considered, and the method of the present invention utilizes the information of voxel grids and pixel coordinates at the same time by means of mixed weights.

[0150] 3. The method of the present invention has the ability to improve image quality and enhance image details to a certain extent, and has application value in the field of aerial photography.

[0151] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A method for enhancing aerial image details based on deep learning, characterized in that: The following steps are involved: Step S1, obtaining a training sample set; each training sample in the training sample set simultaneously includes a high-quality event stream, a low-quality event stream, a low-quality video frame sequence, and a high-quality video frame sequence corresponding to the same time interval and actually acquired from the same area; Wherein: the high-quality event stream and the low-quality event stream are acquired by a visual camera; the high-quality video frame sequence and the low-quality video frame sequence are acquired by an ordinary camera; Step S2, for each training sample, perform data preprocessing to obtain a preprocessed training sample; the preprocessed training sample includes a preprocessed high-quality event stream, a low-quality event stream, a low-quality video frame sequence, and a high-quality video frame sequence, and the high-quality event stream, the low-quality event stream, the low-quality video frame sequence, and the high-quality video frame sequence are respectively represented as: a high-quality event stream FH, a low-quality event stream FL, a low-quality video frame sequence VL, and a high-quality video frame sequence VH; Step S3, constructing a recurrent convolutional neural network with a memory state; the recurrent convolutional neural network with a memory state includes a denoising network, an event reconstruction image network and a fusion network; Step S4, using the preprocessed training samples to train the recurrent convolutional neural network with memory state, to obtain the trained recurrent convolutional neural network with memory state, the specific training method is: Step S4.1, inputting the low-quality event stream FL into the denoising network, performing denoising processing through the denoising network, and generating an event stream FL1; Step S4.2, inputting the event stream FL1 generated in step S4.1 into the event reconstruction image network, and the event reconstruction image network outputs a reconstructed video frame sequence VR; Step S4.3, inputting the reconstructed video frame sequence VR and the low-quality video frame sequence VL obtained in step S4.2 into the fusion network, and the fusion network outputs a reconstructed first high-quality video frame sequence VHR1; Step S4.4, inputting the high-quality event stream FH into the event reconstruction image network, and the event reconstruction image network outputs a reconstructed second high-quality video frame sequence VHR2; Step S4.5, calculating a first loss function value between the reconstructed video frame sequence VR and the reconstructed second high-quality video frame sequence VHR2; Calculate a second loss function value between the reconstructed video frame sequence VR and the high-quality video frame sequence VH; Calculate a third loss function value between the reconstructed first high-quality video frame sequence VHR1 and the high-quality video frame sequence VH; Combining the first loss function value, the second loss function value, and the third loss function value to obtain a comprehensive loss function value; Step S4.6, updating the network parameters of the recurrent convolutional neural network with memory state according to the comprehensive loss function value; Return to step S4.1, use the next training sample, and continue training the recurrent convolutional neural network with memory state until the recurrent convolutional neural network with memory state reaches the training termination condition; Step S5: In a low-light scene, a low-quality video frame sequence VL of the target area is acquired by using a visual camera and a common camera in each acquisition cycle. * and low-quality event stream FL * , input to the trained recurrent convolutional neural network with memory state, and output the reconstructed first high-quality video frame sequence VHR1 * , which is the final video frame sequence after image detail enhancement.

2. The method for enhancing aerial image details based on deep learning according to claim 1, characterized in that: In step S2, the low-quality video frame sequence and the high-quality video frame sequence are preprocessed in the following manner: Each frame image in the low-quality video frame sequence and the high-quality video frame sequence actually acquired is converted into a grayscale image.

3. The method for enhancing aerial image details based on deep learning according to claim 2, characterized in that: Convert to grayscale image using formula (1): I gray =0.2989×I rgb [r]+0.5870×I rgb [g]+0.1140×I rgb [b] (1) Among them: gray is a grayscale image, I rgb [r], I rgb [g] and I rgb [b] are the RGB three-channel image components of each frame of image.

4. The method for enhancing aerial image details based on deep learning according to claim 1, characterized in that: In step S2, the high-quality event stream and the low-quality event stream are preprocessed by converting the event stream into a voxel grid.

5. The method for enhancing aerial image details based on deep learning according to claim 4, characterized in that: Convert the event stream into a voxel grid, specifically: ① Divide the entire event stream into several event stream units with a time interval length of ΔT according to the event timestamp; ② For each event stream unit, the start time is denoted as t0, and the end time is: t0+ΔT. Suppose in the time interval length [t0, t0+ΔT], there are a total of B timestamps, denoted as: t0, t1, …, t B-1 ; Therefore, any timestamp is denoted as t i , i=0,1,…,B-1; ③For each event stream unit, each timestamp t i A container is generated for all events; ④Use formula (2) to convert the timestamp t i Standardize to get the standardized timestamp ⑤Convert all events in each event stream unit into a voxel grid E; The size of the voxel grid E is B×H×W, where B is the number of timestamps and the number of containers; H represents the number of pixels in the length direction, and W represents the number of pixels in the width direction; then for each pixel, its two-dimensional coordinate is (x l ,y m ), where l = 0, 1, ..., H-1, m = 0, 1, ..., W-1; its three-dimensional coordinates with time attributes are: (x l ,y m ,t i ), using formula (3), we get the voxel pixel weighted value E(x l ,y m ,t i ): in: ω is the three-dimensional coordinate (x l ,y m ,t i ) at the pixel value; p jki The meaning is: the event camera asynchronously senses the brightness signal change of the incident light at each pixel position, and generates an event when the absolute value of the brightness signal change exceeds the threshold CT, using the four-tuple e jki =(x j ,y k ,p jki ,t i ) means, where (x j ,y k ) represents the horizontal and vertical coordinates of the pixel where the event is triggered, t i Indicates the timestamp of the event; p jki Indicates the polarity of the event, p jki ∈{-1,1}, that is: p jki -1 means darkening; p jki is 1, indicating brightening; if no event occurs at the corresponding pixel position, then p jki is a null value; α is the mixing weight coefficient, the size of which is determined by the proportion of events generated in the voxel grid E, and the value ranges from 0 to 1, that is: α∈[0,1]; Therefore, in determining the voxel pixel weight value E(x l ,y m ,t i ), the voxel grid E is determined.

6. The method for enhancing aerial image details based on deep learning according to claim 5, characterized in that: When calculating the loss function value between each video frame sequence, the voxel grid E of each video frame sequence is first regularized, and then the regularized voxel grid E is normalized. The loss is calculated using Euclidean distance.

7. The method for enhancing aerial image details based on deep learning according to claim 6, characterized in that: The voxel grid E is regularized as follows: ① For each voxel pixel weighted value E(x l ,y m ,t i ), using formula (4), we get the intermediate variable Mask(x l ,y m ,t i ): ② Using formula (5), all intermediate variables Mask(x k ,y m ,t i ) and sum them to obtain the weighted values ​​of voxels E(x k ,y m ,t i ), denoted as N; ③Use formula (6) to obtain the mean μ: ④Use formula (7) to obtain the variance σ: ⑤Use formula (8) to obtain the voxel pixel weight value W(x l ,y m ,t i ) By weighting each voxel pixel by E(x l ,y m ,t i ) regularization, that is, regularizing the voxel grid E.

Images